Information processing device, display system, display method, and program

Abstract

The loading status of objects within a designated location is understood, along with their relative positions within the location. [Solution] The system includes a loading status determination unit that determines the loading status of objects by a worker or a mobile body controlled by the worker within a predetermined location, based on an image captured from the mobile body within the predetermined location and identification information indicating the object, and an output unit that outputs to a display device, superimposed on a map indicating the predetermined location, at least one of the following as the loading status of the object determined by the loading status determination unit: the loading position of the object, the number of stacking layers of the object at the loading position, and the height of the highest stacked object.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a display system, a display method, and a program.

Background Art

[0002] Conventionally, based on an image captured by a camera attached to a moving body such as a forklift that mainly travels indoors in a warehouse or factory, the position information of the moving body is estimated, and further, the conveyance state (holding / non-holding) of an object (cargo or pallet) moved by the moving body is recognized to obtain the position information of the object. A technique is already known.

[0003] Also, in Patent Document 1, using an RFID attached to an object in which stage information for specifying the loading position of the object and the stage where the object is located is stored, and information of a server that stores the number of objects on stages other than the uppermost stage, A loading status determination system that outputs the loading position of the loaded object and the calculated object number information to a display unit is disclosed.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, although it is possible to output the loading status of an object for each loading position, there is a problem that the loading status of an object within a predetermined base cannot be grasped in conjunction with the positional relationship within the base.

[0005] The present invention has been made in view of the above, and an object thereof is to provide an information processing apparatus, a display system, a display method, and a program capable of grasping the loading status of an object within a predetermined base in conjunction with the positional relationship within the base.

Means for Solving the Problems

[0006] To solve the above-mentioned problems and achieve the objective, the present invention is characterized by comprising: a loading status determination unit that determines the loading status of objects by a worker or a mobile body controlled by the worker within a predetermined location based on an image acquired from the mobile body within the predetermined location and identification information indicating the object; and an output unit that outputs to a display device, superimposed on a map indicating the predetermined location, at least one of the loading position of the object, the number of stacking layers of the object stacked at the loading position, and the height of the highest stacked object, as the loading status of the object determined by the loading status determination unit. [Effects of the Invention]

[0007] According to the present invention, the loading status of objects within a predetermined location can be understood in conjunction with their relative positions within that location. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 illustrates an example of location information for objects within a warehouse. [Figure 2] Figure 2 shows an example of the overall configuration of the display system according to the first embodiment. [Figure 3] Figure 3 shows the relationship between the 360-degree camera on the forklift and the marker. [Figure 4] Figure 4 shows the configuration and installation method of the marker. [Figure 5] Figure 5 is a block diagram showing an example of an on-premises server hardware configuration. [Figure 6] Figure 6 is a block diagram showing an example of the functional configuration of a display system. [Figure 7] Figure 7 shows an example of a stacking type. [Figure 8] Figure 8 is a flowchart showing an example of the cargo loading / unloading position and height estimation process by the cargo loading / unloading position and height estimation unit. [Figure 9] Figure 9 is a flowchart showing the processing flow when the management department attaches the device. [Figure 10] Figure 10 is a flowchart showing the processing flow by the management department when detaching. [Figure 11] Figure 11 shows an example of a UI displayed on a display device according to the output from the output unit. [Figure 12] Figure 12 shows another example of a UI displayed on a display device according to the output from the output unit. [Figure 13] Figure 13 shows a modified example of the UI displayed on the display device according to the output from the output unit. [Figure 14] Figure 14 shows another modified example of the UI displayed on the display device according to the output from the output unit. [Figure 15] Figure 15 shows the configuration of a forklift according to the second embodiment. [Figure 16] Figure 16 is a flowchart showing the process of switching between the two cameras. [Figure 17] Figure 17 shows the configuration of a forklift according to the third embodiment. [Figure 18] Figure 18 shows a modified example 1 of the configuration of a forklift according to the third embodiment. [Figure 19] Figure 19 shows a modified example 2 of the configuration of a forklift according to the third embodiment. [Figure 20] Figure 20 shows an example of the hardware configuration of an HMD used in the display system according to the fourth embodiment. [Modes for carrying out the invention]

[0009] Embodiments of the information processing device, display system, display method, and program will be described in detail below with reference to the attached drawings.

[0010] (First Embodiment) The information processing apparatus according to the first embodiment processes the position information of an object moved by a moving body. For example, the moving body is a counter-type forklift, and the object includes the goods themselves or a pallet. The information processing apparatus according to the embodiment processes the position information of the object moved by the forklift to recognize and track the movement of the object.

[0011] Here, FIG. 1 is a diagram for explaining an example of the position information of an object in a warehouse. FIG. 1 shows a view of the inside and the periphery of the warehouse 100 from above (the ceiling side).

[0012] The warehouse 100 is a terminal warehouse (a warehouse installed at a transportation relay point). This terminal warehouse has a warehouse form called a cross-docking type. In a cross-docking type warehouse, a plurality of pallets 31 for each commodity cargo are received from a factory or a wholesaler, etc., and are temporarily placed in the warehouse. Then, at the time of shipment, while maintaining the loading state of the same pallet 31, a plurality of types of pallets 31 are combined and shipped according to the aspect of the retail store.

[0013] In FIG. 1, there is a truck yard 200 around the warehouse 100. FIG. 1 shows a state where the trailer-transported and separated container 300 and the truck bed are connected to the warehouse in the truck yard 200. The forklift 10 takes out the pallet 31 from at least one of the truck bed of the truck that has arrived at the truck yard 200 or the container 300 transported by the trailer.

[0014] After that, the forklift 10 transports the pallet 31 to the temporary storage place 40 and temporarily stores it. Then, at the time of shipment, the forklift 10 transports the pallet 31 to a place near the truck yard 200 in the warehouse 100, aligns the loads, and then loads the pallet 31 onto the truck bed or the container 300.

[0015] In the temporary storage area 40, there are often no designated compartments for each type of cargo in order to flexibly secure space to accommodate the daily fluctuations in the types and quantities of goods coming in and out. On the other hand, since multiple workers temporarily place pallets 31 in any location, when shipping, it is necessary to search for the desired pallet 31 from among the multiple temporarily placed pallets 31 and perform the cargo handling work (the work of rearranging the cargo).

[0016] To efficiently carry out this cargo handling operation (the operation of rearranging cargo), a display system is needed that makes effective use of space by not designating a temporary storage location for the pallets 31, while recognizing and tracking the movement of the pallets 31 within the warehouse 100 and making it visible. The information processing device according to this embodiment is used in such a display system as an example.

[0017] The following describes a display system having an information processing device according to the first embodiment.

[0018] (Example of the overall configuration of display system 1) Figure 2 shows an example of the overall configuration of the display system 1 according to the first embodiment. As shown in Figure 2, the display system 1 includes a forklift 10, a 360-degree camera 20, an on-premise server 50, and a display device 90. These are connected to each other via a network 400 such as a LAN (Local Area Network). Other devices such as external servers or image forming apparatus may also be connected to the network 400 in a communication manner.

[0019] Forklift 10 is an example of a mobile device that transports pallet 31 and cargo 32 by holding and transporting cargo 32 placed on pallet 31. Transport by a mobile device is an example of movement by a mobile device. Pallet 31 and cargo 32 are examples of objects. In the following, when pallet 31 and cargo 32 are not specifically distinguished, they will be collectively referred to as object 30. Forklift 10 is a general term for multiple forklifts, pallet 31 is a general term for multiple pallets, and cargo 32 is a general term for multiple cargoes.

[0020] The forklift 10 may transport the object 30 in accordance with the operator's driving operations, or it may transport the object 30 by automatic operation without operator intervention.

[0021] The 360-degree camera 20 is an example of an imaging device that is attached to a forklift 10 and images the area around the forklift 10. The 360-degree camera 20 is a camera that can image in all directions (360 degrees) around the 360-degree camera 20 as its field of view. Direction 20a indicates the direction in which the 360-degree camera 20 can image.

[0022] The 360-degree spherical image (omnidirectional image) captured by the 360-degree camera 20 is just one example of an image captured. However, the imaging unit is not limited to the 360-degree camera 20; any device with a wide field of view close to a full sphere and capable of imaging the area around the forklift 10 is acceptable. Furthermore, the captured image does not necessarily have to be a 360-degree spherical image.

[0023] The 360-degree image includes images of the scenery in the direction of transport 11 of the object 30 as viewed from the forklift 10, and the scenery in the direction of vertical upward 12 as viewed from the forklift 10. In other words, the transport direction 11 is in front of the forklift 10, and the vertical upward direction 12 is above the forklift 10. Since the 360-degree camera 20 can capture images in all directions, it can capture both the area in front of and above the forklift 10 in a single image. Note that the transport direction 11 is just one example of a direction of movement.

[0024] The 360-degree camera 20 is preferably mounted on the head guard (roof of the driver's seat) 10a of the forklift 10, or on the outer mast 22 that supports the forks 21. This allows for a suitable field of view for imaging the area in front of and above the forklift 10. Here, the forks 21 and backrest 23 are examples of holding parts provided on the forklift 10 that hold objects and allow them to be raised and lowered. In this embodiment, the 360-degree camera 20 is mounted on the outer mast 22 that supports the forks 21.

[0025] The 360-degree camera 20 has wireless communication capabilities and transmits the captured 360-degree images to the on-premise server 50 via the network 400.

[0026] Here, Figure 3 shows the relationship between the 360-degree camera 20 and marker M1 of the forklift 10. Figure 4 shows the configuration and installation of the marker M1. The forklift 10 shown in Figure 3 is an example in which the 360-degree camera 20 is mounted on the outer mast 22 that supports the forks 21. Figures 3(a) to 3(c) show the rising of the marker M1 as the forks 21 of the forklift 10 rise.

[0027] The outer mast 22 is a support member that supports the inner mast 24 so that it can move up and down. The inner mast 24 moves up and down by chain drive, causing the forks 21 to move up and down, and is a member used for raising and lowering. Generally, the raising of the forks 21 of the forklift 10 is achieved by the inner mast 24 rising, and the backrest 23 and forks 21 rising relative to the inner mast 24.

[0028] As shown in Figure 3, the 360-degree camera 20 is mounted on the outer mast 22 at a point that is not on the same line as the forks 21 of the forklift 10 and the upward and downward movement direction (vertically upward direction 12) of the backrest 23 which moves up and down together with the forks 21. Since the outer mast 22 does not move up and down, it is possible to wire it to the main body of the forklift 10 with a cable.

[0029] In this embodiment, a marker M1 attached to the backrest 23 is captured by a 360-degree camera 20. The type of marker M1 is then detected by image recognition of the image captured by the 360-degree camera 20, and the height of the fork 21 is determined from the detected type and position of the marker M1.

[0030] As shown in Figures 3 and 4, the backrest 23 is equipped with multiple cylindrical markers M1. The markers M1 are captured by the 360-degree camera 20 and image-recognized. The position of the markers M1 is detected from the images of the markers M1 captured by the 360-degree camera 20, and this is used to estimate the height of the forks 21 of the forklift 10 and the object 30. In this embodiment, three markers M1 are provided on the backrest 23 at different heights.

[0031] As shown in Figure 4, the marker M1 installed on the backrest 23 is cylindrical in shape. Furthermore, the marker M1 has different patterns depending on its installation position on the backrest 23. The three markers M1 with different patterns are designated as marker M1a, marker M1b, and marker M1c. By making the marker M1 cylindrical, even when the relative angle between the 360-degree camera 20 used for height detection and the marker M1 changes, the appearance of the marker M1 can be kept approximately the same cylindrical diameter, allowing for stable detection of the marker M1's pattern.

[0032] In this embodiment, cylindrical markers M1 with different patterns are installed on the backrest 23, but this is not the only option. For example, LEDs (Light Emitting Diodes) may be used to form markers M1 with different patterns. Alternatively, the structure of the backrest 23 itself may be used as the marker M1.

[0033] The cargo 32 is provided with a barcode 33, which is an example of identification information for the cargo 32. Such a barcode may also be provided on the pallet 31 and used as identification information for the pallet 31. The barcode 33 is read by a reader such as a barcode reader, and the identification information resulting from the reading is transmitted to the on-premise server 50 via the network 400. Note that the identification information is not limited to a barcode, but may also be a QR code (registered trademark) or an ID (Identifier) ​​number, etc.

[0034] The on-premise server 50 is an example of an information processing device installed in a warehouse 100 that processes location information of objects 30 transported by a forklift 10. The on-premise server 50 can be replaced with a cloud server (a computing device installed outside the warehouse environment). This reduces initial and running costs.

[0035] The on-premise server 50 processes the location information of object 30 based on the 360-degree image and identification information indicating object 30 received via the network 400. Furthermore, by using the 360-degree images captured by multiple 360-degree cameras 20, the location information of object 30 held by multiple forklifts 10 can be obtained.

[0036] Furthermore, the on-premises server 50 performs self-localization calculations for the forklifts 10 based on the 360-degree images received via the network 400. The on-premises server 50 determines the current positions of multiple forklifts 10 by performing self-localization calculations for multiple 360-degree cameras 20.

[0037] The forklift 10 is equipped with, for example, a single-board computer (on-board edge). The forklift 10 has a 360-degree camera 20 connected to the single-board computer via a wired connection. The forklift 10 may perform self-position recognition calculation processing on the single-board computer and transmit the results to the on-premise server 50 via wireless communication over the network 400 for aggregation. This allows the on-premise server 50 to know the current position of each forklift 10.

[0038] The on-premise server 50 reads identification information, current location information, captured images, etc., from the HD504 (see Figure 5) and other devices within the on-premise server 50 to determine the final location information of the forklift 10 and pallet 31, and visualizes it by outputting it to the display device 90. The display device 90 is a device equipped with a display unit 93 such as an LCD (Liquid Crystal Display), and can be a tablet terminal, smartphone, PC (Personal Computer), etc. The display device 90 may be mounted on the forklift 10 or carried by the worker.

[0039] As described above, the display system 1 has an on-premise server 50 outside the forklift 10, and the processing unit is located on the on-premise server 50. This reduces the power consumption of the single-board computer in the forklift 10. Specifically, the power consumption, which was previously around 40W, can be reduced to around 10W.

[0040] However, since the forklift 10 moves around, there is a possibility of communication interruptions between the forklift 10 (or the on-board edge) and the on-premise server 50. Therefore, although not specifically shown in the figures, in this embodiment, the forklift 10 (or the on-board edge) and the on-premise server 50 are each equipped with a communication status confirmation unit. This allows monitoring of the connection status on the network 400. When communication is interrupted, any unsent data from the time of the interruption will be sent all at once after the connection is re-established.

[0041] (Example hardware configuration for 50 on-premises servers) Next, Figure 5 is a block diagram showing an example of the hardware configuration of an on-premises server 50. The on-premises server 50 is built using computers.

[0042] As shown in Figure 5, the on-premise server 50 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, an HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, and a display 506. The on-premise server 50 also includes an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0043] Of these components, CPU 501 controls the operation of the entire on-premises server 50. ROM 502 stores programs used to drive CPU 501, such as IPL. RAM 503 is used as the work area for CPU 501.

[0044] HD504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to HD504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images.

[0045] External device connection I / F 508 is an interface for connecting various external devices. These external devices include, for example, USB (Universal Serial Bus) memory and printers. Network I / F 509 is an interface for data communication with the forklift 10, display device 90, etc., using network 400. Bus line 510 is an address bus, data bus, etc., for electrically connecting various components such as the CPU 501 shown in Figure 5.

[0046] The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The DVD-RW drive 514 controls the reading or writing of various data to the DVD-RW 513, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may also be DVD-R, etc. The media interface 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0047] (Example of the functional configuration of display system 1) Figure 6 is a block diagram showing an example of the functional configuration of the display system 1. As shown in Figure 6, the on-premise server 50 includes a receiving unit 51, a moving object position acquisition unit 52, a holding information acquisition unit 53, an identification information acquisition unit 54, a time acquisition unit 55, an object position acquisition unit 56, an output unit 57, a storage unit 58, a cargo attachment / detachment position / height estimation unit 59, and a management unit 60. The moving object position acquisition unit 52, the holding information acquisition unit 53, the identification information acquisition unit 54, the time acquisition unit 55, the object position acquisition unit 56, the cargo attachment / detachment position / height estimation unit 59, and the management unit 60 function as a loading status determination unit 61.

[0048] The loading status determination unit 61 determines the loading status of objects 30 (cargo 32) by workers or forklifts 10 controlled by workers within a predetermined location, based on captured images (spherical images (omnidirectional images) captured by the 360-degree camera 20) acquired from the forklifts 10 within the predetermined location and identification information indicating the objects 30 (cargo 32).

[0049] Each of these components is a function or means of functioning, which is realized when any of the components shown in Figure 5 operates according to instructions from the CPU 501 that follow a program deployed from HD 504 onto RAM 503.

[0050] The forklift 10 also includes a 360-degree camera 20, a transmission unit 101, and a recording unit 102. The 360-degree camera 20 acquires images of the surroundings (360-degree images). The recording unit 102 records the 360-degree images acquired by the 360-degree camera 20 onto a recording medium. The transmission unit 101 transmits the 360-degree images acquired by the 360-degree camera 20 to the on-premise server 50 via the network 400.

[0051] The functions of the transmitting unit 101 and the recording unit 102 can be realized by an electrical circuit provided in either the forklift 10 or the 360-degree camera 20, or by software (CPU). They may also be realized by multiple circuits or multiple software programs.

[0052] The display device 90 comprises a communication unit 91 and a display control device 92.

[0053] The communication unit 91 receives display information output from the output unit 57 of the on-premise server 50. The display control device 92 displays the display information received by the communication unit 91 on the display unit 93.

[0054] The functions of the communication unit 91 and the display control device 92 can be implemented by electrical circuits, or by software (CPU). They may also be implemented by multiple circuits or multiple software programs.

[0055] The on-premise server 50 acquires the position information of the object 30 based on the position information of the forklift 10 acquired based on the 360-degree image from the 360-degree camera 20, and the holding information indicating whether the forklift 10 is holding or not holding the object 30. The on-premise server 50 can then output the acquired position information of the object 30, etc., via the output unit 57 to the display unit 93 of the display device 90, etc., so that it can be displayed.

[0056] The receiving unit 51 receives the 360-degree images captured by the 360-degree camera 20 and transmitted via the transmitting unit 101 via the network 400, and outputs them to the moving object position acquisition unit 52 and the retained information acquisition unit 53, respectively. The receiving unit 51 also receives identification information read by a reader such as a barcode reader via the network 400, and outputs it to the identification information acquisition unit 54.

[0057] The mobile object position acquisition unit 52 calculates and acquires the position information of the forklift 10 based on the input 360-degree image and outputs it to the object position acquisition unit 56 and the management unit 60. Simultaneous Localization and Mapping (SLAM) technology can be applied to the process of acquiring the position information of the forklift 10 (self-position estimation process) (see, for example, "Explanation of the Current Status and Future Prospects of SLAM," Masahiro Tomonaga, Yoshitaka Hara, Systems / Control / Information, Vol. 64, No. 2, 2020, pp. 45-50, https: / / www.jstage.jst.go.jp / article / isciesci / 64 / 2 / 64_45 / _article / -char / ja / ).

[0058] The holding information acquisition unit 53 calculates and acquires holding information indicating either the holding or non-holding state of the object 30 by the forklift 10 based on the input 360-degree image, and outputs it to the object position acquisition unit 56.

[0059] The identification information acquisition unit 54 acquires identification information by receiving it from the receiving unit 51 and outputs it to the object position acquisition unit 56. However, the acquisition of identification information by the identification information acquisition unit 54 is not limited to via the network 400. For example, the identification information acquisition unit 54 may acquire identification information entered by a user such as an administrator using the keyboard 511 or pointing device 512 shown in Figure 5, or it may acquire identification information that has been pre-stored in the storage unit 58, or it may acquire identification information via the external device connection I / F 508. The administrator is the administrator of the display system 1 or the warehouse 100.

[0060] The time acquisition unit 55 acquires information indicating the time when the receiving unit 51 received the 360-degree image and identification information, and outputs it to the object position acquisition unit 56.

[0061] The object position acquisition unit 56 acquires the position information of the object 30 based on the position information and holding information of the forklift 10. The object position acquisition unit 56 also associates the position information of the object 30 with identification information indicating the object 30 and time information, and stores them in the storage unit 58.

[0062] The cargo attachment / detachment position and height estimation unit 59 acquires the vehicle position estimation result, which is the position information of the forklift 10, from the mobile body position acquisition unit 52, and also acquires the height estimation result, which is the height position information of the forks 21 estimated based on the pattern of the marker M1 of the backrest 23 included in the input 360-degree image, and estimates the attachment / detachment position and height when the forklift 10 attaches or detaches the object 30. The cargo attachment / detachment position and height estimation unit 59 is an example of an estimation unit that estimates the height of the object 30 by detecting the position of the marker M1 from the image of the marker M1 captured by the 360-degree camera 20.

[0063] The cargo attachment / detachment position and height estimation unit 59 outputs the estimated attachment / detachment position and height when the forklift 10 attaches or detaches the object 30 to the management unit 60.

[0064] The management unit 60 manages the attachment (attached) state and the detached state of the object 30 (cargo 32) on the forklift 10, based on the attachment and detachment position and height estimated by the cargo attachment / detachment position and height estimation unit 59 when the forklift 10 attaches or detaches the object 30, and the position information of the forklift 10 calculated by the mobile body position acquisition unit 52.

[0065] In addition, the management unit 60 calculates the number of stacking levels of the object 30 (cargo 32) stacked at the loading position adjacent to the forklift 10, and updates the number of stacking levels of the object 30 (cargo 32) stacked at the loading position adjacent to the forklift 10, based on the loading position and height estimated by the cargo loading / unloading position and height estimation unit 59 when the forklift 10 loads or unloads the object 30.

[0066] Furthermore, the control unit 60 manages the detach height and stacking level of the highest stacked object 30 (cargo 32) at the loading position adjacent to the forklift 10, where the bottom surface of the highest stacked object 30 (cargo 32) is detached from the forklift 10. The detach height is the height (length) from the ground to the position where the object 30 (cargo 32) is detached by the forklift 10. In other words, the detach height is the height of the forks from the ground when the object 30 (cargo 32) is detached by the forklift 10. However, the detach height may be other than the position where the object is detached by the forklift 10. That is, the detach height is not limited to the height (length) from the ground to the position where the object is detached by the forklift 10. Therefore, the height (length) from the ground to the position where the object is detached by the forklift 10, or the height (length) to a position other than the position where the object is detached by the forklift 10, is used as information regarding the height of the highest stacked object.

[0067] Furthermore, the management unit 60 stores definition information for the stacking type (direct stacking / rack stacking) for each location.

[0068] Here, we will explain the different types of stacking (direct stacking / rack stacking).

[0069] Figure 7 shows an example of different stacking methods (direct stacking / rack stacking). As shown in Figure 7, direct stacking involves stacking objects 30 (cargo 32) directly on top of each other. On the other hand, rack stacking involves stacking objects 30 (cargo 32) on shelves (racks).

[0070] For example, in the direct stacking example shown in Figure 7(a), if the height of the object 30 is 0.5m / layer, the detach height of the bottom of the highest stacked object 30 (cargo 32) in direct stacking at a predetermined loading position is "3.0m", and the number of stacking layers of the highest stacked object 30 (cargo 32) is "7 layers". Also, in the rack stacking example shown in Figure 7(b), the number of stacking layers of the highest stacked object 30 (cargo 32) in rack stacking at a predetermined loading position is "4 layers". Note that in the rack stacking example shown in Figure 7(b), the number of shelves (racks) that can store 4 layers are "3 layers" in use, and the number of empty shelves (racks) is "1 layer".

[0071] The storage unit 58 can store identification information indicating an object 30 such as a pallet 31 or cargo 32.

[0072] Furthermore, the storage unit 58 stores the position information of the object 30, the identification information indicating the object 30, and the time information acquired by the object position acquisition unit 56, in association with each other.

[0073] Furthermore, the storage unit 58 stores the number of stacking levels of the objects 30 (cargo 32) loaded at the loading position adjacent to the forklift 10, as updated by the management unit 60.

[0074] The output unit 57 generates display information and outputs it to the display device 90 or the like. More specifically, the output unit 57 outputs location information of the object 30 (cargo 32) stored in the storage unit 58 so that it can be displayed on map M (see Figure 11), which is a map indicating predetermined locations.

[0075] The output unit 57 outputs display information that can be displayed on the map M (see Figure 11) regarding the loading status of the cargo 30 (goods 32), which is the cargo, for loading operations by a worker or a forklift 10 controlled by a worker at a predetermined location.

[0076] More specifically, the output unit 57 outputs display information superimposed on a map M (see Figure 11) indicating a predetermined location, which includes at least the loading position of an object 30 (cargo 32) adjacent to a worker or forklift 10, and at least one of the number of stacking levels of the objects 30 (cargo 32) at the loading position and the height of the highest stacked object 30 (cargo 32).

[0077] (Example of processing using on-premises server 50) Figure 8 is a flowchart illustrating an example of the cargo attachment / detachment position and height estimation process performed by the cargo attachment / detachment position and height estimation unit 59. The preconditions are as follows. The cargo loading / unloading position and height estimation unit 59 constantly performs cargo loading / unloading position and height estimation processing. In practice, it processes based on, for example, 10fps video and updates the information to the management unit 60 at, for example, 1-second intervals. The management unit 60 stores definition information for the stacking type (rack stacking / direct stacking) for each location. • When stacking directly, initially stack one layer at a time. However, after the first layer is stacked, you may stack multiple layers together.

[0078] As shown in Figure 8, first, the cargo attachment / detachment position and height estimation unit 59 obtains the vehicle position estimation result, which is the position information of the forklift 10, from the mobile body position acquisition unit 52, and also obtains the height estimation result, which is the height position information of the forks 21 estimated based on the pattern of the marker M1 of the backrest 23 included in the input 360-degree image (step S1).

[0079] Next, the cargo attachment / detachment position and height estimation unit 59 determines whether the height of the forks 21 of the forklift 10 is near the ground by comparing it with a preset threshold (step S2).

[0080] If the cargo attachment / detachment position and height estimation unit 59 determines that the height of the forks 21 of the forklift 10 is not near the ground (No. in step S2), it stores the vehicle position of the forklift 10 and the height of the forks 21 as history for calculating the high-altitude attachment / detachment position (step S3).

[0081] On the other hand, if the cargo attachment / detachment position and height estimation unit 59 determines that the height of the forks 21 of the forklift 10 is near the ground (Yes in step S2), it determines whether the forks 21 are loaded or not (either still holding the object 30 or not) (step S4).

[0082] The cargo attachment / detachment position and height estimation unit 59 determines, through continuous image recognition over a certain period of time on the input 360-degree image, that there has been no change in the cargo status (Yes in step S4), and terminates processing, assuming that the object 30 is still being held by the fork 21 or is no longer being held by the fork 21.

[0083] Meanwhile, the cargo attachment / detachment position and height estimation unit 59 determines, based on continuous image recognition over a certain period of time of the input 360-degree image, that there has been a change in the cargo status (No. in step S4), and then determines whether to attach (attach the object 30 to the fork 21) or detach (detach the object 30 from the fork 21) (step S5).

[0084] The cargo attachment / detachment position and height estimation unit 59 determines that an attachment event (attaching object 30 to fork 21) has occurred (Yes in step S5) when the state changes from having no cargo to having cargo, based on continuous image recognition of the input 360-degree image over a certain period of time. It then transmits an attachment signal indicating the attachment position and attachment height to the management unit 60 (step S6).

[0085] Meanwhile, the cargo attachment / detachment position and height estimation unit 59 determines that a detachment event (detachment of object 30 from fork 21) has occurred when the state changes from having cargo to not having cargo, based on continuous image recognition of the input 360-degree image over a certain period of time (No. in step S5), and transmits a detachment signal indicating the detachment position and detachment height to the management unit 60 (step S7).

[0086] Next, the cargo attachment / detachment position and height estimation unit 59 determines whether the vehicle position of the forklift 10 and the height of the forks 21 are stored as history for calculating the high-altitude attachment / detachment position (step S8).

[0087] If the cargo attachment / detachment position / height estimation unit 59 determines that the vehicle position of the forklift 10 and the height of the forks 21 are not stored as history for calculating high-altitude attachment / detachment positions (No. in step S8), it calculates the attachment / detachment position of the object 30 near the ground from the vehicle position (temporary record and current position) and set parameters (step S9).

[0088] Next, the cargo attachment / detachment position and height estimation unit 59 transmits the calculated attachment / detachment position and height of the object 30 near the ground to the management unit 60 (step S10), and the process ends.

[0089] On the other hand, if the cargo attachment / detachment position / height estimation unit 59 determines that the vehicle position of the forklift 10 and the height of the forks 21 are stored as history for calculating high-altitude attachment / detachment positions (Yes in step S8), it assumes that there has been a change in the loading state before and after the height of the forks 21 is other than near the ground, and calculates the high-altitude attachment / detachment position of the object 30 from the vehicle position (temporary record and current position) and set parameters (step S11).

[0090] Next, the cargo attachment / detachment position and height estimation unit 59 transmits the calculated attachment / detachment position and height of the object 30 at a high point to the management unit 60 (step S12).

[0091] Subsequently, the cargo attachment / detachment position and height estimation unit 59 resets the history of the height of the forks 21 and the vehicle position of the forklift 10 (step S13), and terminates the process.

[0092] Next, we will explain the processing in the management unit 60.

[0093] Furthermore, there are two methods for stacking the objects 30 (cargo 32): a rack stacking mode in which the objects 30 (cargo 32) are stacked on shelves (racks), and a direct stacking mode in which the objects 30 (cargo 32) are stacked directly on the surface. These can be manually switched by, for example, an operator of the forklift 10.

[0094] First, let's explain the processing flow when the management unit 60 attaches the device. Figure 9 is a flowchart showing the processing flow when the management unit 60 attaches the device.

[0095] When the management unit 60 receives the attachment position and height of the object 30 (cargo 32) calculated by the cargo attachment / detachment position and height estimation unit 59 (step S21), it determines whether the attachment position and height of the object 30 (cargo 32) matches that of an existing object 30 (cargo 32) within a threshold (step S22).

[0096] If the management unit 60 determines that the attachment position and height of the object 30 (cargo 32) match those of an existing object 30 (cargo 32) within a threshold (Yes in step S22), it determines whether the object 30 (cargo 32) is in rack stacking mode (step S23). The mode is determined from the "location-specific stacking type (rack stacking / direct stacking) definition information ("location definition information")" stored in the management unit 60.

[0097] If the management unit 60 determines that it is not rack stacking mode (No. in step S23), it determines that it is direct stacking mode in which objects 30 (cargo 32) are directly stacked, and recognizes that the attached existing objects 30 (cargo 32) and all existing objects 30 (cargo 32) stacked at a higher position than the attached existing objects 30 (cargo 32) are being removed (step S24).

[0098] On the other hand, if the management unit 60 determines that the attachment position and height of the object 30 (cargo 32) do not match those of an existing object 30 (cargo 32) within a threshold (No. in step S22), it determines that it is a new object 30 (cargo 32) (step S25).

[0099] On the other hand, if the management unit 60 determines that it is in rack stacking mode (Yes in step S23), it determines that it is in mode to stack the object 30 (cargo 32) on a rack (shelf), and determines that it is taking out an existing object 30 (cargo 32) (step S26).

[0100] Next, the management unit 60 performs the calculation of the number of items taken out (step S27).

[0101] For example, in step S24, if the attached existing object 30 (cargo 32) and all existing objects 30 (cargo 32) stacked at a higher position than the attached existing object 30 (cargo 32) are determined to be removed, the number of removed objects is "1" to "n". In step S25, if it is determined to be a new object 30 (cargo 32), the number of removed objects is "0". In step S26, if it is determined to be an existing object 30 (cargo 32) being removed, the number of removed objects is "1".

[0102] Next, the management unit 60 performs calculation and update processing of the number of steps information (step S28). For example, as an update to the latest number of steps information, the management unit 60 subtracts because it is an attachment.

[0103] In other words, when the management unit 60 receives an attachment signal, it calculates the number of objects 30 (cargo 32) moved to the fork 21 according to the flow in Figure 9, subtracts the number of layers removed from the designated location, and updates the latest layer count information.

[0104] Next, we will explain the processing flow by the management unit 60 when detaching. Figure 10 is a flowchart showing the processing flow by the management unit 60 when detaching.

[0105] When the management unit 60 receives the detachment position and height of the object 30 (cargo 32) calculated by the cargo attachment / detachment position and height estimation unit 59 (step S31), it determines whether the detachment height of the object 30 (cargo 32) is near the ground by comparing it with a preset threshold (step S32).

[0106] If the management unit 60 determines that the detachment height of the object 30 (cargo 32) is not near the ground (No. in step S32), it determines whether the object 30 (cargo 32) is in rack stacking mode (step S33). The mode is determined from the "location-specific stacking type (rack stacking / direct stacking) definition information ("location definition information")" stored in the management unit 60.

[0107] If the management unit 60 determines that it is not in rack stacking mode (No. in step S33), it determines that it is in direct stacking mode to directly stack the object 30 (cargo 32), and whether the detached position and height of the object 30 (cargo 32) match the position of the existing object 30 (cargo 32) that is laid flat within a threshold (step S34).

[0108] If the management unit 60 determines that the detachment position and height of object 30 (cargo 32) match those of an existing object 30 (cargo 32) within a threshold (Yes in step S34), it records it as a stack (step S35).

[0109] On the other hand, if the management unit 60 determines that the detachment height of the object 30 (cargo 32) is near the ground (Yes in step S32), it records it as flat cargo (step S36).

[0110] Furthermore, if the control unit 60 determines that it is in rack stacking mode (Yes in step S33), it records the detachment position and height of the object 30 (cargo 32) as being in a mode to stack the object 30 (cargo 32) on a rack (shelf) (step S37).

[0111] Furthermore, if the management unit 60 determines that the detachment position and height of object 30 (cargo 32) do not match those of an existing object 30 (cargo 32) within a threshold (No. in step S34), it records the detachment position and height of object 30 (cargo 32) with an unknown flag indicating that it is unknown (step S38).

[0112] As described above, when an action is taken to place another object 30 (cargo 32) on top of an object 30 (cargo 32) already on the ground, or to take an object 30 (cargo 32) from an object 30 (cargo 32) already on the ground, the position information of the stacked (or taken) object 30 (cargo 32) is estimated using the position information of the forklift 10, the height of the detected forks 21, and the position information of the object 30 (cargo 32) that was on the ground.

[0113] Next, the management unit 60 performs the calculation of the stacking count (step S39).

[0114] Next, the management unit 60 performs calculation and update processing of the number of steps information (step S40). For example, as the update of the latest number of steps information, the management unit 60 adds it because it is detached.

[0115] In other words, when the control unit 60 receives a detach signal, it adds the number of objects 30 (cargo 32) that have detached from the fork 21 according to the flow in Figure 10, and updates the number of cargo layers stacked in the designated location. Note that only at the beginning, the stacking is assumed to be done one layer at a time, so the number of objects 30 (cargo 32) added after detaching from the fork 21 will be "1". The assumption of stacking one layer at a time applies only at the beginning, so after the first time, the number of objects 30 (cargo 32) added may be other than "1".

[0116] Next, we will explain the display on the display unit 93 of the display device 90 according to the output from the output unit 57.

[0117] As described above, the display system 1 of this embodiment can track the object 30 (cargo 32) carried by the forklift 10 by indoor positioning and attachment / detachment detection using the 360-degree camera 20, and by linking identification information that represents the object 30 to the object 30, it is possible to record everything, including what was placed where, as location data. Furthermore, the display system 1 of this embodiment can recognize high-altitude cargo handling operations by detecting the height of the forks 21, and can determine "which level it was placed on / from which level it was picked up" in rack stacking or direct stacking.

[0118] Here, Figure 11 is a diagram showing an example of a UI (User Interface) displayed on the display device 90 according to the output from the output unit 57. In the UI example shown in Figure 11, for cargo handling work by a worker or a forklift 10 controlled by a worker within a predetermined location, the output unit 57 outputs a map M showing the loading status of objects 30 (cargo 32) to the display device 90 and displays it on the display unit 93. In the UI example shown in Figure 11, the output unit 57 outputs a map M showing the loading status, which includes a direct loading area A1 where objects 30 (cargo 32) are directly stacked and a rack loading area A2 where objects 30 (cargo 32) are rack-stacked, and displays it on the display unit 93 of the display device 90.

[0119] Furthermore, as shown in Figure 11, in the rack stacking area A2, frames F that mimic the rack (shelf) casing are displayed on the map M. By displaying the method of loading objects 30 (cargo 32) on the map M in a discernible manner, it becomes possible to clearly distinguish whether objects 30 (cargo 32) are stacked on racks (shelves) or directly stacked.

[0120] In the UI example shown in Figure 11, the output unit 57 outputs to the display device 90 and superimposes on the display unit 93 the loading positions (indicated by circular icons) of at least the objects 30 (cargo 32) adjacent to the worker or forklift 10, and the number of stacking layers (indicated by a number on a circular icon) of the objects 30 (cargo 32) stacked at the loading positions, onto a map M that indicates a predetermined location.

[0121] In the UI example shown in Figure 11, the output unit 57 outputs the number of stacking levels of the objects 30 (cargo 32) stacked at the loading position to the display device 90 and displays it on the display unit 93. However, this is not the only option, and the detachment height of the bottom surface of the objects 30 (cargo 32) stacked at the loading position may be indicated by a number (m) on a circular icon.

[0122] Furthermore, in the UI example shown in Figure 11, the output unit 57 also outputs the position of at least one of the worker or the forklift 10 to the display device 90 and displays it on the map M. The black square mark shown in Figure 11 indicates, for example, the position of the forklift 10.

[0123] For example, in conventional systems, the loading area directly in front of the forklift 10 (or worker), indicated by a black square mark for loading operations (three layers of continuous loading (objects 30)), makes it impossible to determine the number of loads (objects 30) further back (whether there is loading capacity, such as two layers or areas where there are no loads (objects 30) to the side). In such cases, conventional systems may cause the worker to move around to load operations within the visible area, or to move around and search for information to understand the surrounding loading situation.

[0124] In this respect, according to this embodiment, in the UI example shown in Figure 11, the status can be checked on the display unit 93 of the display device 90 carried by the worker or the display device 90 mounted on the forklift 10, making it possible to understand that there is a situation where more objects 30 (cargo 32) can be loaded nearby. When the worker or the forklift 10 controlled by the worker performs cargo handling work, they can understand what kind of work should be done at which loading position near the worker or the forklift 10, thereby improving the efficiency of cargo handling work.

[0125] Furthermore, if a worker is not familiar with the warehouse layout, or if they need to accurately understand the relationship between the visible cargo (object 30) and their own position, displaying the position of the worker or the forklift 10 on a display device 9 carried by the worker or mounted on the forklift 10 will enable more efficient cargo handling operations.

[0126] Here, Figure 12 shows another example of the UI displayed on the display device 90 according to the output from the output unit 57. The output unit 57 controls the display device 9 to output and display information about the object 30 (cargo 32) at the selected loading position when, for example, the pointing device 512 selects at least one loading position of the object 30 (cargo 32) on the map M.

[0127] In the UI example shown in Figure 12(a), for example, an example in which two loading positions of object 30 (cargo 32) on map M are selected by the pointing device 512 is shown enclosed in frame X.

[0128] In this way, when at least one loading position of the object 30 (cargo 32) on the map M is selected, as shown by the frame X in the UI example shown in Figure 12(a), the output unit 57 outputs information P about the object 30 (cargo 32) located within frame X to the display device 9 and displays it as a pop-up, as shown in the UI example shown in Figure 12(b). In the example shown in Figure 12(b), the information P about the object 30 (cargo 32) related to the selected loading position includes, for example, the storage date of the object 30 (cargo 32), the storage period (days) of the object 30 (cargo 32), the number of times the object 30 (cargo 32) has been moved, the ID of the forklift being worked on, and the position coordinates of the object 30 (cargo 32) (X coordinate (m), Y coordinate (m), Z coordinate (m)), which are associated with the cargo ID, which is identification information for the object 30 (cargo 32).

[0129] This makes it easy to obtain information about objects 30 (cargo 32) located in the back or at high places that are difficult to see, on the display screen of the display device 9.

[0130] Thus, this embodiment is characterized by the ability to overlay the loading status of objects within a predetermined location onto a map of the location, thereby displaying the loading status of objects in conjunction with their positional relationships within the location. This allows workers to check the loading status of objects on the display screen of a device they carry or a device mounted on a mobile vehicle, enabling them to understand situations where more objects can be loaded closer together, thereby improving the efficiency of cargo handling operations.

[0131] Furthermore, since workers do not need to record / register each item in storage areas in warehouses or shipping areas in factories, the location management of individual shipments can be made more efficient.

[0132] Furthermore, while "cargo handling operations" are usually performed to efficiently utilize the storage area, such as reducing empty space or moving cargo according to shipping timing, according to this embodiment, as long as the object 30 (cargo 32) is moved by the forklift 10, it is always tracked, and its position is automatically updated.

[0133] Furthermore, according to this embodiment, the position of all objects 30 (cargo 32) is visualized in the UI, and the level of occupancy in the storage area, including height, can be immediately grasped, allowing workers to perform cargo handling operations while overseeing the availability.

[0134] Furthermore, while objects 30 (cargo 32) located in the back or at high places are difficult to visually confirm, according to this embodiment, the location of a specific object 30 (cargo 32) can be immediately confirmed by performing a search on the UI.

[0135] [Differentiation] Here, we will explain some variations of the UI.

[0136] Figure 13 shows a modified UI displayed on the display device 90 according to the output from the output unit 57. As shown in Figure 13, the output unit 57 may output to the display device 9 the number of stacking levels A (shown as a number in the upper part of the circular icon) of the objects 30 (cargo 32) stacked at the loading position, as well as the detach height B of the bottom of the objects 30 (cargo 32) stacked at the loading position (shown as a number (m) in the lower part of the circular icon), and display them together.

[0137] By understanding both the number of stacking layers of the objects 30 (cargo 32) piled at the loading position and the detachment height of the bottom surface of the objects 30 (cargo 32) piled at the loading position, it becomes possible to immediately determine whether the objects 30 (cargo 32) targeted for loading can be stacked, taking into account the difference from the maximum possible stacking height, thereby enabling more efficient cargo handling.

[0138] Although not specifically shown in the diagram, the output unit 57 may also output to the display device 9 the detach height of the bottom surface of the uppermost object 30 (cargo 32) located in the loading position, and the height of the top surface of the uppermost object 30 (cargo 32) (indicated by numbers (m) above and below the circular icon, respectively) for display.

[0139] Figure 14 shows another modified example of the UI displayed on the display device 90 according to the output from the output unit 57. As shown in Figure 14, the output unit 57 may output to the display device 9 the number of stacking levels C of objects 30 (cargo 32) stacked in the rack (indicated by a number in the upper row of the circular icon), as well as the remaining number of objects 30 (cargo 32) that can be stacked in the rack D (indicated by a number in the lower row of the circular icon). In particular, the number of empty levels in the rack can be clearly distinguished from the number of stacking levels of objects 30 (cargo 32) stacked in the rack by using a circled number.

[0140] Note that the remaining load capacity may also be expressed as the filling rate. For example, if there is one empty shelf out of four, it should be expressed as 75%. Alternatively, if there is one empty shelf out of four, the remaining load capacity may be expressed as 1 / 4.

[0141] Furthermore, the number of empty shelves (racks) can be clearly distinguished from the number of stacked shelves (racks) by using a different color intensity for the empty shelves compared to the number of stacked shelves.

[0142] The program executed on the on-premise server 50 in this embodiment is provided pre-installed in a ROM 502 or the like. The program executed on the on-premise server 50 in this embodiment may also be configured to be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk).

[0143] Furthermore, the program executed on the on-premises server 50 of this embodiment may be configured to be stored on a computer connected to a network such as the Internet and provided by downloading it over the network. Alternatively, the program executed on the on-premises server 50 of this embodiment may be provided or distributed over a network such as the Internet.

[0144] The program executed on the on-premise server 50 in this embodiment is a module configuration that includes the above-mentioned parts (receiving unit 51, moving object position acquisition unit 52, retained information acquisition unit 53, identification information acquisition unit 54, time acquisition unit 55, object position acquisition unit 56, output unit 57, storage unit 58, cargo attachment / detachment position / height estimation unit 59, and management unit 60). In actual hardware, an example of a processor such as a CPU 501 reads the program from the ROM 502 and executes it, loading the above-mentioned parts onto the main memory, and generating the receiving unit 51, moving object position acquisition unit 52, retained information acquisition unit 53, identification information acquisition unit 54, time acquisition unit 55, object position acquisition unit 56, output unit 57, storage unit 58, cargo attachment / detachment position / height estimation unit 59, and management unit 60 on the main memory.

[0145] In this embodiment, a forklift is shown as an example of a mobile device, but it is not limited to this. For example, the mobile device may be an automated guided vehicle or a drone.

[0146] (Second Embodiment) Next, a second embodiment will be described.

[0147] The second embodiment differs from the first embodiment, which includes a 360-degree camera 20, in that it includes a sub-camera provided on the upper part of the outer mast 22 and a main camera provided on the head guard (roof of the driver's seat) 10a. In the following description of the second embodiment, the description of parts that are the same as those of the first embodiment will be omitted, and the parts that differ from the first embodiment will be described.

[0148] Figure 15 shows the configuration of the forklift 10 according to the second embodiment.

[0149] As shown in Figure 15, the forklift 10 of this embodiment includes a main camera 25 that captures images in the upward direction and a sub-camera 26 that captures images in the downward direction, instead of the 360-degree camera 20 of the first embodiment. The main camera 25 and sub-camera 26 are examples of imaging units provided on the forklift 10.

[0150] As shown in Figure 15, the main camera 25 is mounted on the head guard 10a of the forklift 10. The main camera 25 is an imaging device capable of capturing images with an upward 180-degree field of view around the main camera 25. The direction 25a indicates the direction in which the main camera 25 can capture images.

[0151] The images captured by the main camera 25 are primarily used for positioning, such as determining the location of the forklift 10. When positioning the forklift 10 indoors using the captured images, it is desirable to use an upward-facing main camera 25 that can capture images with a field of view of 180 degrees upwards, in order to easily capture the building structure, such as the ceiling and pillars.

[0152] Furthermore, as shown in Figure 15, the sub-camera 26 is mounted on the outer mast 22 at a location that is not on the same line as the upward and downward movement direction (vertically upward direction 12) of the forks 21 of the forklift 10, or the backrest 23 which moves up and down together with the forks 21. The sub-camera 26 is an imaging device capable of capturing images with a field of view of 180 degrees downwards around the sub-camera 26. The direction 26a indicates the direction in which the sub-camera 26 can capture images.

[0153] The state in which the forks 21 of the forklift 10 hold the object 30 is detected using images captured by the sub-camera 26 when the forks 21 are near the ground.

[0154] As will be explained in more detail later, the images captured by the main camera 25 and the sub-camera 26 are used by the cargo attachment / detachment position / height estimation unit 59 to estimate the attachment / detachment position and height when the forklift 10 attaches or detaches an object 30. In general terms, the cargo attachment / detachment position / height estimation unit 59 uses the image captured by the sub-camera 26 up to a certain height from the ground, and the image captured by the main camera 25 above that height, to estimate the attachment / detachment position and height when the forklift 10 attaches or detaches an object 30. With this method, there is less obstruction from the sub-camera 26 to the marker M1, and when the main camera 25 detects the marker M1, the main camera 25 is far from the rising axis of the marker M1, so height changes are easily reflected as changes in the position of the marker M1 in the image, making it an advantageous method for a counterbalanced forklift 10. When attaching or detaching an object at a height beyond the shooting range of the sub-camera 26, the attachment / detachment position of the object 30 cannot be directly seen from the main camera 25, but the attachment / detachment position and height of the object 30 can be estimated by the cargo attachment / detachment position / height estimation unit 59.

[0155] In this embodiment, a marker M1 attached to the backrest 23 is imaged by either the main camera 25 or the sub-camera 26. The image captured by either the main camera 25 or the sub-camera 26 is then used for image recognition to detect the type of marker M1, and the height of the fork 21 is determined from the detected type and position of the marker M1.

[0156] Next, we will explain how the cargo attachment / detachment position and height estimation unit 59 switches between the two cameras (main camera 25 and sub-camera 26).

[0157] Here, Figure 16 is a flowchart showing the flow of the switching process between the two cameras. As shown in Figure 16, if the cargo attachment / detachment position and height estimation unit 59 can detect the marker M1 by image recognition of the image captured by the sub-camera 26 (Yes in step S41), it estimates the attachment / detachment position and height of the object 30 using the image captured by the sub-camera 26 (step S42).

[0158] On the other hand, if the cargo attachment / detachment position and height estimation unit 59 cannot detect marker M1 by image recognition of the image captured by the sub-camera 26 (No. in step S41), it determines whether it can detect marker M1 by image recognition of the image captured by the main camera 25 (step S43).

[0159] If the cargo attachment / detachment position and height estimation unit 59 can detect the marker M1 by image recognition of the image captured by the main camera 25 (Yes in step S43), it estimates the attachment / detachment position and height of the object 30 using the image captured by the main camera 25 (step S44).

[0160] On the other hand, if the cargo attachment / detachment position and height estimation unit 59 cannot detect the marker M1 by image recognition of the image captured by the main camera 25 (No. in step S43), it terminates the process without estimating the attachment / detachment position and height of the object 30 (step S45).

[0161] As described above, according to this embodiment, a marker is installed on the backrest that moves up and down in conjunction with the forks of the forklift, a main camera capable of capturing images with an upward 180-degree field of view is attached to the head guard of the forklift, and a sub-camera capable of capturing images with a downward 180-degree field of view is attached at a location not on the same line as the upward and downward movement direction of the marker, and by acquiring images and estimating the height of the forks through image recognition, the height can be estimated not only from the size of the marker that appears in the image, but also from the information of the marker's position in the image. This makes it possible to improve the accuracy of detecting the height of an object moved by a moving body.

[0162] (Third embodiment) Next, a third embodiment will be described.

[0163] In the first or second embodiment, a counterbalanced forklift 10 was used as the mobile unit, but the third embodiment differs from the first or second embodiment in that a reach-type forklift is used as the mobile unit. In the following description of the third embodiment, the description of parts that are the same as those in the first or second embodiment will be omitted, and the parts that differ from the first and second embodiments will be described.

[0164] Figure 17 shows the configuration of a forklift according to the third embodiment. As shown in Figure 17, the forklift 70 of this embodiment is a reach-type forklift in which the forks 81, the outer mast 82 which is a support member, the backrest 83, and the inner mast 84 which is a member used for raising and lowering extend and retract (move back and forth).

[0165] As shown in Figure 17, the forklift 70 has a 360-degree camera 80, which is an imaging device, mounted on the head guard (roof of the driver's seat) 70a of the forklift 70. The direction 80a indicates the direction in which the 360-degree camera 80 can capture images. In addition, the forklift 70 in this embodiment has a marker M2 attached to the outer mast 82, which is different from the marker M1 on the backrest 83.

[0166] In this embodiment, the marker M2 is installed on the outer mast 82, but this is not the only option. For example, the marker M2 may be formed using an LED (Light Emitting Diode). Alternatively, the structure of the outer mast 82 itself may be used as the marker M2.

[0167] As shown in Figure 17, the forklift 70 has a structure in which parts including the forks 81, outer mast 82, backrest 83, inner mast 84, etc., extend outwards (reach).

[0168] With this configuration, the cargo attachment / detachment position and height estimation unit 59 of this embodiment can determine the height of the fork 81 by performing image recognition on an image captured by a 360-degree camera 80 attached to the head guard 70a and detecting the type of marker M1 on the backrest 83.

[0169] In addition, the cargo attachment / detachment position and height estimation unit 59 of this embodiment can determine the reach of the forks 81 by detecting the marker M2 on the outer mast 82 using image recognition of the image captured by the 360-degree camera 80 attached to the head guard 70a. By determining the reach, the relative positional relationship between the forklift 70 and the object 30 being transported, i.e., the attachment / detachment position of the object 30, can be determined more accurately.

[0170] In this embodiment, the 360-degree camera 80 is mounted on the head guard (roof of the driver's seat) 70a of the forklift 70, but this is not the only possible configuration.

[0171] [Example 1] Here, Figure 18 shows a modified example 1 of the configuration of the forklift according to the third embodiment. In modified example 1 shown in Figure 18, the forklift 70 may be equipped with a main camera 85 mounted on the head guard (roof of the driver's seat) 70a and a sub-camera 86 mounted on the upper part of the outer mast 82.

[0172] The main camera 85 is an imaging device capable of capturing images in an upward 180-degree direction around the main camera 85 as its field of view. Direction 85a indicates the direction in which the main camera 85 can capture images.

[0173] The sub-camera 86 is an imaging device capable of capturing images with a field of view of 180 degrees downwards around the sub-camera 86. The direction 86a indicates the direction in which the sub-camera 86 can capture images.

[0174] Furthermore, in the modified example 1 shown in Figure 18, a marker M2, which is different from the marker M1 on the backrest 83, is attached to the main body 71 of the forklift 70.

[0175] In this case, the cargo attachment / detachment position and height estimation unit 59 can determine the height of the fork 81 by performing image recognition on images captured by the main camera 85 attached to the head guard 70a or the sub-camera 86 attached to the outer mast 82 to detect the type of marker M1 on the backrest 83.

[0176] In addition, the forklift 70 can determine the reach of the forks 81 by detecting the marker M2 on the main body of the forklift 70 using image recognition captured by a sub-camera 86 located on the top of the outer mast 82. Furthermore, the cargo attachment / detachment position and height estimation unit 59 can stably detect the height of the forks 81 because the marker M1 on the backrest 23 can be easily recognized from the sub-camera 86 without obstruction.

[0177] [Differentiation 2] Here, Figure 19 shows a modified example 2 of the configuration of the forklift according to the third embodiment. In modified example 2 shown in Figure 19, the 360-degree camera 80 is mounted on the upper part of the outer mast 82, rather than on the head guard (roof of the driver's seat) 70a of the forklift 70.

[0178] In this case, the cargo attachment / detachment position and height estimation unit 59 can perform vehicle positioning, load detection, and fork height detection all based on the outer mast 82, thus eliminating the need to detect the amount of overhang using the marker M2 in a reach-type forklift.

[0179] (Fourth embodiment) Next, a fourth embodiment will be described.

[0180] In the first to third embodiments, a display device 90 equipped with a display screen such as a tablet terminal, smartphone, or PC was used as a display device to show the loading status of the object 30 (cargo 32). However, the fourth embodiment differs from the first or second embodiment in that an HMD (head-mounted display) is used as the display device to show the loading status of the object 30 (cargo 32). In the following description of the fourth embodiment, the description of parts that are the same as those of the first to third embodiments will be omitted, and the parts that differ from the first to third embodiments will be described.

[0181] Figure 20 shows an example of the hardware configuration of the HMD used in the display system 1 according to the fourth embodiment.

[0182] As shown in Figure 20, the HMD700, as a display device, is a computer equipped with a CPU701, ROM702, RAM753, external device connection I / F705, display707, control unit708, media I / F709, bus line710, speaker712, electronic compass718, gyro sensor719, and accelerometer720. Each of these components is connected to the bus line710.

[0183] Of these components, the CPU 701 controls the overall operation of the HMD 700. The ROM 702 stores programs used to drive the CPU 701, such as the IPL. The RAM 753 is used as the work area for the CPU 701.

[0184] The External Device Connection I / F 705 is an interface for connecting various external devices. These external devices include communication management servers, earphones with microphones, etc.

[0185] Display 707 is a type of display unit that displays various images, such as liquid crystal displays (LCDs) and organic EL (Electro-Luminescence) displays.

[0186] The operation unit 708 is an input means for selecting and executing various instructions such as various operation buttons, power switches, physical buttons, and eye-tracking circuits that detect and operate the user's gaze, as well as selecting processing targets and moving the cursor.

[0187] The media interface 709 controls the reading or writing (storage) of data to or from the recording media 709m, such as flash memory. The recording media 709m includes DVDs and Blu-ray® Discs, among others.

[0188] Speaker 712 is a circuit that converts electrical signals into physical vibrations to produce sound such as music and speech.

[0189] The 718 electronic compass calculates the HMD's orientation from the Earth's magnetic field and outputs orientation information.

[0190] The gyro sensor 719 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) associated with the movement of the HMD.

[0191] The accelerometer 720 is a sensor that detects acceleration in three axes.

[0192] Bus line 710 is an address bus, data bus, etc., used to electrically connect various components such as the CPU 701.

[0193] Thus, the display system 1 according to the fourth embodiment can be used to obtain the same effects and advantages as the first embodiment.

[0194] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each of the functions described above.

[0195] Furthermore, the information processing device is not limited to the on-premise server 50, as long as it is a device equipped with communication capabilities. The information processing device may also be, for example, an image forming apparatus, a PJ (Projector), an IWB (Interactive White Board: an electronic whiteboard with the ability to communicate with each other), an output device such as a digital signage system, a HUD (Head Up Display) device, industrial machinery, an imaging device, a sound collection device, medical equipment, networked home appliances, an automobile (Connected Car), a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet device, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, or a desktop PC.

[0196] Examples of the present invention are as follows: <1> A loading status determination unit determines the loading status of objects by a worker or a mobile body controlled by the worker within a predetermined location, based on captured images obtained from the mobile body within the predetermined location and identification information indicating the object. An output unit outputs to a display device, superimposed on a map indicating a predetermined location, at least one of the following information regarding the loading status of the object determined by the loading status determination unit: the loading position of the object, the number of stacking layers of the object at the loading position, and the height of the highest stacked object; An information processing device characterized by comprising: <2> The output unit outputs the position of at least one of the worker and the moving object on the map. Characterized by <1> The information processing device described above. <3> The output unit outputs the loading position of the object, the number of stacking layers of the object at the loading position, and the height of the highest stacked object, superimposed on a map indicating the predetermined location, so that they can be displayed on a display device. Characterized by <1> or <2> The information processing device described above. <4> The output unit outputs the method of loading the object on the map indicating the base in a manner that allows for identification. Characterized by <1> or <3> An information processing device as described in any one of the following. <5> The output unit outputs a frame that mimics the rack's casing in the area on the map indicating the base where the object is stacked on the rack. Characterized by <4> The information processing device described above. <6> The output unit outputs information relating to the object at the selected loading location when at least one loading location of the object on the map indicating the base is selected. Characterized by <1> or <5> An information processing device as described in any one of the following. <7> The output unit outputs the loading position of the object, at least the loading position of the object adjacent to the worker or the mobile body, in a manner that can be displayed. Characterized by <1> or <6> An information processing device as described in any one of the following. <8> The output unit outputs the detachment height, which is the height of the tallest stacked object, where the bottom surface of the tallest stacked object is detached from the moving body. Characterized by <1> or <7> An information processing device as described in any one of the following. <9> A display system including an information processing device and a display device that is communicably connected to the information processing device via a network, The aforementioned information processing device is A loading status determination unit determines the loading status of objects by a worker or a mobile body controlled by the worker within a predetermined location, based on captured images obtained from the mobile body within the predetermined location and identification information indicating the object. An output unit outputs to the display device, superimposed on a map indicating the predetermined location, at least one of the following information regarding the loading status of the object determined by the loading status determination unit: the loading position of the object, the number of stacking layers of the object at the loading position, and the height of the highest stacked object; Equipped with, The aforementioned display device is A display unit that superimposes and displays on a map indicating the predetermined location at least one of the following: the loading position of the object, the number of stacking layers of the object at the loading position, and the height of the highest stacked object. A display system characterized by comprising the following features. <10> A display method in an information processing device connected to a display device, A loading status determination step that determines the loading status of objects by a worker or a mobile body controlled by the worker within a predetermined location, based on captured images obtained from the mobile body within the predetermined location and identification information indicating the object, Output step: Outputs to the display device, superimposed on a map indicating the predetermined location, at least one of the following information regarding the loading status of the object determined by the loading status determination step: the loading position of the object, the number of stacking layers of the object at the loading position, and the height of the highest stacked object; A method of display characterized by including <11> Computers, A loading status determination unit determines the loading status of objects by a worker or a mobile body controlled by the worker within a predetermined location, based on captured images obtained from the mobile body within the predetermined location and identification information indicating the object. An output unit outputs to a display device, superimposed on a map indicating a predetermined location, at least one of the following information regarding the loading status of the object determined by the loading status determination unit: the loading position of the object, the number of stacking layers of the object at the loading position, and the height of the highest stacked object; A program designed to function as such. [Explanation of Symbols]

[0197] 1 Display System 10, 70 Mobile 30 Object 50 Information Processing Devices 57 Output section 61 Loading status determination unit 90, 700 display device 93, 707 Display section [Prior art documents] [Patent Documents]

[0198] [Patent Document 1] International Publication No. 2006 / 114812

Claims

1. A loading status determination unit determines the loading status of objects by a worker or a mobile body controlled by the worker within a predetermined location, based on captured images obtained from the mobile body within the predetermined location and identification information indicating the object. An output unit outputs to a display device, superimposed on a map indicating a predetermined location, at least one of the following information regarding the loading status of the object determined by the loading status determination unit: the loading position of the object, the number of stacking layers of the object at the loading position, and the height of the highest stacked object; An information processing device characterized by comprising:

2. The output unit outputs the position of at least one of the worker and the moving object on the map. The information processing apparatus according to feature 1.

3. The output unit outputs the loading position of the object, the number of stacking layers of the object at the loading position, and the height of the highest stacked object, superimposed on a map indicating the predetermined location, so that they can be displayed on a display device. The information processing apparatus according to feature 1.

4. The output unit outputs the method of loading the object on the map indicating the base in a manner that allows for identification. The information processing apparatus according to feature 1.

5. The output unit outputs a frame that mimics the rack's casing in the area on the map indicating the base where the object is stacked on the rack. The information processing apparatus according to feature 4.

6. The output unit outputs information relating to the object at the selected loading location when at least one loading location of the object on the map indicating the base is selected. The information processing apparatus according to feature 1.

7. The output unit outputs the loading position of the object, at least the loading position of the object adjacent to the worker or the mobile body, in a manner that can be displayed. The information processing apparatus according to feature 1.

8. The output unit outputs the detachment height, which is the height of the tallest stacked object, where the bottom surface of the tallest stacked object is detached from the moving body. The information processing apparatus according to feature 1.

9. A display system including an information processing device and a display device that is communicably connected to the information processing device via a network, The aforementioned information processing device is A loading status determination unit determines the loading status of objects by a worker or a mobile body controlled by the worker within a predetermined location, based on captured images obtained from the mobile body within the predetermined location and identification information indicating the object. An output unit outputs to the display device, superimposed on a map indicating the predetermined location, at least one of the following information regarding the loading status of the object determined by the loading status determination unit: the loading position of the object, the number of stacking layers of the object at the loading position, and the height of the highest stacked object; Equipped with, The aforementioned display device is A display unit that superimposes and displays on a map indicating the predetermined location at least one of the following: the loading position of the object, the number of stacking layers of the object at the loading position, and the height of the highest stacked object. A display system characterized by comprising the following features.

10. A display method in an information processing device connected to a display device, A loading status determination step that determines the loading status of objects by a worker or a mobile body controlled by the worker within a predetermined location, based on captured images obtained from the mobile body within the predetermined location and identification information indicating the object, Output step: Outputs to the display device, superimposed on a map indicating the predetermined location, at least one of the following information regarding the loading status of the object determined by the loading status determination step: the loading position of the object, the number of stacking layers of the object at the loading position, and the height of the highest stacked object; A method of display characterized by including

11. Computers, A loading status determination unit determines the loading status of objects by a worker or a mobile body controlled by the worker within a predetermined location, based on captured images obtained from the mobile body within the predetermined location and identification information indicating the object. An output unit outputs to a display device, superimposed on a map indicating a predetermined location, at least one of the following information regarding the loading status of the object determined by the loading status determination unit: the loading position of the object, the number of stacking layers of the object at the loading position, and the height of the highest stacked object; A program designed to function as such.

Images