Information processing device, information processing system, and program

Abstract

To improve convenience of goods management in a warehouse where goods are arranged in a free location.SOLUTION: An information processing device according to an embodiment includes a first acquisition unit, a first recognition unit, a second recognition unit, a second acquisition unit, and an output unit. The first acquisition unit acquires an image from a camera mounted on a moving object that transports goods. The first recognition unit recognizes a symbol attached to the goods from within the image and acquires identification information capable of identifying the goods from the symbol. The second recognition unit recognizes an action that the moving object performs to the goods based on a change in size of the symbol recognized from within the image by the first recognition unit. The second acquisition unit acquires position information on the goods in recognizing an action of the moving object. The output unit outputs correspondence information that associates the identification information on the goods indicated by the symbol with the position information on the goods according to an action content recognized by the second recognition unit.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing device, an information processing system, and a program. [Background technology]

[0002] Conventionally, in a warehouse where items are freely located, a system is known that acquires location information of the item and associates the acquired location information with the item's identification information to manage the location of the item in the warehouse. In such a system, the system recognizes that an item has been picked up (loaded) and that an item has been unloaded by receiving input from a user. Summary of the Invention [Problem to be solved by the invention]

[0003] An object of the present invention is to improve the convenience of item management in a warehouse where items are arranged in a free location. [Means for solving the problem]

[0004] The information processing device according to the embodiment includes a first acquisition unit, a first recognition unit, a second recognition unit, a second acquisition unit, and an output unit. The first acquisition unit acquires an image from a camera mounted on a moving object that transports an item. The first recognition unit recognizes a symbol attached to the item from within the image and acquires identification information capable of identifying the item from the symbol. The second recognition unit recognizes an action performed by the moving object on the item based on a size change of the symbol recognized from within the image by the first recognition unit. The second acquisition unit acquires position information of the item when the action of the moving object is recognized. The output unit outputs correspondence information that associates the identification information of the item indicated by the symbol with the position information of the item according to the content of the action recognized by the second recognition unit. [Brief description of the drawings]

[0005] [Figure 1]FIG. 1 is a block diagram showing an example of the configuration of an information processing system according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of the server device according to the first embodiment. [Diagram 3] FIG. 3 is a block diagram illustrating an example of a functional configuration of the server device according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a display screen generated by the server device according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of a display screen generated by the server device according to the first embodiment. [Figure 6] FIG. 6 is a block diagram showing an example of a hardware configuration of the camera device according to the first embodiment. [Figure 7] FIG. 7 is a schematic diagram showing an example of the installation position of the camera in the forklift according to the first embodiment. [Figure 8] FIG. 8 is a schematic diagram showing an example of the installation position of the camera in the forklift according to the first embodiment. [Figure 9] FIG. 9 is a block diagram showing an example of the functional configuration of the camera device according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a process for recognizing that a forklift has picked up an article according to the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a process for recognizing that a forklift has picked up an article according to the first embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of a process for recognizing that a forklift has picked up an article according to the first embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of a process for recognizing that a forklift has picked up an article according to the first embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of a process for recognizing that a forklift has picked up an article according to the first embodiment. [Figure 15]FIG. 15 is a diagram illustrating an example of a process for recognizing that a forklift has picked up an article according to the first embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of a process for recognizing that an article has been unloaded from a forklift according to the first embodiment. [Figure 17] FIG. 17 is a diagram illustrating an example of a process for recognizing that an article has been unloaded from a forklift according to the first embodiment. [Figure 18] FIG. 18 is a diagram illustrating an example of the location acquisition process according to the first embodiment. [Figure 19] FIG. 19 is a block diagram illustrating an example of a hardware configuration of the worker terminal according to the first embodiment. [Figure 20] FIG. 20 is a block diagram illustrating an example of a functional configuration of the worker terminal according to the first embodiment. [Figure 21] FIG. 21 is a sequence diagram illustrating an example of processing executed by the information processing system according to the first embodiment. [Figure 22] FIG. 22 is a sequence diagram illustrating an example of processing executed by the information processing system according to the first embodiment. [Figure 23] FIG. 23 is a sequence diagram illustrating an example of processing executed by the information processing system according to the first embodiment. [Figure 24] FIG. 24 is a block diagram showing an example of the configuration of an information processing system according to the second embodiment. [Diagram 25] FIG. 25 is a block diagram illustrating an example of a hardware configuration of a server device according to the second embodiment. As shown in FIG. [Figure 26] FIG. 26 is a block diagram illustrating an example of a functional configuration of a server device according to the second embodiment. As shown in FIG. [Figure 27] FIG. 27 is a block diagram showing an example of a hardware configuration of a camera device according to the second embodiment. [Figure 28] FIG. 28 is a block diagram showing an example of the functional configuration of the camera device according to the second embodiment. As shown in FIG. [Figure 29]FIG. 29 is a block diagram showing an example of a hardware configuration of a drive control device according to the second embodiment. [Diagram 30] FIG. 30 is a block diagram showing an example of a functional configuration of the drive control device according to the second embodiment. [Diagram 31] FIG. 31 is a sequence diagram illustrating an example of processing executed by the information processing system according to the second embodiment. [Diagram 32] FIG. 32 is a sequence diagram illustrating an example of processing executed by the information processing system according to the second embodiment. [Diagram 33] FIG. 33 is a sequence diagram illustrating an example of processing executed by the information processing system according to the second embodiment. [Diagram 34] FIG. 34 is a schematic diagram showing an example of the installation positions of the camera and the gyro sensor in the forklift according to the first modification. [Diagram 35] FIG. 35 is a schematic diagram showing an example of the installation positions of the camera and the gyro sensor in the forklift according to the first modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] Hereinafter, embodiments will be described with reference to the drawings.

[0007] (First embodiment) 1 is a diagram showing an example of the overall configuration of an information processing system S according to the first embodiment. The information processing system S includes a server device 1, a camera device 2, and an operator terminal 3. The server device 1, the camera device 2, and the operator terminal 3 are connected to each other via a network N so as to be able to communicate with each other.

[0008] 1, one each of the server device 1, the camera device 2, and the worker terminal 3 are connected, but the number of devices is not limited to this. For example, the information processing system S may include multiple camera devices 2 and multiple worker terminals 3.

[0009] The server device 1 is a device that manages the placement positions of items in a warehouse where items are placed at a free location. Here, the warehouse means a facility for storing tangible items, such as a warehouse in a logistics center. The configuration of the server device 1 will be described below with reference to Figs. 2 to 5.

[0010] First, a hardware configuration of the server device 1 will be described. Fig. 2 is a block diagram showing an example of a hardware configuration of the server device 1 according to the first embodiment. The server device 1 includes a CPU (Central Processing Unit) 41, a RAM (Random Access Memory) 42, a ROM (Read Only Memory) 43, a storage 44, a communication I / F (Interface) 45, and a user I / F 46. These units 41 to 46 are connected to each other via a data bus.

[0011] The CPU 41 performs overall control of the server device 1. For example, the CPU 41 uses the RAM 42 as a working area and executes various processes by executing programs stored in the ROM 43, the storage 44, etc. The storage 44 is a non-volatile memory in which data can be written and rewritten.

[0012] The communication I / F 45 is a device for establishing a predetermined type of communication with the camera device 2 and the worker terminal 3. The communication type is not particularly limited, but examples include those conforming to standards such as Ethernet (registered trademark) and IEEE802.11.

[0013] The user I / F 46 is a device that enables receiving input from an operator of the server device 1, such as a system administrator, and outputting information to the operator. Examples of the user I / F 46 include a keyboard, a pointing device, a display, and a speaker.

[0014] 2 is merely an example, and the hardware configuration of the server device 1 is not limited to the above. The server device 1 may be, for example, a system in which a plurality of computers operate in cooperation with each other, a system using cloud computing, or the like.

[0015] Next, the functional configuration of the server device 1 will be described. Fig. 3 is a block diagram showing an example of the functional configuration of the server device 1 according to the first embodiment. In this embodiment, the CPU 41 of the server device 1 executes a program stored in the storage 44 or the like to realize various functions of the server device 1. The CPU 41 has a communication control unit 411, a storage control unit 412, an instruction unit 413, and a screen generation unit 414. The storage 44 has management information 441.

[0016] The communication control unit 411 controls transmission and reception of information between the camera device 2 and the worker terminal 3. For example, the communication control unit 411 receives various notifications from the camera device 2 via the communication I / F 45, such as a notification that an item is being moved, a notification that arrival of an item has been completed, a notification that shipment of an item has been completed, etc. Similarly, for example, the communication control unit 411 transmits instructions for arrival of an item, instructions for shipping an item, instructions for moving an item, etc. to the worker terminal 3.

[0017] The storage control unit 412 performs control to store various information in the storage device. For example, the storage control unit 412 performs control to associate an item name with a location and store it in the storage 44 as management information 441. The location is an example of position information.

[0018] The instruction unit 413 generates instructions for workers working in the warehouse. For example, when an item is unloaded at a receiving berth, which is a space for receiving items, the instruction unit 413 generates a receiving instruction that instructs a worker to perform receiving processing of the item. The generated receiving instruction is transmitted to the worker terminal 3 by the communication control unit 411.

[0019] Furthermore, for example, when the instruction unit 413 receives an instruction to ship an item, the instruction unit 413 generates a shipping instruction to instruct a worker to perform a shipping process to move the item stored on the shelf to a shipping berth, which is a space for shipping the item. The generated shipping instruction is transmitted to the worker terminal 3 by the communication control unit 411.

[0020] Furthermore, for example, when the instruction unit 413 receives an instruction to move an item, the instruction unit 413 generates a moving instruction to instruct a worker to perform a moving process to move the item stored in a specific area to another area. The generated shipping instruction is transmitted to the worker terminal 3 by the communication control unit 411.

[0021] The screen generating unit 414 generates various screens. Such a screen may be image data representing the screen itself, or may be content data for displaying the screen on a display destination device. For example, the screen generating unit 414 generates an item list display screen when it receives an instruction to display an item list display screen via the worker terminal 3. The item list display screen displays the item names and locations (storage positions) of all items stored in a warehouse managed by the server device 1. The generated item list display screen is provided to, for example, the worker terminal 3, and is displayed on a display 67 described below.

[0022] As an example, the screen generation unit 414 generates an item list display screen based on management information 441 described later. Here, Fig. 4 is a diagram showing an example of a screen generated by the server device 1. Fig. 4 shows an example of an item list display screen DA.

[0023] As shown in Fig. 4, the item list display screen DA includes a location display field AR. The location display field AR shows an association between an item name and a location indicating the position in the warehouse where the item is stored. As an example, the first line of the location display field in Fig. 4 shows that an item with an item name of "12345" is stored at a location indicated by the warehouse location "A-12-3."

[0024] Furthermore, for example, the screen generating unit 414 generates an item search screen when an instruction to display an item search screen is received via the worker terminal 3. The item search screen is a screen for searching for items stored in a warehouse managed by the server device 1. The generated item search screen is provided to, for example, the worker terminal 3 and displayed on a display 67 described below.

[0025] As an example, the screen generation unit 414 generates an item search screen for searching the information registered in the management information 441 using an item name as a keyword. Here, Fig. 5 is a diagram showing an example of a screen generated by the server device 1. Fig. 5 shows an example of an item search screen DB. As shown in Fig. 5, the item search screen DB includes an input form IF, a search button SB, and a result display field DR.

[0026] The input form IF is a form for accepting input of an item name. A searcher, such as a worker, can search for the location of the item by inputting the item name into the input form IF. The search button SB is a button for confirming the item name entered into the input form IF. A searcher can search for the location of the item by inputting the item name into the input form IF and pressing the search button SB. The result display field DR is a display field for displaying the search results.

[0027] As an example, the result display field DR of the search screen DB in Fig. 5 shows an example of a search result when a searcher enters "12345" in the input form IF and presses the search button SB. The result display field DR in Fig. 5 shows that an item with the item name "12345" is stored at the location indicated by the warehouse location "A-12-3."

[0028] The item list display screen DA and the item search screen DB may be displayed on a display device of a device other than the worker terminal 3. For example, they may be displayed on a display of a search terminal (not shown).

[0029] The screen generated by the screen generating unit 414 is not limited to the above. For example, a screen for searching for items stored at a location from the location may be generated. Various display screens may be generated directly by the worker terminal 3, etc.

[0030] The management information 441 is information for managing items stored in the warehouse. For example, the management information 441 stores an item name and a location where the item is currently stored in association with each other. The management information 441 is updated by the storage control unit 412 in response to the executed processing.

[0031] The program for causing the server device 1 to execute the processes for realizing the above-mentioned functions can be provided by recording it in an installable or executable file format on a computer-readable recording medium such as a CD (Compact Disc)-ROM, a flexible disk (FD), a CD-R (Recordable), a DVD (Digital Versatile Disk), etc. The program may also be provided or distributed via a network such as the Internet.

[0032] Returning to Fig. 1, the description will continue. The camera device 2 is a device that takes pictures of the surroundings of a forklift placed in a warehouse with a camera and supports the management of goods using the taken images. For example, the camera device 2 is mounted on a forklift. The configuration of the camera device 2 will be described below with reference to Figs. 6 to 18.

[0033] First, a description will be given of the hardware configuration of the camera device 2. Fig. 6 is a block diagram showing an example of the hardware configuration of the camera device according to the first embodiment. The camera device 2 includes a controller 50, a first camera 56, and a second camera 57.

[0034] The controller 50 includes a CPU 51, a RAM 52, a ROM 53, a storage 54, and a communication I / F 55. These units 51 to 55, a first camera 56, and a second camera 57 are connected to each other via a data bus. The controller 50 and the first camera 56, and the controller 50 and the second camera 57 may be connected by a wireless communication standard such as Bluetooth (registered trademark). The controller 50 is an example of an information processing device in the present disclosure.

[0035] The CPU 51 comprehensively controls the camera device 2. For example, the CPU 51 uses the RAM 52 as a working area and executes various processes by executing programs stored in the ROM 53, the storage 54, etc. The storage 54 is a non-volatile memory to which data can be written and rewritten. The communication I / F 55 is a device for establishing communication with the server device 1 and the worker terminal 3 in a predetermined format.

[0036] The first camera 56 is a camera for recognizing a character string representing the name of an item to be processed. The character string representing the name of the item is an example of a symbol. The second camera 57 is a camera for acquiring a location. The first camera 56 and the second camera 57 are provided on a forklift. The forklift is an example of a moving body that transports items. An example of the installation positions of the first camera 56 and the second camera 57 will be described below.

[0037] Figures 7 and 8 are schematic diagrams showing an example of the installation position of the camera in the forklift according to the first embodiment. Figure 7 shows an example of the forklift as viewed from the side, and Figure 8 shows an example of the forklift as viewed from the front.

[0038] 7 and 8, the forklift 5 includes a fork 71, a backrest 72, a mast 73, a head guard 74, driving wheels 75, and steering wheels 76. The fork 71 is a component used to lift an object. The fork 71 is also called a claw.

[0039] The backrest 72 is a component that prevents an object lifted by the forks 71 from falling. The mast 73 is a component that moves the forks 71 of the forklift 5 up and down. By sliding the mast 73 up and down, an object can be picked up or put down. The head guard 74 is a component that protects the operator of the forklift 5 from falling objects.

[0040] The driving wheels 75 are wheels that directly receive the power generated by the prime mover, convert it into rotational motion, and move using the rotational motion. The steering wheels 76 are wheels for steering. In this embodiment, the forklift 5 travels within the warehouse by controlling the driving wheels 75 and steering wheels according to the operation of the operator.

[0041] In this embodiment, the first camera 56 is provided near the center of the backrest 72. The second camera 57 is provided above the mast 73. In this embodiment, the forklift 5 is provided with two cameras, the first camera 56 and the second camera 57, but the number of cameras may be one. In this case, recognition of a character string indicating the name of the target item and recognition of the location are performed from an image captured by one camera.

[0042] In this embodiment, the controller 50 is built into the first camera 56 or the second camera 57, but the controller 50 may be provided separately from the first camera 56 and the second camera 57, and may be provided near the driver's seat of the forklift 5. Also, a PC (Personal Computer) or the like installed in a location other than the forklift 5 may be wirelessly connected to the first camera 56 and the second camera 57, and the PC may function as the controller 50.

[0043] Next, a functional configuration of the camera device 2 will be described. Fig. 9 is a block diagram showing an example of the functional configuration of the camera device 2 according to the first embodiment. In this embodiment, a CPU 51 of the camera device 2 executes a program stored in a storage 54 or the like to realize various functions of the camera device 2. The CPU 51 has a communication control unit 511, a first acquisition unit 512, a first recognition unit 513, a second recognition unit 514, a third recognition unit 515, and a second acquisition unit 516.

[0044] The communication control unit 511 controls the transmission and reception of information between the server device 1 and the worker terminal 3. For example, the communication control unit 511 transmits various notifications to the server device 1 via the communication I / F 55, such as a notification that an item is being moved, a notification that the arrival of an item has been completed, a notification that the shipment of an item has been completed, etc. Similarly, for example, the communication control unit 511 transmits a notification to the worker terminal 3 that an item has been picked up or dropped off.

[0045] The first acquisition unit 512 acquires images from a camera mounted on the forklift 5. For example, the first acquisition unit 512 acquires images captured by the first camera 56 and the second camera 57 via the communication I / F 55 in real time.

[0046] The first recognition unit 513 recognizes a symbol attached to an item from within the image, and acquires identification information capable of identifying the item from the symbol. For example, the first recognition unit 513 recognizes a character string describing the item name of the target item, which is included in the image acquired by the first acquisition unit 512, as a symbol, and acquires the character string as an item name capable of identifying the target item.

[0047] As an example, a piece of paper on which a character string representing the name of each item is written is attached to each of all items stored in the warehouse. In this embodiment, the first recognition unit 513 analyzes the image captured by the first camera 56 and acquired by the first acquisition unit 512 using a known image recognition technique. When the first recognition unit 513 recognizes that the image contains a symbol (in this case, a character string describing the name of the item), it acquires the character string as the name of the item.

[0048] In this embodiment, the first recognition unit 513 recognizes a character string describing the product name as a symbol, but the first recognition unit 513 may recognize a barcode, a two-dimensional code, or the like as a symbol.

[0049] The second recognition unit 514 recognizes an action performed by the forklift 5 on an object based on a change in size of a symbol included in images acquired successively over time by the first acquisition unit 512. For example, the second recognition unit 514 recognizes that the forklift 5 has picked up a target object based on a change in size of a symbol when the forklift 5 approaches the target object. Picking up the target object is an example of a first action.

[0050] For example, the second recognition unit 514 recognizes that the forklift 5 has picked up the target item based on the size of the character string acquired by the first recognition unit 513 as the item name of the target item in the image captured by the first camera 56 and acquired by the first acquisition unit 512. When a barcode, a two-dimensional code, or the like is recognized as a symbol, the second recognition unit 514 recognizes that the target item has been picked up based on the size of the barcode, the two-dimensional code, or the like.

[0051] 10 and 11, a process of recognizing that the forklift 5 has picked up a target item by the second recognition unit 514 will be described below. Fig. 10 is a diagram for explaining an example of a process of recognizing that the forklift 5 has picked up an item.

[0052] Fig. 10 shows a forklift 5 approaching an item ME to pick it up. In Fig. 10, the item ME is loaded on a pallet PT, and a paper PR having a character string indicating the item name of the item ME is attached to the item ME.

[0053] The forklift 5 picks up the item ME by inserting the forks 71 into the pallet PT. At this time, the size of the character string written on the paper PR in the image captured by the first camera 56 provided on the backrest 72 of the forklift 5 becomes larger as the forklift 5 approaches the item ME.

[0054] Furthermore, when the forklift 5 reaches a position where it can pick up the item ME, it does not approach the item ME any further and raises the forks 71. For this reason, the size of the character string written on the paper PR in the image captured by the first camera 56 does not change from the time the forklift 5 starts picking up the item ME until the item ME is lowered and the forks 71 start to be removed from the pallet PT.

[0055] For this reason, the second recognition unit 514 recognizes that the forklift 5 has picked up the target item when the size of the character string does not change even after a predetermined period of time has elapsed. Here, Fig. 11 is a diagram illustrating an example of a process for recognizing that the forklift 5 has picked up an item. Fig. 11 shows the change over time in the size of the character string WD written on the paper PR in the image captured by the first camera 56.

[0056] The horizontal axis of Fig. 11 represents the passage of time. In Fig. 11, it is assumed that the forklift 5 starts, from time t0, an operation to insert the forks 71 into the pallet PT on which the goods ME are loaded, in accordance with the operation of the worker who is the operator of the forklift 5. Similarly, it is assumed that the forklift 5 starts, from time t1, an operation to pick up the goods ME loaded on the pallet PT into which the forks 71 are inserted.

[0057] From time t0 to time t1, the forklift 5 approaches the item ME. Therefore, as shown in Fig. 11, the size of the character string WD written on the paper PR in the image captured by the first camera 56 increases over time.

[0058] On the other hand, after time t1, the forklift 5 does not move any closer to or any further away from the item ME until it starts to unload the item ME and pull the forks 71 out of the pallet PT. Therefore, as shown in Fig. 11, after time t1, the size of the character string WD written on the paper PR in the image captured by the first camera 56 does not change.

[0059] As an example, the second recognition unit 514 recognizes the time when the size of the character string WD written on the paper PR stops changing (time t1 in FIG. 11) based on the image captured in real time by the first camera 56. Then, when the second recognition unit 514 recognizes that a predetermined time has passed since the time when the size of the character string WD written on the paper PR stopped changing (when the second recognition unit 514 recognizes that time t2 has passed in FIG. 11), it recognizes that the target object has been picked up.

[0060] However, even if the forklift 5 does not pick up the item ME, if the forklift 5 stops for some reason while the first camera 56 is photographing the character string WD written on the paper PR, the second recognition unit 514 may erroneously recognize that the target item has been picked up. Below, several methods for reducing the possibility of such erroneous recognition will be described with reference to Figs. 12 to 15.

[0061] First, a method using the size of a character string in an image will be described. For example, when the size of the character string in an image captured by the first camera 56 does not change for a predetermined time and the size of the character string in the image exceeds a predetermined size, the second recognition unit 514 recognizes that the target item has been picked up.

[0062] Fig. 12 is a diagram for explaining an example of a process for recognizing that an article has been picked up by the forklift 5. Fig. 12 (1) shows a case where the size of the character string WD in the image is smaller than a predetermined size. In this case, the second recognition unit 514 does not recognize that the target article has been picked up even if the size of the character string WD in the image captured by the first camera 56 has not changed for a predetermined period of time.

[0063] 12(2) shows a case where the size of the character string WD in the image is a predetermined size. In this case, the second recognition unit 514 recognizes that the target item has been picked up if the size of the character string WD in the image captured by the first camera 56 does not change for a predetermined period of time.

[0064] Next, a method using a known object recognition technology will be described. For example, when the size of a character string in an image captured by the first camera 56 does not change for a predetermined time and an object called "paper" is recognized in the image, the second recognition unit 514 recognizes that the target item has been picked up.

[0065] Fig. 13 is a diagram for explaining an example of a process for recognizing that an object has been picked up by the forklift 5. In Fig. 13, a frame OR indicates that paper PR has been recognized as an object. In other words, Fig. 13 (3) indicates a case in which the object "paper" is not recognized in the image captured by the first camera 56. In this case, the second recognition unit 514 does not recognize that the target object has been picked up even if the size of the character string WD in the image captured by the first camera 56 has not changed for a predetermined period of time.

[0066] 13(4) shows a case where an object called "paper" is recognized in an image captured by the first camera 56. In this case, the second recognition unit 514 recognizes that the target article has been picked up if the size of the character string WD in the image captured by the first camera 56 does not change for a predetermined period of time.

[0067] Next, a method using the position of a character string in an image will be described. For example, when the size of the character string in the image captured by the first camera 56 does not change for a predetermined time and the character string is present at a predetermined position in the image, the second recognition unit 514 recognizes that the target item has been picked up.

[0068] Fig. 14 is a diagram for explaining an example of a process for recognizing that an item ME has been picked up by a forklift 5. In Fig. 14, an image PI represents an image captured by the first camera 56. (5) in Fig. 14 represents a case in which the position of the character string WD in the image PI is not in a predetermined position (in this example, the center of the image). In this case, the second recognition unit 514 does not recognize that the target item has been picked up even if the size of the character string WD in the image captured by the first camera 56 has not changed for a predetermined period of time.

[0069] 14(6) shows a case where the character string WD in the image PI is at a predetermined position. In this case, the second recognition unit 514 recognizes that the target item has been picked up if the size of the character string WD in the image captured by the first camera 56 does not change for a predetermined period of time.

[0070] Next, a method of using the distance from the forklift 5 will be described. For example, an optical sensor LS having a light source and a light receiving unit is provided on the backrest 72 of the forklift 5. The light source of the distance measuring sensor LS irradiates a laser light LR onto the article ME. The light receiving unit of the optical sensor LS receives the reflected light of the laser light LR and detects the distance to the article ME based on the amount of light received.

[0071] As an example, the second recognition unit 514 recognizes that the target item has been picked up if the size of the character string in the image captured by the first camera 56 does not change for a predetermined time and the distance to the item ME detected by the optical sensor LS is within a predetermined range.

[0072] The above-mentioned several methods may be used alone or in combination. In addition, when the forklift 5 is equipped with a sensor that detects lifting of an object, the detection result of the sensor may be used to reduce the possibility of erroneous recognition by the second recognition unit 514.

[0073] Returning to FIG. 9, the explanation will be continued. The third recognition unit 515 recognizes the action of the forklift 5 unloading the target item. The third recognition unit 515 is an example of the second recognition unit. For example, the third recognition unit 515 recognizes that the forklift 5 has unloaded the target item based on the size of the character string acquired by the first recognition unit 513 as the item name of the target item in the image captured by the first camera 56 acquired by the first acquisition unit 512. Unloading the target item is an example of the second action.

[0074] 16 and 17, a process of recognizing that the forklift 5 has unloaded the target article by the third recognition unit 515 will be described below. Fig. 16 is a diagram for explaining an example of a process of recognizing that the article has been unloaded from the forklift 5.

[0075] Fig. 16 shows a state in which the forklift 5 lowers the forks 71 to unload the item ME, and then moves away from the item ME to remove the forks 71 from the pallet PT. In Fig. 16, the item ME is loaded on the pallet PT, and a piece of paper PR having a character string indicating the item name of the item ME is attached to the item ME.

[0076] After lowering the forks 71, the forklift 5 backs up and removes the forks 71 from the pallet PT, thereby unloading the item ME from the forklift 5. At this time, the size of the characters written on the paper PR in the image captured by the first camera 56 provided on the backrest 72 of the forklift 5 becomes smaller as the forklift 5 moves away from the item ME.

[0077] For this reason, the third recognition unit 515 recognizes that the forklift 5 has unloaded the target item when the rate of reduction in the size of the character string exceeds a predetermined rate, based on the size of the character string at the time when the size of the character string starts to reduce. Here, Fig. 17 is a diagram for explaining an example of a process for recognizing that an item has been unloaded from the forklift 5. Fig. 17 shows a time-series change in the size of the character string WD written on the paper PR in an image captured by the first camera 56.

[0078] The horizontal axis of Fig. 17 represents the passage of time. In Fig. 17, the forklift 5 lowers the forks 71 in accordance with the operation of the worker, and starts to move away from the article ME from time t3 in order to remove the forks 71 from the pallet PT on which the article ME is loaded.

[0079] Until time t3, the palette PT remains inserted in the fork 71. For this reason, as shown in Fig. 17, the size of the character string WD written on the paper PR in the image captured by the first camera 56 does not change until time t3 has elapsed.

[0080] On the other hand, after time t3, the forklift 5 moves away from the item ME in order to remove the forks 71 from the pallet PT. Therefore, as shown in Fig. 17, after time t3, the size of the character string WD written on the paper PR in the image captured by the first camera 56 decreases.

[0081] As an example, the third recognition unit 515 recognizes the time when the size of the character string WD written on the paper PR starts to decrease (time t3 in FIG. 17) based on the image captured by the first camera 56 acquired in real time. Then, when the third recognition unit 515 recognizes that the reduction rate of the size of the character string WD written on the paper PR exceeds a predetermined rate based on the size of the character string WD at that time (when it recognizes that time t4 has passed in FIG. 17), it recognizes that the target item has been put down.

[0082] In this embodiment, the second recognition unit 514 and the third recognition unit 515 are separate functional units, but the functions of the second recognition unit 514 and the third recognition unit 515 may be integrated into one functional unit.

[0083] Returning to Fig. 9, the description will be continued. The second acquisition unit 516 acquires position information of the object when it recognizes the operation of the forklift 5. For example, the second acquisition unit 516 acquires position information indicating the position of the target object based on the position of the forklift 5 at the time when the target object is unloaded.

[0084] As an example, the second acquisition unit 516 acquires the location at the time when the third recognition unit 515 recognizes that the article has been dropped off based on the two-dimensional code for location recognition included in the image captured by the second camera 57 acquired by the first acquisition unit 512. The two-dimensional code is an example of a code symbol. The location acquisition process will be described below with reference to FIG. 18.

[0085] Fig. 18 is a diagram for explaining an example of location acquisition processing. In Fig. 18, an area TA for storing items is formed by shelves SF. An item ME is placed in the area TA, and a piece of paper PR with the character string "12345" written thereon is attached to the front of the item ME.

[0086] In addition, a two-dimensional code TQ encoding the location of area TA is placed in the upper left of area TA. In addition, a two-dimensional code NQ encoding the location of the storage area adjacent to area TA on the right is placed in the upper right of area TA. Note that a barcode may be used instead of the two-dimensional code.

[0087] For example, when the third recognition unit 515 recognizes that the target item has been put down based on the size of the character string WD written on the paper PR, the second acquisition unit 516 reads the two-dimensional code present in the upper left area of ​​the image captured by the second camera 57 and acquired in real time by the first acquisition unit 512. As an example, in Fig. 18, the second acquisition unit 516 reads the two-dimensional code TQ and acquires the location corresponding to the two-dimensional code.

[0088] The location where the target item was dropped off, that is, the location indicating the location of the target item, acquired as described above, is transmitted by the communication control unit 511 to the server device 1 together with the item name acquired by the first recognition unit 513 as the name of the target item. In this case, the communication control unit 511 is an example of an output unit. The location and item name received by the communication control unit 411 of the server device 1 are associated with each other and stored as management information 441 by the storage control unit 412.

[0089] The second acquisition unit 516 may continuously acquire the location while the forklift 5 is moving.

[0090] As an example, when a two-dimensional code is included in the image captured by the second camera 57, the second acquisition unit 516 reads the two-dimensional code regardless of whether the target article has been unloaded. Also, when there are multiple two-dimensional codes in the image captured by the second camera 57, the second acquisition unit 516 reads the code located further to the left in the image with priority. This makes it possible to constantly grasp the current location of the forklift 5.

[0091] The program for causing the camera device 2 to execute the processes for realizing the above-mentioned functions can be provided by recording it in an installable or executable format on a computer-readable recording medium such as a CD-ROM, FD, CD-R, DVD, etc. The program may also be provided or distributed via a network such as the Internet.

[0092] Returning to Fig. 1, the description will continue. The worker terminal 3 is a device that supports the work of a worker who works in a warehouse managed by the server device 1. The configuration of the worker terminal 3 will be described below with reference to Figs. 19 and 20.

[0093] First, the hardware configuration of the worker terminal 3 will be described. Fig. 19 is a block diagram showing an example of the hardware configuration of the worker terminal 3 according to the first embodiment. The worker terminal 3 includes a CPU 61, a RAM 62, a ROM 63, a storage 64, a communication I / F 65, an input I / F 66, and a display 67. These units 61 to 67 are connected to each other via a data bus.

[0094] The CPU 61 comprehensively controls the worker terminal 3. For example, the CPU 61 uses the RAM 62 as a working area and executes various processes by executing programs stored in the ROM 63, the storage 64, etc. The storage 64 is a non-volatile memory in which data can be written and rewritten.

[0095] The communication I / F 65 is a device for establishing a predetermined type of communication with the server device 1 and the camera device 2. The input I / F 66 is a device that enables input from a worker to be received. An example of the input I / F 66 is a touch panel. The display 67 is a display device that displays various information. For example, the display 67 displays an item list display screen, an item search screen, etc., generated by the server device 1.

[0096] Next, a functional configuration of the worker terminal 3 will be described. Fig. 20 is a block diagram showing an example of the functional configuration of the worker terminal 3 according to the first embodiment. In this embodiment, the CPU 61 of the worker terminal 3 executes programs stored in the storage 64 or the like to realize various functions of the worker terminal 3. The CPU 61 has a communication control unit 611, a display control unit 612, and a collation unit 613.

[0097] The communication control unit 611 controls transmission and reception of information between the server device 1 and the camera device 2. For example, the communication control unit 611 receives various instructions, such as an instruction to receive an item, an instruction to ship an item, an instruction to move an item, etc., from the server device 1 via the communication I / F 65. Similarly, for example, the communication control unit 611 receives various notifications, such as a notification that an item has been picked up or dropped off, from the camera device 2.

[0098] Similarly, for example, when the communication control unit 611 completes a comparison between the item name of an item instructed to be shipped or moved and the item name notified from the camera device 2, it sends a comparison completion notification to the camera device 2.

[0099] The display control unit 612 controls the display of various information on the display device. For example, the display control unit 612 controls the display 67 to display an item list display screen, an item search screen, and the like generated by the server device 1.

[0100] Furthermore, when the display control unit 612 receives a notification from the camera device 2 that an article has been picked up, the display control unit 612 controls the display 67 to display a message to the effect that the article has been picked up. Furthermore, when the display control unit 612 receives a notification from the camera device 2 that an article has been put down, the display control unit 612 controls the display 67 to display a message to the effect that the article has been put down.

[0101] The collation unit 613 compares the name of the item instructed by the server device 1 to be shipped or moved with the name of the item recognized as having been picked up by the camera device 2. For example, when the collation unit 613 receives a notification from the camera device 2 that an item has been picked up, the collation unit 613 compares the name of the item included in the notification with the name of the item instructed by the server device 1 to be shipped or moved.

[0102] If the item names do not match as a result of the comparison, the comparison unit 613 may work in cooperation with the display control unit 612 to display a warning message on the display 67 to the effect that an item other than the item instructed to be shipped or moved may have been picked up.

[0103] The program for causing the worker terminal 3 to execute the processes for realizing the above-mentioned functions can be provided by recording it in an installable or executable format on a computer-readable recording medium such as a CD-ROM, FD, CD-R, or DVD. The program may also be provided or distributed via a network such as the Internet.

[0104] Next, the processing executed by the information processing system S will be described. First, the receiving processing of goods will be described. Fig. 21 is a sequence diagram showing an example of the processing executed by the information processing system S according to the first embodiment. Fig. 21 shows an example of the processing when a worker performs the receiving processing.

[0105] The first acquisition unit 512 of the camera device 2 acquires an image captured by the first camera 56 (step ST1). The first acquisition unit 512 also acquires an image captured by the second camera 57 (step ST2). Note that in FIG. 21, the process of acquiring the image captured by the first camera 56 is described as step ST1, and the process of acquiring the image captured by the second camera 57 is described as step ST2, but these processes are executed continuously.

[0106] Furthermore, the communication control unit 411 of the server device 1 transmits an instruction to receive the goods to the worker terminal 3 (step ST3). For example, the communication control unit 411 transmits the instruction to receive the goods to the worker terminal 3 via the communication I / F 45. The communication control unit 611 of the worker terminal 3 receives the instruction to receive the goods via the communication I / F 65.

[0107] The worker operates the forklift 5 in accordance with the receiving instructions for the goods, and moves the forklift 5 to the receiving berth. The worker also operates the forklift 5, and performs a process of inserting the forks 71 into the pallet PT loaded with the goods that has been placed at the receiving berth.

[0108] The second recognition unit 514 of the camera device 2 recognizes that the item has been picked up based on the image captured by the first camera 56 acquired in step ST1 (step ST4). For example, the first recognition unit 513 recognizes a character string WD indicating the item name of the item that is written on the paper PR attached to the front of the item, and acquires the character string WD as the item name of the item to be received.

[0109] Moreover, the second recognition unit 514 recognizes that an article has been picked up when a predetermined time has elapsed since the size of the character string WD in the image captured by the first camera 56 stopped changing.

[0110] Next, the second acquisition unit 516 acquires the location information of the item at the time when it is recognized that the item has been picked up (step ST5). For example, the second acquisition unit 516 reads the two-dimensional code included in the image captured by the second camera 57 at the time when the second recognition unit 514 recognizes that the item has been picked up in step ST4. The second acquisition unit 516 acquires the location corresponding to the two-dimensional code as a location indicating the current position of the item.

[0111] The communication control unit 511 also transmits a notification to the worker terminal 3 that it has recognized that the item has been picked up (step ST6). For example, when the second recognition unit 514 recognizes that the item has been picked up in step ST4, the communication control unit 511 transmits a notification to the worker terminal 3 via the communication I / F 55 that it has recognized that the item has been picked up. The notification includes the name of the item to be received that has been acquired by the first recognition unit 513.

[0112] In addition, in FIG. 21, the processing of step ST6 is described as processing after step ST5, but the processing of step ST6 may be performed in parallel with the processing of step ST5, or may be performed after step ST4 and before the processing of step ST5.

[0113] The communication control unit 611 of the worker terminal 3 receives the notification that it has been recognized that the item has been picked up from the camera device 2. Next, the display control unit 612 causes the display 67 to display that the item has been picked up (step ST7).

[0114] For example, the display control unit 612 causes the display 67 to display the name of the item to be received, which is included in the notification, and a message indicating that the item has been picked up. This allows the worker to understand that it is now OK to operate the forklift 5. The worker then operates the forklift 5 to transport the item.

[0115] Furthermore, the communication control unit 511 of the camera device 2 transmits a notification to the server device 1 that the item to be received is in transit (step ST8). For example, the communication control unit 511 transmits a notification to the server device 1 that the item to be received is in transit via the communication I / F 55. The notification includes the item name and location of the item to be received.

[0116] The communication control unit 411 of the server device 1 receives the notification that the item to be received is in transit. Next, the storage control unit 412 registers that the item to be received is in transit (step ST9). For example, the storage control unit 412 associates the item name and location included in the notification that the item to be received is in transit, and further adds a status of "in transit" to store the result in the management information 441.

[0117] The worker searches for an empty area in the warehouse while moving the forklift 5. When the worker finds an empty area, he operates the forklift 5 to lower the forks 71. After that, the worker operates the forklift 5 to remove the forks 71 from the pallet PT.

[0118] In this example, the worker visually searches for an empty area in the warehouse, but the server device 1 may specify a storage location for the goods to be received. As an example, the server device 1 refers to the management information 441, identifies areas where the goods are not stored, and specifies one of the areas as a storage location. In this case, the communication control unit 411 transmits information on the storage location to the worker terminal 3 together with an instruction to receive the goods.

[0119] Next, the second recognition unit 514 of the camera device 2 recognizes that the article has been put down based on the image captured by the first camera 56 acquired in step ST1 (step ST10). For example, the second recognition unit 514 recognizes that the article has been put down when the reduction rate of the size of the character string WD in the image captured by the first camera 56 exceeds a predetermined ratio.

[0120] Next, the second acquisition unit 516 acquires the location information of the article at the time when it is recognized that the article has been dropped off (step ST11). For example, the second acquisition unit 516 reads a two-dimensional code included in the image captured by the second camera 57 at the time when the second recognition unit 514 recognizes in step ST11 that the article has been dropped off. The second acquisition unit 516 acquires a location corresponding to the two-dimensional code as a location indicating the current location of the article.

[0121] The communication control unit 511 also transmits a notification to the worker terminal 3 that it has recognized that the item has been unloaded (step ST12). For example, when the second recognition unit 514 recognizes that the item has been unloaded in step ST11, the communication control unit 511 transmits a notification to the worker terminal 3 via the communication I / F 55 that it has recognized that the item has been unloaded. The notification includes the name of the item to be received that has been acquired by the first recognition unit 513.

[0122] In addition, in FIG. 21, the processing of step ST12 is described as processing after step ST11, but the processing of step ST12 may be performed in parallel with the processing of step ST11, or may be performed after step ST10 and before the processing of step ST11 is performed.

[0123] The communication control unit 611 of the worker terminal 3 receives the notification that it has been recognized that the article has been unloaded from the camera device 2. Next, the display control unit 612 causes the display 67 to display that the article has been unloaded (step ST13).

[0124] For example, the display control unit 612 displays the name of the item to be received, which is included in the notification, and a message indicating that the item has been unloaded on the display 67. This allows the worker to understand that the receiving process of the item has been completed.

[0125] Furthermore, the communication control unit 511 of the camera device 2 transmits a receipt completion notification indicating that the arrival of the item has been completed to the server device 1 (step ST14). For example, the communication control unit 511 transmits the receipt completion notification to the server device 1 via the communication I / F 55. The notification includes the item name and location of the item to be received.

[0126] The communication control unit 411 of the server device 1 receives the arrival completion notification. Next, the storage control unit 412 registers the arrival of the item (step ST15) and ends this process. For example, the storage control unit 412 associates the item name and location included in the arrival completion notification, erases the status of "in transit", and stores it in the management information 441.

[0127] Next, the shipment process of goods will be described. Fig. 22 is a sequence diagram showing an example of the process executed by the information processing system S according to the first embodiment. Fig. 22 shows an example of the process when an operator performs the shipment process. Steps ST31 and ST32 are similar to steps ST1 and ST2 in Fig. 21, and therefore the description will be omitted.

[0128] The communication control unit 411 of the server device 1 transmits an instruction to ship the item to the worker terminal 3 (step ST33). For example, the communication control unit 411 transmits the instruction to ship the item to the worker terminal 3 via the communication I / F 45. The instruction to ship the item includes the name of the item to be shipped and the location where the item is stored. The communication control unit 611 of the worker terminal 3 receives the instruction to ship the item via the communication I / F 65.

[0129] Next, the display control unit 612 causes the display 67 to display the item name and location information related to the item shipping instruction (step ST34). For example, the display control unit 612 causes the display 67 to display the item name of the item to be shipped and the location where the item is stored, which are included in the item shipping instruction. This makes it easier for the worker to grasp information related to the item to be shipped.

[0130] The worker operates the forklift 5 in accordance with the shipping instructions for the goods, and moves the forklift 5 to the location displayed on the display in step ST34. The worker also operates the forklift 5 to insert the forks 71 into the pallet PT on which the goods stored at that location are loaded.

[0131] Steps ST35 to ST37 are similar to steps ST4 to ST6 in Fig. 21, and therefore description thereof will be omitted. After step ST37, the collation unit 613 collates the item name of the item related to the shipping instruction with the item name related to the notice that the item has been picked up (step ST38). For example, the collation unit 613 collates the item name included in the shipping instruction for the item sent from the server device 1 with the item name included in the notice that the item has been picked up.

[0132] Although not shown, if the item names do not match as a result of the comparison, the comparison unit 613 may cooperate with the display control unit 612 to display a message on the display 67 to warn the worker that an item other than the item related to the shipping instruction may have been picked up. This makes it possible to prevent the wrong item from being transported to the shipping berth.

[0133] If the comparison result shows that the article names match, the communication control unit 611 transmits a comparison completion notification to the camera device 2 via the communication I / F 65 (step ST39). The communication control unit 511 of the camera device 2 receives the comparison completion notification via the communication I / F 55. Steps ST40 to ST46 are similar to steps ST7 to ST13 in FIG. 21, and therefore description thereof will be omitted.

[0134] During shipping processing, the display control unit 612 displays on the display 67 a message indicating that the goods have been unloaded, so that the worker can understand that the goods to be shipped have been unloaded at the shipping berth.

[0135] The communication control unit 511 of the camera device 2 transmits a shipment completion notice indicating that the shipment of the item has been completed to the server device 1 (step ST47). For example, the communication control unit 511 transmits the shipment completion notice to the server device 1 via the communication I / F 55. The notice includes the name of the item to be received and a location indicating the position of the shipping berth.

[0136] The communication control unit 411 of the server device 1 receives the shipping completion notification. Next, the memory control unit 412 deletes the information of the item related to the shipping from the management information 441 (step ST48), and ends this process. For example, the memory control unit 412 deletes the information of the item corresponding to the shipping instruction transmitted in step ST33 from the management information 441.

[0137] Next, the movement processing of an item will be described. Fig. 23 is a sequence diagram showing an example of the processing executed by the information processing system S according to the first embodiment. Fig. 23 shows an example of the processing when a worker performs the movement processing. Steps ST51 and ST52 are similar to steps ST31 and ST32 in Fig. 22, and therefore the description will be omitted.

[0138] The communication control unit 411 of the server device 1 transmits an instruction to move the item to the worker terminal 3 (step ST53). For example, the communication control unit 411 transmits the instruction to move the item to the worker terminal 3 via the communication I / F 45. The instruction to move the item includes the name of the item to be moved, the location where the item is stored, and a location indicating the destination. The communication control unit 611 of the worker terminal 3 receives the instruction to move the item via the communication I / F 65.

[0139] Next, the display control unit 612 causes the display 67 to display the name of the item related to the instruction to move the item, the current location information of the item, and the location information of the destination (step ST54). For example, the display control unit 612 causes the display 67 to display the name of the item to be moved, the location where the item is stored, and the location indicating the destination, which are included in the instruction to move the item. This makes it easier for the worker to grasp information about the item to be moved.

[0140] The worker operates the forklift 5 according to the instruction to move the goods, and moves the forklift 5 to the location that is displayed on the display in step ST54 and indicates the current position of the goods to be moved. The worker also operates the forklift 5 to insert the forks 71 into the pallet PT that is loaded with the goods stored at that location.

[0141] Steps ST55 to ST66 are similar to steps ST35 to ST46 in Fig. 22, and therefore description thereof will be omitted. During the movement process, the display control unit 612 displays on the display 67 the fact that the item has been unloaded, so that the worker can understand that the item to be moved has been unloaded in the area of ​​the movement destination.

[0142] The communication control unit 511 of the camera device 2 transmits a movement completion notification indicating that the movement of the item has been completed to the server device 1 (step ST67). For example, the communication control unit 511 transmits the movement completion notification to the server device 1 via the communication I / F 55. The notification includes the name of the item to be moved and a location indicating the position of the shipping berth.

[0143] The communication control unit 411 of the server device 1 receives the movement completion notification. Next, the storage control unit 412 registers the movement of the item (step ST68) and ends this process. For example, the storage control unit 412 associates the item name and location included in the movement completion notification, erases the status of "in movement", and stores it in the management information 441.

[0144] As described above, the camera device 2 according to this embodiment recognizes a symbol such as a character string representing the name of an item based on an image captured by the camera, and recognizes that an item has been picked up or put down based on a change in the size of the symbol in the image. The camera device 2 also obtains location information representing the location where the item was put down, and outputs the name of the item and the location information in association with each other.

[0145] As a result, the camera device 2 according to the present embodiment can automatically recognize that an item has been picked up or dropped off by simply capturing an image with the camera. Moreover, the camera device 2 according to the present embodiment can recognize that an item has been moved by recognizing that an item has been picked up. Furthermore, the camera device 2 according to the present embodiment can recognize that an item has been dropped off and can recognize that an item has been placed by acquiring location information indicating the location where the item has been dropped off. However, when placing items in a free location, it can be said that knowing that an item has been moved or placed is an important matter for properly managing the items. Since the camera device 2 according to the present embodiment can automatically recognize that an item has been picked up or dropped off, the worker can know that an item has been moved or placed without having to judge for himself that the item has been picked up or dropped off. In other words, according to the present embodiment, the convenience of item management in a warehouse where items are placed in a free location can be improved.

[0146] Second embodiment In the above-described first embodiment, a description has been given of a mode in which receiving and shipping are performed by an operator operating the forklift 5. In the second embodiment, a description will be given of a mode in which receiving and shipping are performed by an unmanned forklift.

[0147] In the following, differences from the above-described embodiment will be mainly described, and detailed description of commonalities with the contents already described will be omitted. Also, each embodiment described below may be implemented individually or in appropriate combination.

[0148] Fig. 24 is a diagram showing an example of the overall configuration of an information processing system Sa according to the second embodiment. The information processing system Sa includes a server device 11, a camera device 21, and a drive control device. The server device 1, the camera device 2, and the worker terminal 3 are connected to each other via a network N so as to be able to communicate with each other. The configuration of the server device 11 will be described below with reference to Figs. 25 and 26.

[0149] First, a description will be given of the hardware configuration of the server device 11. Fig. 25 is a block diagram showing an example of the hardware configuration of the server device 11 according to the second embodiment. The server device 11 has a similar configuration to the server device 1 in Fig. 2, but differs from the server device 1 in Fig. 2 in that it includes a CPU 47 instead of the CPU 41.

[0150] Next, a functional configuration of the server device 11 will be described. Fig. 26 is a block diagram showing an example of a functional configuration of the server device 11 according to the second embodiment. In this embodiment, the CPU 47 of the server device 1 executes a program stored in the storage 44 or the like to realize various functions of the server device 11.

[0151] The CPU 47 according to this embodiment has a communication control unit 471, a memory control unit 472, and an instruction unit 473. The storage 44 has management information 441. The memory control unit 472 and the instruction unit 473 are similar to the memory control unit 412 and the instruction unit 413 in FIG. 3, and therefore description thereof will be omitted.

[0152] The communication control unit 471 controls transmission and reception of information with the camera device 21. For example, the communication control unit 471 transmits an instruction to receive an item, an instruction to ship an item, an instruction to move an item, etc. to the camera device 21 via the communication I / F 45. Similarly, for example, the communication control unit 471 receives various notifications from the camera device 21, such as a notification that an item is being moved, a notification that the arrival of an item has been completed, a notification that the shipment of an item has been completed, etc.

[0153] Next, the configuration of camera device 21 will be described with reference to Fig. 27 and Fig. 28. Fig. 27 is a block diagram showing an example of a hardware configuration of a camera device according to the first embodiment. Camera device 21 has a similar configuration to camera device 2 in Fig. 6, but differs from camera device 2 in Fig. 6 in that controller 50 includes CPU 58 and distance measurement sensor 59.

[0154] The distance measuring sensor 59 detects an object around the forklift 5. The distance measuring sensor 59 also detects the distance to the object. The distance measuring sensor 59 is, for example, a sensor device that detects an object and measures the distance using ultrasonic waves, or a sensor device such as LiDAR (Light Detection and Ranging) that detects an object using laser light.

[0155] Next, a functional configuration of the camera device 21 will be described. Fig. 28 is a block diagram showing an example of the functional configuration of the camera device 21 according to the second embodiment. In this embodiment, the CPU 58 of the camera device 21 executes a program stored in the storage 54 or the like to realize various functions of the camera device 21.

[0156] The CPU 58 has a communication control unit 581, a first acquisition unit 582, a first recognition unit 583, a second recognition unit 584, a third recognition unit 585, a second acquisition unit 586, a search unit 587, a collation unit 588, and a determination unit 589. The first acquisition unit 582, the first recognition unit 583, the second recognition unit 584, the third recognition unit 585, and the second acquisition unit 586 are similar to the first acquisition unit 512, the first recognition unit 513, the second recognition unit 514, the third recognition unit 515, and the second acquisition unit 516 in Fig. 9, and therefore description thereof will be omitted.

[0157] The communication control unit 581 controls transmission and reception of information between the server device 11 and the drive control device 4. For example, the communication control unit 581 receives, via the communication I / F 55, from the server device 1, instructions for receiving an item, instructions for shipping an item, instructions for moving an item, and the like.

[0158] Similarly, for example, the communication control unit 581 transmits various requests, such as a movement request to request that the forklift 5 be moved to a specific location in the warehouse, to the drive control unit 4. For example, the communication control unit 511 transmits a notification to the drive control unit 4 that an article has been picked up or dropped off.

[0159] The search unit 587 searches for the situation around the forklift 5. For example, the search unit 587 searches for an area where items can be stored (hereinafter also referred to as an empty shelf) around the forklift 5 based on the sensing result of the distance measurement sensor 59. Note that information on an area recognized as an empty shelf as a result of the search by the search unit 587 may be stored in the storage 54 or the like. In this case, the search unit 587 may search for an empty shelf taking that information into account.

[0160] Also, for example, the search unit 587 searches for an item. As an example, during the receiving process of an item, when the forklift 5 moves to the receiving berth, the search unit 587 searches for the item to be received based on the sensing result of the distance measurement sensor 59. Here, the fact that the forklift 5 has moved to the receiving berth can be recognized by, for example, the second acquisition unit 586 acquiring the location in real time.

[0161] As another example, when an item is shipped or moved, the search unit 587 searches for the item to be shipped or moved based on the sensing results of the distance measurement sensor 59 when the forklift 5 moves to the location of the item to be shipped or moved.

[0162] Here, in this embodiment, when the search unit 587 detects the presence of an object that is thought to be an article from the sensing results of the distance measurement sensor 59, the first recognition unit 583 performs a process of obtaining the name of the article.

[0163] The collation unit 588 compares the name of the item instructed by the server device 11 to be shipped or moved with the name of the item found as a result of the search. For example, when the search unit 587 finds an item, the collation unit 588 compares the name of the item acquired by the first recognition unit 583 as the name of the item with the name of the item instructed by the server device 11 to be shipped or moved.

[0164] If the comparison result indicates that the article names do not match, the comparison unit 588 may cooperate with the communication control unit 581 to transmit a request to the drive control device 4 to unload the article. When the forklift 5 unloads the article under the control of the drive control device 4, the search unit 587 executes the article search process again.

[0165] The determination unit 589 determines whether there is a space to unload the goods. For example, when the forklift 5 moves to a location that is recognized as an empty shelf during the receiving process of the goods, the determination unit 589 determines whether there is a space to unload the goods based on the sensing result of the distance measurement sensor 59.

[0166] Here, even if the search unit 587 recognizes an area as an empty shelf, there is a possibility that an item may be placed in the area after the recognition, or that an object other than an item to be managed in the warehouse may be placed in the area by a worker, etc. For this reason, in this embodiment, before the item is actually unloaded, the determination unit 589 determines whether there is space to unload the item.

[0167] The description will continue by returning to Fig. 24. The drive control device 4 is a device that controls the drive of the forklift 5. The configuration of the drive control device 4 will be described below with reference to Figs. 29 and 30.

[0168] First, a description will be given of the hardware configuration of the drive control device 4. Fig. 29 is a block diagram showing an example of the hardware configuration of the drive control device 4 according to the second embodiment. The drive control device 4 includes a CPU 81, a RAM 82, a ROM 83, a storage 84, and a communication I / F 85. These units 81 to 85 are connected to each other via a data bus.

[0169] The CPU 81 comprehensively controls the drive control device 4. For example, the CPU 81 uses the RAM 82 as a working area and executes various processes by executing programs stored in the ROM 83, storage 84, etc. The storage 84 is a non-volatile memory to which data can be written and rewritten. The communication I / F 85 is a device for establishing communication with the camera device 21 in a predetermined format.

[0170] Next, a functional configuration of the drive control device 4 will be described. Fig. 30 is a block diagram showing an example of the functional configuration of the drive control device 4 according to the second embodiment. In this embodiment, the CPU 81 of the drive control device 4 executes programs stored in the storage 84 or the like to realize various functions of the drive control device 4. The CPU 81 has a communication control unit 811 and a drive control unit 812.

[0171] The communication control unit 811 controls transmission and reception of information with the camera device 21. For example, the communication control unit 811 receives various requests from the camera device 21 via the communication I / F 85, such as a request to move an item to an incoming berth, a request to move an item to a shipping berth, and a request to move an item to a storage position.

[0172] Similarly, for example, the communication control unit 811 receives various notifications from the camera device 21, such as a notification that an item has been picked up, a notification that an item has been dropped off, a notification that an item has been found, and a notification that permission to drop off an item is granted.

[0173] The drive control unit 812 controls the drive of the forklift 5. For example, in response to a request to move to the receiving berth transmitted from the camera device 21, the drive control unit 813 controls the drive wheels 75 and steering wheels 76 of the forklift 5 to move the forklift 5 from the current location to the receiving berth.

[0174] As an example, the request to move to the receiving berth includes information indicating the route from the current location of the forklift 5 to the receiving berth, and the drive control unit 812 controls the drive wheels 75 and steering wheels 76 so that the forklift 5 moves along this route. Note that the same processing as above is performed when moving to a shipping berth or to a shelf that is the destination of the move.

[0175] Furthermore, when the driving control unit 812 receives a notification from the camera device 21 that an article has been found, the driving control unit 812 performs control to pick up the article. For example, the driving control unit 812 inserts the fork 71 into a pallet on which the articles are loaded, and controls the operation of the mast 73 to raise the position of the fork 71, thereby picking up the article. At this time, the sensing result of the distance measurement sensor 59 of the camera device 21 may be used to accurately insert the fork 71 into the pallet.

[0176] Furthermore, when the driving control unit 812 receives a notification from the camera device 21 permitting the unloading of the article, the driving control unit 812 performs control to unload the article. For example, the driving control unit 812 controls the operation of the mast 73 to lower the position of the fork 71, and then performs control to move the forklift 5 backward to unload the article.

[0177] The program for causing the drive control device 4 to execute the processes for realizing the above-mentioned functions can be provided by recording it in an installable or executable file format on a computer-readable recording medium such as a CD-ROM, FD, CD-R, DVD, etc. The program may also be provided or distributed via a network such as the Internet.

[0178] Next, the processing executed by the information processing system Sa will be described. First, the receiving processing of goods will be described. FIG. 31 is a sequence diagram showing an example of the processing executed by the information processing system Sa according to the second embodiment. FIG. 31 shows an example of the processing when an unmanned, automatically operated forklift 5 performs receiving processing. Steps ST71 and ST72 are similar to steps ST1 and ST2 in FIG. 21, and therefore the description will be omitted.

[0179] The search unit 587 of the camera device 21 searches for an empty shelf (step ST73). For example, the search unit 587 searches for an empty shelf around the forklift 5 based on the sensing result of the distance measurement sensor 59.

[0180] 31, the process of searching for an empty shelf is described as step ST73, but the process of step ST73 is executed continuously. Furthermore, information on an area recognized as an empty shelf as a result of the search is stored as empty shelf information in the storage 55 or the like.

[0181] Furthermore, the communication control unit 471 of the server apparatus 11 transmits an instruction to receive the item to the camera device 21 (step ST74). For example, the communication control unit 471 transmits the instruction to receive the item to the camera device 21 via the communication I / F 45. The communication control unit 581 of the camera device 21 receives the instruction to receive the item via the communication I / F 65.

[0182] Next, the communication control unit 581 requests the forklift 5 to move to the receiving berth (step ST75). For example, the communication control unit 581 transmits a request for movement to the receiving berth, including information indicating a route from the current location of the forklift 5 to the receiving berth, via the communication I / F 55. The communication control unit 811 of the drive control device 4 receives the request for movement.

[0183] Next, the drive control unit 812 controls the forklift 5 to move to the receiving berth (step ST76). For example, the drive control unit 812 controls the drive wheels 75 and the steering wheels 76 based on information indicating a route from the current location to the receiving berth, which is included in the movement request received from the camera device 21, to control the forklift 5 to move to the receiving berth.

[0184] When the forklift 5 moves to the receiving berth, the search unit 587 of the camera device 21 recognizes that the forklift 5 has moved to the receiving berth (step ST77). For example, based on the location acquired in real time by the second acquisition unit 586, the search unit 587 recognizes that the forklift 5 has moved to the receiving berth.

[0185] Next, the search unit 587 searches for the article (step ST78). For example, the search unit 587 cooperates with the drive control device 4 to move the forklift 5 within the receiving berth and searches for the article based on the sensing result of the distance measurement sensor 59.

[0186] When an article is found as a result of the search, the first recognition unit 583 acquires the name of the article from the character string written on the paper attached to the article based on the image captured by the first camera 56. At this time, the second recognition unit 584 also recognizes the size of the article on the image captured by the first camera 56 (step ST79).

[0187] Next, the communication control unit 581 transmits a notification that the article has been found to the drive control device 4 via the communication I / F 55 (step ST80). The communication control unit 811 of the drive control device 4 receives the notification that the article has been found. When the notification is received, the drive control unit 812 controls the forklift 5 to pick up the article (step ST81).

[0188] For example, the drive control unit 812 controls the forklift 5 to move to the front of the goods and then pick up the goods by inserting the forks 71 into the pallet on which the goods are loaded. At this time, the drive control unit 812 may use the size of the goods recognized in step ST79 and the sensing result of the distance measurement sensor 59 to move the forklift 5 to an appropriate position for picking up the goods.

[0189] Steps ST82 to ST85 are similar to steps ST4, ST5, ST8, and ST9 in Fig. 21, and therefore will not be described. After step ST85, the communication control unit 581 requests the forklift 5 to move to an empty shelf (step ST86).

[0190] For example, the communication control unit 581 transmits a request for movement to a vacant shelf, including information indicating a route from the current location of the forklift 5 to the vacant shelf, via the communication I / F 55. As an example, the communication control unit 581 refers to the vacant shelf information and identifies the vacant shelf that is closest to the current location of the forklift 5 (inside the receiving berth). The communication control unit 581 transmits the request for movement, including the route from the current location of the forklift 5 to the vacant shelf. The communication control unit 811 of the drive control device 4 receives the request for movement.

[0191] Next, the drive control unit 812 controls the forklift 5 to move to a vacant shelf (step ST87). For example, the drive control unit 812 controls the drive wheels 75 and steering wheels 76 based on information indicating a route from the current location to a vacant shelf, which is included in the movement request received from the camera device 21, to control the forklift 5 to move to a vacant shelf.

[0192] When the forklift 5 moves to the vacant shelf, the determination unit 589 of the camera device 21 recognizes that the forklift 5 has arrived at the vacant shelf (step ST88). For example, the determination unit 589 recognizes that the forklift 5 has arrived at the vacant shelf based on the location acquired in real time by the second acquisition unit 586.

[0193] Next, the determination unit 589 determines whether or not there is a space for unloading the article on the vacant shelf where the article has arrived (step ST89). For example, the determination unit 589 determines whether or not there is a space for unloading the article based on the size of the article recognized in step ST79 and the sensing result of the distance measurement sensor 59.

[0194] Although not shown, when it is determined that there is no space to unload the article, the communication control unit 581 may refer to the vacant shelf information and identify the vacant shelf that is closest to the current location of the forklift 5. In this case, the communication control unit 581 may transmit to the drive control device 4 a movement request that includes information indicating a route from the current location of the forklift 5 to the identified vacant shelf.

[0195] When it is determined in step ST89 that there is space to unload the article, the communication control unit 581 transmits a notification permitting the unloading of the article to the drive control device 4 via the communication I / F 55 (step ST90). The communication control unit 811 of the drive control device 4 receives the notification.

[0196] When the notification is received, the drive control unit 812 performs control to lower the articles (step ST91). For example, the drive control unit 812 controls the operation of the mast 73 to lower the forks 71 and to reverse the forklift 5, thereby removing the forks 71 from the pallet on which the articles are loaded, and performing control to lower the articles.

[0197] Steps ST92 to ST95 are similar to steps ST10, ST11, ST14, and ST15 in FIG. 21, and therefore will not be described.

[0198] Next, the shipping process of an article will be described. Fig. 32 is a sequence diagram showing an example of the process executed by the information processing system Sa according to the second embodiment. Fig. 32 shows an example of the process when an unmanned automatically operated forklift 5 performs the shipping process. Steps ST101 to ST103 are similar to steps ST71 to ST73 in Fig. 31, and therefore the description will be omitted.

[0199] The communication control unit 471 of the server apparatus 1 transmits an instruction to ship the item to the camera device 21 (step ST104). For example, the communication control unit 471 transmits the instruction to ship the item to the camera device 21 via the communication I / F 45. The instruction to ship the item includes the name of the item to be shipped and the location where the item is stored. The communication control unit 581 of the camera device 21 receives the instruction to ship the item via the communication I / F 55.

[0200] Next, the communication control unit 581 requests the forklift 5 to move to the storage location of the item instructed to be shipped (step ST105). For example, the communication control unit 581 transmits a request for movement to the storage location of the item, including information indicating a route from the current location of the forklift 5 to the storage location of the item, via the communication I / F 55. The communication control unit 811 of the drive control device 4 receives the request for movement.

[0201] Next, the drive control unit 812 controls the forklift 5 to move to the storage position of the goods (step ST106). For example, the drive control unit 812 controls the drive wheels 75 and steering wheels 76 based on information indicating a route from the current location to the storage position of the goods, which is included in the movement request received from the camera device 21, to control the forklift 5 to move to the storage position of the goods.

[0202] When the forklift 5 moves to the storage position of the item, the search unit 587 of the camera device 21 recognizes that the forklift 5 has moved to the storage position of the item (step ST107). For example, based on the location acquired in real time by the second acquisition unit 586, the search unit 587 recognizes that the forklift 5 has moved to the storage position of the item.

[0203] Steps ST108 to ST113 are similar to steps ST78 to ST83 in Fig. 31, and therefore the description will be omitted. After step ST113, the collation unit 588 collates the item name of the item related to the shipping instruction with the item name of the found item (step ST114). For example, the collation unit 613 collates the item name included in the shipping instruction for the item transmitted from the server device 1 with the item name acquired by the first recognition unit 583 when searching for the item.

[0204] Although not shown, if the comparison result shows that the article names do not match, the search section 587 may perform article search processing in step ST108 until the article names match.

[0205] If the comparison result shows that the article names match, the process proceeds to step ST115, but steps ST115 and ST116 are similar to steps ST84 and ST85 in Fig. 31 and therefore will not be described. After step ST116, the communication control unit 581 requests the forklift 5 to move to the shipping berth (step ST117).

[0206] For example, the communication control unit 581 transmits a request for movement to a vacant shelf, including information indicating a route from the current location of the forklift 5 to the shipping berth, via the communication I / F 55. As an example, the communication control unit 581 transmits a request for movement including a route from the current location of the forklift 5 to the shipping berth. The communication control unit 811 of the drive control device 4 receives the request for movement.

[0207] Next, the drive control unit 812 controls the forklift 5 to move to the shipping berth (step ST118). For example, the drive control unit 812 controls the drive wheels 75 and the steering wheels 76 based on information indicating a route from the current location to the shipping berth, which is included in the movement request received from the camera device 21, to control the forklift 5 to move to the shipping berth.

[0208] When the forklift 5 moves to the shipping berth, the determination unit 589 of the camera device 21 recognizes that the forklift 5 has arrived at the shipping berth (step ST119). For example, the determination unit 589 recognizes that the forklift 5 has arrived at the shipping berth based on the location acquired in real time by the second acquisition unit 586.

[0209] Next, the determination unit 589 determines whether or not there is a space for unloading the goods at the shipping berth where the goods have arrived (step ST120). For example, the determination unit 589 determines whether or not there is a space for unloading the goods based on the size of the goods recognized in step ST109 and the sensing result of the distance measurement sensor 59.

[0210] Although not shown, when it is determined that there is no space to unload the goods, the communication control unit 581 may transmit a message to the administrator of the server device 11, the administrator of the warehouse, or the like, to the effect that there is no space to place the goods to be shipped to the shipping berth. In addition, the searching unit 587 may cooperate with the drive control device 4 to search for a space to unload the goods while moving the forklift 5 within the shipping berth.

[0211] Steps ST121 to ST124 are similar to steps ST90 to ST93 in Fig. 31, and therefore their explanation will be omitted. Also, steps ST125 and ST126 are similar to steps ST47 and ST48 in Fig. 22, and therefore their explanation will be omitted.

[0212] Next, the article movement processing will be described. Fig. 33 is a sequence diagram showing an example of the processing executed by the information processing system Sa according to the second embodiment. Fig. 33 shows an example of the processing when an unmanned automatically operated forklift 5 performs the movement processing. Steps ST131 to ST133 are similar to steps ST101 to ST103 in Fig. 32, and therefore the description will be omitted.

[0213] The communication control unit 471 of the server apparatus 1 transmits an instruction to move the item to the camera device 21 (step ST134). For example, the communication control unit 471 transmits the instruction to move the item to the camera device 21 via the communication I / F 45. The instruction to move the item includes the name of the item to be moved, the location where the item is stored, and a location indicating the destination of the item. The communication control unit 581 of the camera device 21 receives the instruction to move the item via the communication I / F 55.

[0214] Steps ST135 to ST146 are similar to steps ST105 to ST116 in Fig. 32, and therefore description thereof will be omitted. After step ST146, the communication control section 581 requests the forklift 5 to move the article to the destination (step ST147).

[0215] For example, the communication control unit 581 transmits a movement request to the destination including information indicating a route from the current location of the forklift 5 to the destination via the communication I / F 55. As an example, the communication control unit 581 transmits a movement request including a route from the current location of the forklift 5 to the destination of the item included in the movement instruction received from the server device 11. The communication control unit 811 of the drive control device 4 receives the movement request.

[0216] Next, the drive control unit 812 controls the forklift 5 to move to the destination of the article (step ST148). For example, the drive control unit 812 controls the drive wheels 75 and the steering wheels 76 based on information indicating a route from the current location to the destination of the article, which is included in the movement request received from the camera device 21, to control the forklift 5 to move to the destination of the article.

[0217] When the forklift 5 moves to the destination of the article, the determination unit 589 of the camera device 21 recognizes that the forklift 5 has arrived at the destination of the article (step ST149). For example, the determination unit 589 recognizes that the forklift 5 has arrived at the destination of the article based on the location acquired in real time by the second acquisition unit 586.

[0218] Next, the determination unit 589 determines whether or not there is a space to unload the item at the destination of the arrived item (step ST150). For example, the determination unit 589 determines whether or not there is a space to unload the item based on the size of the item recognized in step ST139 and the sensing result of the distance measurement sensor 59.

[0219] Although not shown, if it is determined that there is no space to unload the item, the communication control unit 581 may send a message to the administrator of the server device 11, the administrator of the warehouse, etc., to the effect that there is no space to place the item in the area specified as the destination.

[0220] Steps ST151 to ST154 are similar to steps ST121 to ST124 in Fig. 32, and therefore their explanation will be omitted. Also, steps ST155 and ST156 are similar to steps ST67 and ST68 in Fig. 23, and therefore their explanation will be omitted.

[0221] As described above, in the information processing system Sa according to this embodiment, the unmanned and automatically operated forklift 5 performs processes such as receiving, shipping, and moving of goods.

[0222] For example, when an unmanned, automatically operated forklift is used to transport goods and place the goods in a free location, the location where the goods are actually placed may differ from the location where the goods are recorded. This is thought to occur because the system may recognize that the goods have been unloaded even though they have not actually been unloaded, or may recognize that the goods have not been unloaded even though they have actually been unloaded. In order to prevent this, a measure can be considered in which a human being makes the decision as to whether the goods have been picked up or unloaded, but in this case, even though the goods are transported unmanned, a human being must make the decision, and there is a possibility that the benefits of this system cannot be fully obtained.

[0223] In contrast, the camera device 21 according to this embodiment can automatically recognize that an item has been picked up or dropped off from an image captured by the camera. Therefore, the camera device 21 according to this embodiment can accurately record the placement position (storage position) of an item without relying on human judgment, even if the item is transported unmanned and the item is placed at a free location. In other words, this embodiment can improve the convenience of item management in a warehouse where items are placed at a free location.

[0224] The above-described first and second embodiments can be modified as appropriate by changing a portion of the configuration or function of each device of the information processing system S (Sa). Therefore, some modified examples of the above-described embodiments will be described below as other embodiments. The following mainly describes the differences from the above-described embodiments, and detailed descriptions of the commonalities with the contents already described will be omitted. The modified examples described below may be implemented individually or in appropriate combination.

[0225] (Variation 1) In the above-described first and second embodiments, a configuration has been described in which a location (position information) is acquired based on the second camera 57 and a two-dimensional code for location recognition. In this modification, a configuration will be described in which position information is acquired using a sensor such as a gyro sensor.

[0226] The forklift 5 of this modified example is provided with a gyro sensor instead of the second camera 57. An example of the installation position of the gyro sensor in the forklift 5 will be described below with reference to Figs. 34 and 35.

[0227] Figures 34 and 35 are schematic diagrams showing an example of the installation position of the camera in the forklift 5 according to the modified example 1. Figure 34 shows an example of the forklift 5 according to the modified example 1 as viewed from the side. Figure 35 shows an example of the forklift 5 according to the modified example 1 as viewed from the front.

[0228] As shown in Fig. 34 and Fig. 35, the forklift 5 according to this modification is provided with a gyro sensor 70 on a head guard 74. Also, unlike the forklift 5 according to the first and second embodiments, the mast 73 is not provided with a second camera 57. The gyro sensor 70 is a sensor that measures a rotational angular velocity. The gyro sensor 70 is an example of a position measurement sensor.

[0229] The second acquisition unit 516 (586) of this modification acquires position information indicating the current location of the forklift 5 based on the sensing result of the gyro sensor 70, and acquires the position of the target article from the position information. For example, the second acquisition unit 516 (586) identifies the current location of the forklift 5 by detecting a change in the traveling direction of the forklift 5 based on the sensing result of the gyro sensor 70.

[0230] In this modification, an offset value is prepared in advance for converting position information indicating the position where the gyro sensor 70 is provided into position information of the tips of the forks 71. The second acquisition unit 516 uses the offset value to convert position information indicating the current location of the forklift 5 identified by the gyro sensor 70 into position information indicating the position of the tips of the forks 71. Then, the second acquisition unit 516 (586) acquires the converted position information as position information of the target item.

[0231] In this modification, the second acquisition unit 516 (586) acquires position information representing the current location of the forklift 5 at the time when the second recognition unit 514 (584) recognizes that the article has been unloaded. The second acquisition unit 516 (586) also converts the position information into position information of the tips of the forks 71 using a previously prepared offset value. The second acquisition unit 516 (586) then acquires the converted position information as position information representing the position of the article at the time when the second recognition unit 514 (584) recognizes that the article has been unloaded.

[0232] The communication control unit 511 (581) transmits the location information to the server device 1 (11) together with the item name acquired by the first recognition unit 513 (583).

[0233] The second acquisition unit 516 (586) may continuously acquire location information indicating the current location of the forklift 5 regardless of whether the second recognition unit 514 (584) recognizes that the article has been unloaded. This allows the manager of the server device 1 (11) or the manager of the warehouse to always know the current location of the forklift 5.

[0234] According to this modified example, location information indicating the location where an article has been dropped off can be obtained without providing a barcode, two-dimensional code, or the like for location recognition on the shelf.

[0235] In this modified example, the gyro sensor is used to acquire location information indicating the current location of the forklift 5. However, the location information indicating the current location of the forklift 5 may be acquired using a sensor other than the gyro sensor, such as an acceleration sensor or a GPS sensor.

[0236] (Variation 2) In the above-mentioned first embodiment, second embodiment, and modification 1, the mobile body that transports the goods is a forklift, but the mobile body that transports the goods is not limited to a forklift. For example, the mobile body that transports the goods may be a crane. As an example, when the mobile body that transports the goods is a crane, at least one camera is provided near the hook of the crane. Also, a sensor for position measurement may be provided together with the camera.

[0237] According to this modified example, even when goods are transported using a transport device other than a forklift, the convenience of goods management in a warehouse where goods are placed in a free location can be improved.

[0238] Although the embodiment of the present invention has been described above, the embodiment is presented as an example and is not intended to limit the scope of the invention. The above novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0239] 1, 11 Server device 2. 21 Camera Device 3. Worker terminal 4 Drive control device 5. Forklift 41, 47, 51, 58, 61, 81 CPU 411, 471, 511, 581, 611, 811 Communication control unit 412, 472 Memory control unit 413, 473 Instruction section 414 Screen generation section 512, 582 1st acquisition part 513, 583 1st recognition part 514, 584 2nd recognition part 515, 585 3rd recognition part 516, 586 2nd acquisition part 587 Search Department 588, 613 Collation Department 589 Judgment section 612 Display control unit 812 Drive control unit [Prior art documents] [Patent documents]

[0240] [Patent Document 1] JP 2020-19647 A

Claims

1. A first acquisition unit that acquires an image from a camera mounted on a moving object that transports an article; a first recognition unit that recognizes a symbol attached to the item from within the image and acquires identification information that can identify the item from the symbol; a second recognition unit that recognizes an action performed by the moving object on the item based on a size change of the symbol recognized from within the image by the first recognition unit; A second acquisition unit that acquires position information of the item when a motion of the moving object is recognized; an output unit that outputs correspondence information that associates the identification information of the item indicated by the symbol with the position information of the item according to the operation content recognized by the second recognition unit; An information processing device comprising:

2. the second recognition unit recognizes that a first action of picking up the item has been performed when the size of the symbol continuously increases and the increased state continues for a predetermined time, and recognizes that a second action of unloading the item has been performed when the size of the symbol continuously decreases from the increased state; the output unit outputs the correspondence information when at least one of the first motion and the second motion is recognized. The information processing device according to claim 1 .

3. The second recognition unit detects that the size of the symbol is continuously enlarged and that the enlarged state continues for a predetermined time, (a) the size of the symbol on the image is a predetermined size; (b) when the symbol is printed on paper and attached to the object, the paper can be recognized as an object; (c) the symbol is present at a predetermined position on the image; (d) determining that the first action has been performed when at least one of the following conditions is satisfied: a distance to the article measured using a distance measuring sensor is within a predetermined range; The information processing device according to claim 2 .

4. The item is received at or shipped from a warehouse that stores the item, A code symbol encoding information representing a location within the warehouse is placed in the warehouse, The second acquisition unit acquires, when the second recognition unit recognizes that the action of the moving object on the item has been performed, a position within the warehouse specified based on a reading result of the code symbol included in the image as the position information of the item. The information processing device according to claim 1 .

5. the second acquisition unit acquires the position information of the article from a position of the moving body identified based on a measurement result of a position measurement sensor mounted on the moving body when the second recognition unit recognizes that the moving body has performed an action on the article. The information processing device according to claim 1 .

6. The information processing device and the server device are provided. The information processing device includes: A first acquisition unit that acquires an image from a camera mounted on a moving object that transports an article; a first recognition unit that recognizes a symbol attached to an article from within the image and acquires identification information that can identify the article from the symbol; a second recognition unit that recognizes an action performed by the moving object on the item based on a size change of the symbol recognized from within the image by the first recognition unit; A second acquisition unit that acquires position information of the item when a motion of the moving object is recognized; an output unit that outputs correspondence information that associates the identification information of the item indicated by the symbol with the position information of the item according to the operation content recognized by the second recognition unit; Equipped with The server device includes: A receiving unit that receives the correspondence information; a storage control unit that stores the correspondence information in a storage device; An information processing system comprising:

7. Computer, A first acquisition unit that acquires an image from a camera mounted on a moving object that transports an article; a first recognition unit that recognizes a symbol attached to the item from within the image and acquires identification information that can identify the item from the symbol; a second recognition unit that recognizes an action performed by the moving object on the item based on a size change of the symbol recognized from within the image by the first recognition unit; A second acquisition unit that acquires position information of the item when a motion of the moving object is recognized; an output unit that outputs correspondence information that associates the identification information of the item indicated by the symbol with the position information of the item according to the operation content recognized by the second recognition unit; A program that functions as a

Images