Information processing device, information processing system, and program

Abstract

To improve convenience of goods management in a warehouse where goods are arranged in a free location.SOLUTION: An information processing device according to an embodiment includes a first acquisition unit, a first recognition unit, a second recognition unit, a second acquisition unit, and an output unit. The first acquisition unit acquires an image from a camera mounted on a moving object that transports goods. The first recognition unit recognizes a symbol attached to the goods from within the image and acquires identification information capable of identifying the goods from the symbol. The second recognition unit recognizes an action that the moving object performs to the goods based on a change in size of the symbol recognized from within the image by the first recognition unit. The second acquisition unit acquires position information on the goods in recognizing an action of the moving object. The output unit outputs correspondence information that associates the identification information on the goods indicated by the symbol with the position information on the goods according to an action content recognized by the second recognition unit.SELECTED DRAWING: Figure 9

Description

Technical Field

[0004] , , [Figure 1] , ,

[0005] , ,

[0001] Embodiments of the present invention relate to an information processing apparatus, an information processing system, and a program.

Background Art

[0002] Conventionally, in a warehouse where articles are placed in a free location, a system for managing the positions of articles in the warehouse is known, which acquires the position information of the placed articles and associates the acquired position information with the identification information of the articles. In such a system, the fact that an article has been picked up (loaded) and the fact that an article has been unloaded are recognized by receiving an input from the user.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The problem to be solved by the embodiments of the present invention is to improve the convenience of article management in a warehouse where articles are placed in a free location.

Means for Solving the Problems

Brief Description of the Drawings

[0005] [Figure 1]Figure 1 is a block diagram showing an example of the configuration of an information processing system according to the first embodiment. [Figure 2] Figure 2 is a block diagram showing an example of the hardware configuration of a server device according to the first embodiment. [Figure 3] Figure 3 is a block diagram showing an example of the functional configuration of a server device according to the first embodiment. [Figure 4] Figure 4 shows an example of a display screen generated by the server device according to the first embodiment. [Figure 5] Figure 5 shows an example of a display screen generated by the server device according to the first embodiment. [Figure 6] Figure 6 is a block diagram showing an example of the hardware configuration of a camera device according to the first embodiment. [Figure 7] Figure 7 is a schematic diagram showing an example of the camera installation position in a forklift according to the first embodiment. [Figure 8] Figure 8 is a schematic diagram showing an example of the camera installation position in a forklift according to the first embodiment. [Figure 9] Figure 9 is a block diagram showing an example of the functional configuration of a camera device according to the first embodiment. [Figure 10] Figure 10 illustrates an example of the process for recognizing that a forklift has picked up an item according to the first embodiment. [Figure 11] Figure 11 illustrates an example of the process for recognizing that a forklift has picked up an item according to the first embodiment. [Figure 12] Figure 12 illustrates an example of the process for recognizing that an item has been picked up by a forklift according to the first embodiment. [Figure 13] Figure 13 illustrates an example of the process for recognizing that an item has been picked up by a forklift according to the first embodiment. [Figure 14] Figure 14 illustrates an example of the process for recognizing that an item has been picked up by a forklift according to the first embodiment. [Figure 15]Figure 15 illustrates an example of the process for recognizing that a forklift has picked up an item according to the first embodiment. [Figure 16] Figure 16 illustrates an example of the process for recognizing that an item according to the first embodiment has been unloaded from a forklift. [Figure 17] Figure 17 illustrates an example of the process for recognizing that an item according to the first embodiment has been unloaded from a forklift. [Figure 18] Figure 18 is a diagram illustrating an example of the location acquisition process according to the first embodiment. [Figure 19] Figure 19 is a block diagram showing an example of the hardware configuration of a worker terminal according to the first embodiment. [Figure 20] Figure 20 is a block diagram showing an example of the functional configuration of a worker terminal according to the first embodiment. [Figure 21] Figure 21 is a sequence diagram showing an example of a process performed by the information processing system according to the first embodiment. [Figure 22] Figure 22 is a sequence diagram showing an example of a process performed by the information processing system according to the first embodiment. [Figure 23] Figure 23 is a sequence diagram showing an example of a process performed by the information processing system according to the first embodiment. [Figure 24] Figure 24 is a block diagram showing an example of the configuration of an information processing system according to the second embodiment. [Figure 25] Figure 25 is a block diagram showing an example of the hardware configuration of a server device according to the second embodiment. [Figure 26] Figure 26 is a block diagram showing an example of the functional configuration of a server device according to the second embodiment. [Figure 27] Figure 27 is a block diagram showing an example of the hardware configuration of a camera device according to the second embodiment. [Figure 28] Figure 28 is a block diagram showing an example of the functional configuration of a camera device according to the second embodiment. [Figure 29]FIG. 29 is a block diagram showing an example of the hardware configuration of the drive control device according to the second embodiment. [Figure 30] FIG. 30 is a block diagram showing an example of the functional configuration of the drive control device according to the second embodiment. [Figure 31] FIG. 31 is a sequence diagram showing an example of the processing executed by the information processing system according to the second embodiment. [Figure 32] FIG. 32 is a sequence diagram showing an example of the processing executed by the information processing system according to the second embodiment. [Figure 33] FIG. 33 is a sequence diagram showing an example of the processing executed by the information processing system according to the second embodiment. [Figure 34] FIG. 34 is a schematic diagram showing an example of the installation positions of the camera and the gyro sensor in the forklift according to Modification 1. [Figure 35] FIG. 35 is a schematic diagram showing an example of the installation positions of the camera and the gyro sensor in the forklift according to Modification 1.

MODE FOR CARRYING OUT THE INVENTION

[0006] Hereinafter, embodiments will be described with reference to the drawings.

[0007] (First Embodiment) FIG. 1 is a diagram showing an example of the overall configuration of the information processing system S according to the first embodiment. The information processing system S includes a server device 1, a camera device 2, and a worker terminal 3. The server device 1, the camera device 2, and the worker terminal 3 are connected to each other via a network N so as to be communicable.

[0008] In FIG. 1, one server device 1, one camera device 2, and one worker terminal 3 are connected, but the number of devices is not limited thereto. For example, the information processing system S may include a plurality of camera devices 2 and a plurality of worker terminals 3.

[0009] Server device 1 is a device that manages the placement location of items in a warehouse where items are placed in a free location system. Here, a warehouse refers to a facility for storing tangible goods, such as a warehouse in a logistics center. The configuration of server device 1 will be explained below using Figures 2 to 5.

[0010] First, the hardware configuration of server device 1 will be described. Figure 2 is a block diagram showing an example of the hardware configuration of server device 1 according to the first embodiment. Server device 1 includes a CPU (Central Processing Unit) 41, RAM (Random Access Memory) 42, ROM (Read Only Memory) 43, storage 44, communication I / F (Interface) 45, and user I / F 46. These parts 41 to 46 are connected to each other via a data bus.

[0011] The CPU 41 provides overall control over the server device 1. For example, the CPU 41 uses the RAM 42 as a workspace and executes various processes by running programs stored in the ROM 43, storage 44, etc. The storage 44 is a non-volatile memory that allows data to be written to and rewritten.

[0012] The communication interface 45 is a device for establishing a predetermined form of communication with the camera device 2 and the worker terminal 3. The form of communication is not particularly limited, but examples include those conforming to standards such as Ethernet® and IEEE 802.11.

[0013] User I / F46 is a device that enables the reception of input from operators of server device 1, such as system administrators, and the output of information to those operators. Examples of user I / F46 include keyboards, pointing devices, displays, and speakers.

[0014] Note that the configuration shown in Figure 2 is an example, and the hardware configuration of Server Device 1 is not limited to that described above. Server Device 1 may be, for example, a system in which multiple computers work together, or a system that utilizes cloud computing.

[0015] Next, the functional configuration of the server device 1 will be described. Figure 3 is a block diagram showing an example of the functional configuration of the server device 1 according to the first embodiment. In this embodiment, the CPU 41 of the server device 1 realizes various functions of the server device 1 by executing programs stored in the storage 44, etc. The CPU 41 has a communication control unit 411, a storage control unit 412, an instruction unit 413, and a screen generation unit 414. The storage 44 also has management information 441.

[0016] The communication control unit 411 controls the transmission and reception of information between the camera device 2 and the worker terminal 3. For example, the communication control unit 411 receives various notifications from the camera device 2 via the communication I / F 45, such as notifications that an item is in motion, notifications that the item has arrived, and notifications that the item has been shipped. Similarly, the communication control unit 411 also transmits instructions for receiving items, shipping items, moving items, etc., to the worker terminal 3.

[0017] The memory control unit 412 controls the storage of various types of information in the storage device. For example, the memory control unit 412 controls the storage of item names and locations as management information 441 in the storage device 44. Location is an example of positional information.

[0018] The instruction unit 413 generates instructions for workers performing tasks in the warehouse. For example, when goods are unloaded at the receiving berth, which is a space for receiving goods, the instruction unit 413 generates a receiving instruction that instructs a worker to process the goods. The generated receiving instruction is transmitted to the worker terminal 3 by the communication control unit 411.

[0019] Furthermore, for example, when the instruction unit 413 receives a shipping instruction for goods, it generates a shipping instruction that instructs the worker to perform shipping processing, which involves moving the goods stored on the shelves to the shipping berth, which is the space for shipping goods. The generated shipping instruction is transmitted to the worker terminal 3 by the communication control unit 411.

[0020] Furthermore, for example, when the instruction unit 413 receives an instruction to move an item, it generates a move instruction that instructs the worker to perform a move operation to move an item stored in a specific area to another area. The generated shipping instruction is transmitted to the worker terminal 3 by the communication control unit 411.

[0021] The screen generation unit 414 generates various screens. Such screens may be image data representing the screen itself, or content data for displaying the screen on the display device. For example, when the screen generation unit 414 receives a display instruction for the item list screen via the worker terminal 3, it generates the item list screen. The item list screen displays the item name and location (storage position) for all items stored in the warehouse managed by the server device 1. The generated item list screen is provided, for example, to the worker terminal 3 and displayed on the display 67 described later.

[0022] As an example, the screen generation unit 414 generates an item list display screen based on the management information 441 described later. Here, Figure 4 is a diagram showing an example of a screen generated by the server device 1. Figure 4 shows an example of the item list display screen DA.

[0023] As shown in Figure 4, the item list display screen DA includes a location display field AR. The location display field AR shows the correspondence between the item name and the location within the warehouse where the item is stored. For example, the first row of the location display field in Figure 4 indicates that the item with the item name "12345" is stored at the warehouse location indicated as "A-12-3".

[0024] Furthermore, for example, the screen generation unit 414 generates an item search screen when it receives a display instruction for the item search screen via the worker terminal 3. The item search screen is a screen for searching for items stored in the warehouse managed by the server device 1. The generated item search screen is then provided, for example, to the worker terminal 3 and displayed on the display 67 described later.

[0025] As an example, the screen generation unit 414 generates an item search screen that searches for information registered in the management information 441 using the item name as a keyword. Here, Figure 5 shows an example of a screen generated by the server device 1. Figure 5 shows an example of an item search screen DB. As shown in Figure 5, the item search screen DB includes an input form IF, a search button SB, and a result display field DR.

[0026] The input form IF is a form for receiving the name of an item. A worker or other searcher can search for the location of an item by entering the item name in the input form IF. The search button SB is a button for confirming the item name entered in the input form IF. The searcher can search for the location of an item by entering the item name in the input form IF and pressing the search button SB. The results display field DR is a display field for showing the search results.

[0027] As an example, the search results display column DR in the search screen DB of Figure 5 shows an example of search results when the searcher enters "12345" in the input form IF and presses the search button SB. The results display column DR in Figure 5 shows that the item with the item name "12345" is stored at the location indicated by warehouse location "A-12-3".

[0028] The item list display screen DA and item search screen DB described above may also be displayed on a display device other than the worker terminal 3. For example, they may be displayed on the display of a search terminal (not shown).

[0029] Furthermore, the screens generated by the screen generation unit 414 are not limited to those described above. For example, a screen for searching for items stored at a given location may be generated. In addition, various display screens may be generated directly by the worker terminal 3 or the like.

[0030] Management information 441 is information for managing items stored in the warehouse. For example, management information 441 stores the item name and the location where the item is currently stored in association with each other. Management information 441 is updated by the storage control unit 412 according to the processing that has been performed.

[0031] Furthermore, the program that causes the server device 1 to execute the processing necessary to realize the above functions can be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD (Compact Disc)-ROM, flexible disk (FD), CD-R (Recordable), or DVD (Digital Versatile Disk). This program may also be provided or distributed via a network such as the Internet.

[0032] Returning to Figure 1, let's continue the explanation. Camera device 2 is a device that uses a camera to photograph the area around a forklift placed in a warehouse and uses the captured images to support the management of goods. For example, camera device 2 is mounted on a forklift. The configuration of camera device 2 will be explained below using Figures 6 to 18.

[0033] First, the hardware configuration of the camera device 2 will be described. Figure 6 is a block diagram showing an example of the hardware configuration of the camera device according to the first embodiment. The camera device 2 comprises a controller 50, a first camera 56, and a second camera 57.

[0034] The controller 50 includes a CPU 51, RAM 52, ROM 53, storage 54, and a communication I / F 55. These components 51-55, the first camera 56, and the second camera 57 are connected to each other via a data bus. The controller 50 and the first camera 56, and the controller 50 and the second camera 57 may be connected by a wireless communication standard such as Bluetooth®. The controller 50 is an example of an information processing device in this disclosure.

[0035] The CPU 51 comprehensively controls the camera device 2. For example, the CPU 51 uses the RAM 52 as a workspace and executes various processes by running programs stored in the ROM 53, storage 54, etc. Storage 54 is a non-volatile memory that allows data to be written to and rewritten. The communication interface 55 is a device for establishing communication in a predetermined format with the server device 1 and the worker terminal 3.

[0036] The first camera 56 is a camera for recognizing a string representing the name of the item to be processed. The string representing the name of the item is an example of a symbol. The second camera 57 is a camera for acquiring location. The first camera 56 and the second camera 57 are installed on a forklift. A forklift is an example of a mobile device that transports goods. The following describes an example of the installation location of the first camera 56 and the second camera 57.

[0037] Figures 7 and 8 are schematic diagrams showing examples of camera installation positions in a forklift according to the first embodiment. Figure 7 shows an example of the forklift viewed from the side. Figure 8 shows an example of the forklift viewed from the front.

[0038] As shown in Figures 7 and 8, the forklift 5 is equipped with forks 71, a backrest 72, a mast 73, a head guard 74, drive wheels 75, and steering wheels 76. The forks 71 are the parts used to lift goods. The forks 71 are sometimes also called claws.

[0039] The backrest 72 is a component that prevents items lifted by the forks 71 from falling. The mast 73 is a component that raises and lowers the forks 71 of the forklift 5. By sliding the mast 73 up or down, items can be picked up or lowered. The head guard 74 is a component that protects the operator of the forklift 5 from falling objects.

[0040] The drive wheels 75 are wheels that directly receive the force generated by the prime mover and convert it into rotational motion, and then move using that rotational motion. The steering wheels 76 are wheels used for steering. In this embodiment, the forklift 5 controls the drive wheels 75 and steering wheels according to the operator's commands and travels within the warehouse.

[0041] In this embodiment, the first camera 56 is located near the center of the backrest 72. The second camera 57 is located above the mast 73. In this embodiment, the forklift 5 is equipped with two cameras, the first camera 56 and the second camera 57, but the number of cameras may be just one. In this case, the name of the target item and its location are recognized from the image captured by the single camera.

[0042] Furthermore, in this embodiment, the controller 50 is built into the first camera 56 or the second camera 57. However, the controller 50 may be provided separately from the first camera 56 and the second camera 57, and may be installed near the driver's seat of the forklift 5, for example. Alternatively, the first camera 56 and the second camera 57 may be wirelessly connected to a PC (Personal Computer) installed in a location other than the forklift 5, and the PC may perform the role of the controller 50.

[0043] Next, the functional configuration of the camera device 2 will be described. Figure 9 is a block diagram showing an example of the functional configuration of the camera device 2 according to the first embodiment. In this embodiment, the CPU 51 of the camera device 2 realizes various functions of the camera device 2 by executing a program stored in the storage 54, etc. The CPU 51 includes a communication control unit 511, a first acquisition unit 512, a first recognition unit 513, a second recognition unit 514, a third recognition unit 515, and a second acquisition unit 516.

[0044] The communication control unit 511 controls the transmission and reception of information between the server device 1 and the worker terminal 3. For example, the communication control unit 511 sends various notifications to the server device 1 via the communication interface 55, such as notifications that an item is in transit, notifications that the item has been received, and notifications that the item has been shipped. Similarly, the communication control unit 511 also sends notifications to the worker terminal 3, for example, that an item has been picked up or unloaded.

[0045] The first acquisition unit 512 acquires images from cameras mounted on the forklift 5. For example, the first acquisition unit 512 acquires images taken by the first camera 56 and the second camera 57 in real time via the communication interface 55.

[0046] The first recognition unit 513 recognizes symbols attached to articles within the image and obtains identification information that allows for the identification of the articles from those symbols. For example, the first recognition unit 513 recognizes a string of characters containing the article name of the target article, which is included in the image acquired by the first acquisition unit 512, as a symbol, and obtains that string of characters as the article name that allows for the identification of the target article.

[0047] As an example, a piece of paper with a string representing the name of each item is attached to every item stored in the warehouse. In this embodiment, the first recognition unit 513 analyzes the image captured by the first camera 56, which is acquired by the first acquisition unit 512, using known image recognition technology. When the first recognition unit 513 recognizes that a symbol (in this case, a string representing the name of an item) is contained in the image, it acquires the string as the name of the item.

[0048] In this embodiment, the first recognition unit 513 recognizes a string of characters containing the name of an item as a symbol, but the first recognition unit 513 may also recognize barcodes, two-dimensional codes, etc., as symbols.

[0049] The second recognition unit 514 recognizes the actions that the forklift 5 performs on an object based on the size changes of symbols included in images acquired by the first acquisition unit 512 over time. For example, based on the size changes of symbols when the forklift 5 approaches the target object, it recognizes that the forklift 5 has picked up the target object. Picking up the target object is one example of the first action.

[0050] For example, the second recognition unit 514 recognizes that the forklift 5 has picked up the target item based on the size of the string obtained by the first recognition unit 513 as the item name of the target item in the image captured by the first camera 56, which was acquired by the first acquisition unit 512. In the case of recognizing barcodes, two-dimensional codes, etc., as symbols, the recognition of the picking up of the target item is based on the size of the barcodes, two-dimensional codes, etc.

[0051] The process by which the second recognition unit 514 recognizes that the forklift 5 has picked up an item will be explained below using Figures 10 and 11. Figure 10 is a diagram illustrating an example of the process by which the forklift 5 recognizes that an item has been picked up.

[0052] Figure 10 shows forklift 5 approaching item ME to pick it up. In Figure 10, item ME is loaded onto pallet PT, and a paper label PR with the item name ME is attached to item ME.

[0053] Forklift 5 picks up item ME by inserting its forks 71 into pallet PT. At this time, the size of the text written on paper PR in the image captured by the first camera 56 mounted on the backrest 72 of forklift 5 increases as forklift 5 approaches item ME.

[0054] Furthermore, once the forklift 5 reaches a position where it can pick up item ME, it will raise its forks 71 without approaching item ME any further. Therefore, from the time the item ME is picked up until the item ME is lowered and the forks 71 are removed from the pallet PT, the size of the text written on the paper PR in the image captured by the first camera 56 will not change.

[0055] Therefore, the second recognition unit 514 recognizes that the forklift 5 has picked up the target item when the size of the string no longer changes after a predetermined period of time. Here, Figure 11 is a diagram illustrating an example of the process of recognizing that the forklift 5 has picked up an item. Figure 11 shows the time-series change in the size of the string WD written on paper PR in the image captured by the first camera 56.

[0056] The horizontal axis in Figure 11 represents the passage of time. In Figure 11, it is assumed that forklift 5, following the instructions of the operator, began the operation of inserting its forks 71 into pallet PT on which goods ME were loaded at time t0. Similarly, it is assumed that forklift 5 began the operation of picking up goods ME loaded onto pallet PT with its forks 71 inserted at time t1.

[0057] From time t0 to time t1, the forklift 5 approaches the item ME. As a result, as shown in Figure 11, the size of the string WD written on paper PR in the image captured by the first camera 56 increases over time.

[0058] On the other hand, from time t1 onward, the forklift 5 will neither approach nor move away from the item ME until it begins the operation of unloading the item ME and withdrawing the forks 71 from the pallet PT. Therefore, as shown in Figure 11, from time t1 onward, the size of the string WD written on the paper PR in the image captured by the first camera 56 will not change.

[0059] For example, the second recognition unit 514 recognizes the time (time t1 in Figure 11) when the size of the string WD written on the paper PR stops changing, based on the image captured by the first camera 56, which is acquired in real time. Then, when the second recognition unit 514 recognizes that a predetermined time has elapsed since the time when the size of the string WD written on the paper PR stops changing (when it recognizes that time t2 in Figure 11 has elapsed), it recognizes that the target item has been picked up.

[0060] Incidentally, even if the forklift 5 has not picked up item ME, if, for some reason, the forklift 5 stops while the first camera 56 is photographing the string WD written on paper PR, the second recognition unit 514 may mistakenly recognize that the item has been picked up. Below, several methods to reduce the possibility of such misrecognition will be explained using Figures 12 to 15.

[0061] First, let's explain the method that uses the size of the text in the image. For example, the second recognition unit 514 recognizes that the target item has been picked up if the size of the text in the image captured by the first camera 56 has not changed for a predetermined time, and the size of the text in the image exceeds a predetermined size.

[0062] Figure 12 illustrates an example of the process for recognizing that the forklift 5 has picked up an item. Figure 12(1) shows the case where the size of the string WD in the image is smaller than a predetermined size. In this case, the second recognition unit 514 does not recognize that the target item has been picked up, even if the size of the string WD in the image captured by the first camera 56 has not changed for a predetermined time.

[0063] On the other hand, Figure 12(2) shows the case where the size of the text string in the image is a predetermined size. In this case, the second recognition unit 514 recognizes that the target item has been picked up if the size of the text string WD in the image captured by the first camera 56 has not changed for a predetermined time.

[0064] Next, we will describe a method that uses known object recognition techniques. For example, the second recognition unit 514 recognizes that the target item has been picked up if the size of the text in the image captured by the first camera 56 has not changed for a predetermined time, and the object "paper" is recognized in the image.

[0065] Figure 13 illustrates an example of the process for recognizing that the forklift 5 has picked up an item. In Figure 13, frame OR indicates that the paper PR has been recognized as an object. In other words, (3) in Figure 13 represents a case where the object "paper" is not recognized in the image captured by the first camera 56. In this case, the second recognition unit 514 does not recognize that the target item has been picked up, even if the size of the string WD in the image captured by the first camera 56 has not changed for a predetermined time.

[0066] On the other hand, (4) in Figure 13 shows the case where the first camera 56 recognizes an object called "paper" in the image it has captured. In this case, the second recognition unit 514 recognizes that the target item has been picked up if the size of the string WD in the image captured by the first camera 56 has not changed for a predetermined time.

[0067] Next, a method using the position of text within an image will be described. For example, the second recognition unit 514 recognizes that an object has been picked up if the size of the text in the image captured by the first camera 56 has not changed for a predetermined time, and the text in the image is located in a predetermined position.

[0068] Figure 14 illustrates an example of the process for recognizing that the forklift 5 has picked up item ME. In Figure 14, image PI represents an image captured by the first camera 56. Figure 14(5) shows a case where the position of the string WD in image PI is not in a predetermined position (in this example, the center of the image). In this case, the second recognition unit 514 does not recognize that the target item has been picked up, even if the size of the string WD in the image captured by the first camera 56 has not changed for a predetermined time.

[0069] On the other hand, (6) in Figure 14 shows the case where the string WD in image PI is in a predetermined position. In this case, the second recognition unit 514 recognizes that the target item has been picked up if the size of the string WD in the image captured by the first camera 56 has not changed for a predetermined time.

[0070] Next, a method using the distance from the forklift 5 will be described. For example, an optical sensor LS having a light source and a light receiving unit is provided on the backrest 72 of the forklift 5. The light source of the distance measuring sensor LS irradiates the item ME with laser light LR. The light receiving unit of the optical sensor LS receives the reflected light of the laser light LR and detects the distance to the item ME based on the amount of light received.

[0071] For example, the second recognition unit 514 recognizes that the target item has been picked up if the size of the text in the image captured by the first camera 56 has not changed for a predetermined time, and the distance to the item ME detected by the optical sensor LS is within a predetermined range.

[0072] The above methods may be used individually or in combination. Furthermore, if the forklift 5 is equipped with a sensor that detects when an object is lifted, the detection results from this sensor may be used to reduce the possibility of misrecognition by the second recognition unit 514.

[0073] Returning to Figure 9, let's continue the explanation. The third recognition unit 515 recognizes the action of the forklift 5 unloading the target item. The third recognition unit 515 is an example of the second recognition unit. For example, the third recognition unit 515 recognizes that the forklift 5 has unloaded the target item based on the size of the string obtained by the first recognition unit 513 as the item name of the target item in the image captured by the first camera 56, which is acquired by the first acquisition unit 512. Unloading the target item is an example of the second action.

[0074] The following describes the process by which the third recognition unit 515 recognizes that the forklift 5 has unloaded the target item, using Figures 16 and 17. Figure 16 is a diagram illustrating an example of the process of recognizing that an item has been unloaded from the forklift 5.

[0075] Figure 16 shows the forklift 5 lowering its forks 71 to unload the item ME, and then moving away from the item ME to remove the forks 71 from the pallet PT. In Figure 16, the item ME is loaded onto the pallet PT, and a paper label PR with a string of characters representing the name of the item ME is attached to the item ME.

[0076] The forklift 5 lowers its forks 71, then reverses and removes the forks 71 from the pallet PT, thereby unloading the item ME from the forklift 5. At this time, the size of the text written on the paper PR in the image captured by the first camera 56, which is mounted on the backrest 72 of the forklift 5, decreases as the forklift 5 moves away from the item ME.

[0077] Therefore, the third recognition unit 515 recognizes that the forklift 5 has unloaded the item when the reduction rate of the string size exceeds a predetermined percentage, based on the size of the string at the point when the string size begins to shrink. Here, Figure 17 is a diagram illustrating an example of the process of recognizing that an item has been unloaded from the forklift 5. Figure 17 shows the time-series change in the size of the string WD written on paper PR in the image captured by the first camera 56.

[0078] The horizontal axis in Figure 17 represents the passage of time. In Figure 17, the forklift 5 lowers its forks 71 in accordance with the operator's instructions and begins to move away from the item ME at time t3 in order to remove the forks 71 from the pallet PT on which the item ME is loaded.

[0079] Until time t3, the pallet PT remains inserted into the fork 71. Therefore, as shown in Figure 17, the size of the string WD written on the paper PR in the image captured by the first camera 56 does not change until before time t3 has elapsed.

[0080] On the other hand, from time t3 onward, the forklift 5 moves away from the item ME in order to remove the forks 71 from the pallet PT. As a result, as shown in Figure 17, from time t3 onward, the size of the text WD written on the paper PR in the image captured by the first camera 56 decreases.

[0081] For example, the third recognition unit 515 recognizes the time (time t3 in Figure 17) when the size of the string WD written on the paper PR begins to shrink, based on the image captured by the first camera 56, which is acquired in real time. Then, using the size of the string WD at that time as a reference, the third recognition unit 515 recognizes that the target item has been lowered when it recognizes that the reduction rate of the size of the string WD written on the paper PR exceeds a predetermined percentage (when it recognizes that time t4 has passed in Figure 17).

[0082] In this embodiment, the second recognition unit 514 and the third recognition unit 515 are treated as separate functional units, but the functions of the second recognition unit 514 and the third recognition unit 515 may be integrated and configured as a single functional unit.

[0083] Returning to Figure 9, let's continue the explanation. The second acquisition unit 516 acquires the position information of the item when it recognizes the operation of the forklift 5. For example, the second acquisition unit 516 acquires position information representing the position of the target item based on the position of the forklift 5 at the time the target item is unloaded.

[0084] As an example, the second acquisition unit 516 acquires the location at the time the third recognition unit 515 recognizes that the item has been unloaded, based on the two-dimensional code for location recognition included in the image captured by the second camera 57 acquired by the first acquisition unit 512. The two-dimensional code is an example of a code symbol. The location acquisition process will be explained below with reference to Figure 18.

[0085] Figure 18 illustrates an example of the location acquisition process. In Figure 18, a shelf SF forms an area TA for storing items. An item ME is placed in area TA, and a paper PR with the string "12345" written as WD is attached to the front of the item ME.

[0086] Furthermore, a two-dimensional code TQ, which encodes the location of Area TA, is placed in the upper left corner of Area TA. Also, a two-dimensional code NQ, which encodes the location of the storage area to the right of Area TA, is placed in the upper right corner of Area TA. Note that barcodes may be used instead of two-dimensional codes.

[0087] For example, when the third recognition unit 515 recognizes that the target item has been unloaded based on the size of the string WD written on the paper PR, the second acquisition unit 516 reads the two-dimensional code located in the upper left region of the image captured by the second camera 57, which is acquired in real time by the first acquisition unit 512. As an example, in Figure 18, the second acquisition unit 516 reads the two-dimensional code TQ and acquires the location corresponding to that two-dimensional code.

[0088] As described above, the location where the target item was unloaded, that is, the location representing the target item, is transmitted by the communication control unit 511 to the server device 1 along with the item name acquired by the first recognition unit 513 as the name of the target item. In this case, the communication control unit 511 is an example of an output unit. The location and item name received by the communication control unit 411 of the server device 1 are associated by the storage control unit 412 and stored as management information 441.

[0089] Furthermore, the second acquisition unit 516 may continuously acquire location data while the forklift 5 is moving.

[0090] For example, if the image captured by the second camera 57 contains a two-dimensional code, the second acquisition unit 516 will read the code regardless of whether the target item has been unloaded. Furthermore, if multiple two-dimensional codes exist in the image captured by the second camera 57, the second acquisition unit 516 will prioritize reading the one located further to the left in the image. This makes it possible to constantly know the current location of the forklift 5.

[0091] Furthermore, the program that causes the camera device 2 to perform the processing necessary to realize the above functions can be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, floppy disk, CD-R, or DVD. The program may also be provided or distributed via a network such as the Internet.

[0092] Returning to Figure 1, let's continue the explanation. The worker terminal 3 is a device that supports the work of workers performing tasks in the warehouse managed by the server device 1. The configuration of the worker terminal 3 will be explained below using Figures 19 and 20.

[0093] First, the hardware configuration of the worker terminal 3 will be described. Figure 19 is a block diagram showing an example of the hardware configuration of the worker terminal 3 according to the first embodiment. The worker terminal 3 includes a CPU 61, RAM 62, ROM 63, storage 64, communication I / F 65, input I / F 66, and display 67. These parts 61 to 67 are connected to each other via a data bus.

[0094] The CPU 61 provides comprehensive control over the worker terminal 3. For example, the CPU 61 uses the RAM 62 as a workspace and executes various processes by running programs stored in the ROM 63, storage 64, etc. The storage 64 is a non-volatile memory that allows data to be written to and rewritten.

[0095] Communication I / F 65 is a device for establishing communication in a predetermined format with server device 1 and camera device 2. Input I / F 66 is a device that enables the acceptance of input from workers. Examples of input I / F 66 include touch panels. Display 67 is a display device that displays various information. For example, display 67 displays the item list screen and item search screen generated by server device 1.

[0096] Next, the functional configuration of the worker terminal 3 will be described. Figure 20 is a block diagram showing an example of the functional configuration of the worker terminal 3 according to the first embodiment. In this embodiment, the CPU 61 of the worker terminal 3 realizes various functions of the worker terminal 3 by executing programs stored in the storage 64, etc. The CPU 61 has a communication control unit 611, a display control unit 612, and a verification unit 613.

[0097] The communication control unit 611 controls the transmission and reception of information between the server device 1 and the camera device 2. For example, the communication control unit 611 receives various instructions from the server device 1 via the communication interface 65, such as instructions for receiving goods, instructions for shipping goods, and instructions for moving goods. Similarly, the communication control unit 611 also receives various notifications from the camera device 2, such as notifications that an item has been picked up or that an item has been unloaded.

[0098] Furthermore, for example, the communication control unit 611 also sends a matching completion notification to the camera device 2 when it has completed matching the item name of the item instructed to be shipped or moved with the item name notified by the camera device 2.

[0099] The display control unit 612 controls the display of various information on the display device. For example, the display control unit 612 controls the display of the item list screen, item search screen, etc., generated by the server device 1 on the display 67.

[0100] Furthermore, when the display control unit 612 receives notification from the camera device 2 that an item has been picked up, it controls the display 67 to display a message indicating that the item has been picked up. Also, when the display control unit 612 receives notification from the camera device 2 that an item has been put down, it controls the display 67 to display a message indicating that the item has been put down.

[0101] The matching unit 613 compares the name of the item instructed to be shipped or moved from the server device 1 with the name of the item recognized as having been picked up by the camera device 2. For example, when the matching unit 613 receives a notification from the camera device 2 that an item has been picked up, it compares the name of the item included in the notification with the name of the item instructed to be shipped or moved from the server device 1.

[0102] If the item name does not match as a result of the matching, the matching unit 613 may cooperate with the display control unit 612 to display a warning message on the display 67 indicating that an item other than the one instructed to be shipped or moved may have been picked up.

[0103] Furthermore, the program that causes the worker terminal 3 to execute the processes necessary to realize the above functions can be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, floppy disk, CD-R, or DVD. The program may also be provided or distributed via a network such as the Internet.

[0104] Next, we will explain the processes performed by the information processing system S. First, we will explain the process of receiving goods. Figure 21 is a sequence diagram showing an example of the processes performed by the information processing system S according to the first embodiment. Figure 21 shows an example of the process when a worker performs the receiving process.

[0105] The first acquisition unit 512 of the camera device 2 acquires an image captured by the first camera 56 (step ST1). The first acquisition unit 512 also acquires an image captured by the second camera 57 (step ST2). In Figure 21, the process of acquiring an image captured by the first camera 56 is described as step ST1, and the process of acquiring an image captured by the second camera 57 is described as step ST2, but these processes are assumed to be executed continuously.

[0106] Furthermore, the communication control unit 411 of the server device 1 transmits an item arrival instruction to the worker terminal 3 (step ST3). For example, the communication control unit 411 transmits the item arrival instruction to the worker terminal 3 via the communication I / F 45. The communication control unit 611 of the worker terminal 3 receives the item arrival instruction via the communication I / F 65.

[0107] The worker operates forklift 5 according to the goods receiving instructions, moving forklift 5 to the receiving berth. The worker also operates forklift 5, inserting the forks 71 into pallets PT loaded with goods that are placed at the receiving berth.

[0108] The second recognition unit 514 of the camera device 2 recognizes that an item has been picked up based on the image captured by the first camera 56 acquired in step ST1 (step ST4). For example, the first recognition unit 513 recognizes the string WD representing the item name of the item, which is written on the paper PR attached to the front of the item, and acquires the string WD as the item name of the item to be received.

[0109] Furthermore, the second recognition unit 514 recognizes that an item has been picked up when a predetermined time has elapsed since the size of the string WD in the image captured by the first camera 56 stopped changing.

[0110] Next, the second acquisition unit 516 acquires the location information of the item at the time it recognizes that the item has been picked up (step ST5). For example, at the time the second recognition unit 514 recognizes that the item has been picked up in step ST4, the second acquisition unit 516 reads the two-dimensional code contained in the image captured by the second camera 57. The second acquisition unit 516 acquires the location corresponding to the two-dimensional code as the location representing the current location of the item.

[0111] Furthermore, the communication control unit 511 sends a notification to the worker terminal 3 indicating that it has recognized that an item has been picked up (step ST6). For example, if the second recognition unit 514 recognizes in step ST4 that an item has been picked up, the communication control unit 511 sends a notification to the worker terminal 3 via the communication interface 55 indicating that it has recognized that an item has been picked up. This notification includes the name of the item to be received, which has been acquired by the first recognition unit 513.

[0112] In Figure 21, the processing of step ST6 is shown as processing after step ST5, but the processing of step ST6 may be performed in parallel with the processing of step ST5, or it may be performed after step ST4 and before the processing of step ST5.

[0113] The communication control unit 611 of the worker terminal 3 receives a notification from the camera device 2 that it has recognized that an item has been picked up. Next, the display control unit 612 displays on the display 67 that an item has been picked up (step ST7).

[0114] For example, the display control unit 612 displays on the display 67 the name of the item to be received and a statement that the item has been picked up, as included in the notification. This allows the worker to understand that it is now OK to move the forklift 5. After this, the worker operates the forklift 5 and proceeds to transport the item.

[0115] Furthermore, the communication control unit 511 of the camera device 2 sends a notification to the server device 1 that the item to be received is in transit (step ST8). For example, the communication control unit 511 sends a notification to the server device 1 via the communication I / F 55 that the item to be received is in transit. This notification includes the name of the item to be received and its location.

[0116] The communication control unit 411 of server device 1 receives a notification that the item to be received is in transit. Next, the storage control unit 412 registers that the item to be received is in transit (step ST9). For example, the storage control unit 412 associates the item name and location included in the notification that the item to be received is in transit, and further adds the status of "in transit" before storing it in the management information 441.

[0117] The worker then moves forklift 5 to search for an empty area in the warehouse. Once an empty area is found, the worker operates forklift 5 to lower forks 71. After that, the worker operates forklift 5 to remove forks 71 from pallet PT.

[0118] In this example, the worker is visually searching for an empty area in the warehouse, but the server device 1 may also specify the storage location for the incoming goods. For example, the server device 1 refers to the management information 441, identifies an area where no goods are stored, and designates one of those areas as the storage location. In this case, the communication control unit 411 transmits the storage location information to the worker terminal 3 along with the goods incoming instruction.

[0119] Next, the second recognition unit 514 of the camera device 2 recognizes that the item has been lowered based on the image captured by the first camera 56 acquired in step ST1 (step ST10). For example, the second recognition unit 514 recognizes that the item has been lowered when the reduction rate of the size of the string WD in the image captured by the first camera 56 exceeds a predetermined percentage.

[0120] Next, the second acquisition unit 516 acquires the location information of the item at the time it recognizes that the item has been unloaded (step ST11). For example, at the time the second recognition unit 514 recognizes in step ST11 that the item has been unloaded, the second acquisition unit 516 reads the two-dimensional code contained in the image captured by the second camera 57. The second acquisition unit 516 acquires the location corresponding to the two-dimensional code as the location representing the current position of the item.

[0121] Furthermore, the communication control unit 511 sends a notification to the worker terminal 3 indicating that it has recognized that the item has been unloaded (step ST12). For example, if the second recognition unit 514 recognizes in step ST11 that the item has been unloaded, the communication control unit 511 sends a notification to the worker terminal 3 via the communication I / F 55 indicating that it has recognized that the item has been unloaded. This notification includes the name of the item to be received, which has been acquired by the first recognition unit 513.

[0122] In Figure 21, the processing in step ST12 is shown as processing after step ST11, but the processing in step ST12 may be performed in parallel with the processing in step ST11, or it may be performed after step ST10 and before the processing in step ST11.

[0123] The communication control unit 611 of the worker terminal 3 receives a notification from the camera device 2 that it has recognized that the item has been lowered. Next, the display control unit 612 displays on the display 67 that the item has been lowered (step ST13).

[0124] For example, the display control unit 612 displays on the display 67 the name of the item to be received and a statement that the item has been unloaded, as included in the notification. This allows the worker to understand that the receiving process for the item has been completed.

[0125] Furthermore, the communication control unit 511 of the camera device 2 sends a delivery completion notification to the server device 1 indicating that the delivery of the goods has been completed (step ST14). For example, the communication control unit 511 sends the delivery completion notification to the server device 1 via the communication I / F 55. The notification includes the name of the goods to be delivered and the location.

[0126] The communication control unit 411 of server device 1 receives the arrival completion notification. Next, the storage control unit 412 registers the arrival of the goods (step ST15) and terminates this process. For example, the storage control unit 412 associates the item name and location included in the arrival completion notification, erases the status of "in transit," and stores it in the management information 441.

[0127] Next, the shipping process for goods will be described. Figure 22 is a sequence diagram showing an example of the process performed by the information processing system S according to the first embodiment. Figure 22 shows an example of the process when a worker performs the shipping process. Steps ST31 and ST32 are the same as steps ST1 and ST2 in Figure 21, so their explanation will be omitted.

[0128] The communication control unit 411 of the server device 1 sends a shipping instruction for the goods to the worker terminal 3 (step ST33). For example, the communication control unit 411 sends a shipping instruction for the goods to the worker terminal 3 via the communication interface 45. The shipping instruction for the goods includes the name of the goods to be shipped and the location where the goods are stored. The communication control unit 611 of the worker terminal 3 receives the shipping instruction for the goods via the communication interface 65.

[0129] Next, the display control unit 612 displays the item name and location information related to the item shipment instruction on the display 67 (step ST34). For example, the display control unit 612 displays the item name of the item to be shipped and the location where the item is stored, which are included in the item shipment instruction, on the display 67. This makes it easier for workers to understand information about the items to be shipped.

[0130] The worker operates the forklift 5 according to the shipping instructions for the goods, moving it to the location displayed on the screen in step ST34. The worker also operates the forklift 5 to insert the forks 71 into the pallet PT loaded with the goods stored at that location.

[0131] Steps ST35 to ST37 are the same as steps ST4 to ST6 in Figure 21, so their explanation is omitted. After step ST37, the matching unit 613 matches the item name of the item related to the shipping instruction with the item name related to the notification that the item has been picked up (step ST38). For example, the matching unit 613 matches the item name included in the shipping instruction for the item transmitted from the server device 1 with the item name included in the notification that the item has been picked up.

[0132] Although not shown in the diagram, if the item name does not match as a result of the comparison, the comparison unit 613 may work with the display control unit 612 to display a message on the display 67 warning the worker that they may have picked up an item other than the one specified in the shipping instruction. This prevents the wrong item from being transported to the shipping berth.

[0133] If the item names match as a result of the matching, the communication control unit 611 sends a matching completion notification to the camera device 2 via the communication interface 65 (step ST39). The communication control unit 511 of the camera device 2 receives the matching completion notification via the communication interface 55. Steps ST40 to ST46 are the same as steps ST7 to ST13 in Figure 21, so their explanation is omitted.

[0134] During the shipping process, the display control unit 612 displays on the display 67 that the goods have been unloaded, allowing the worker to know that the goods related to shipping have been unloaded at the shipping berth.

[0135] The communication control unit 511 of the camera device 2 sends a shipment completion notification to the server device 1 indicating that the shipment of the goods has been completed (step ST47). For example, the communication control unit 511 sends the shipment completion notification to the server device 1 via the communication I / F 55. The notification includes the name of the goods to be received and a location indicating the location of the shipment berth.

[0136] The communication control unit 411 of server device 1 receives a shipment completion notification. Next, the storage control unit 412 deletes the information of the items related to the shipment from the management information 441 (step ST48), and terminates this process. For example, the storage control unit 412 deletes the information of the items corresponding to the shipment instruction sent in step ST33 from the management information 441.

[0137] Next, the process of moving items will be described. Figure 23 is a sequence diagram showing an example of the process performed by the information processing system S according to the first embodiment. Figure 23 shows an example of the process when a worker performs the movement process. Steps ST51 and ST52 are the same as steps ST31 and ST32 in Figure 22, so their explanation is omitted.

[0138] The communication control unit 411 of the server device 1 transmits an item movement instruction to the worker terminal 3 (step ST53). For example, the communication control unit 411 transmits the item movement instruction to the worker terminal 3 via the communication interface 45. The item movement instruction includes the name of the item to be moved, the location where the item is stored, and the location representing the destination. The communication control unit 611 of the worker terminal 3 receives the item movement instruction via the communication interface 65.

[0139] Next, the display control unit 612 displays the name of the item to be moved, the current location information of the item, and the location information of the destination on the display 67 (step ST54). For example, the display control unit 612 displays on the display 67 the name of the item to be moved, the location where the item is stored, and the location representing the destination, which are included in the item move instruction. This makes it easier for the worker to grasp information about the item to be moved.

[0140] The worker operates the forklift 5 according to the instructions for moving the goods, moving it to the location shown on the display in step ST54, which represents the current position of the goods to be moved. The worker also operates the forklift 5 to insert the forks 71 into the pallet PT loaded with the goods stored at that location.

[0141] Steps ST55 to ST66 are the same as steps ST35 to ST46 in Figure 22, so their explanation is omitted. During the movement process, the display control unit 612 displays on the display 67 that the item has been unloaded, so that the worker can understand that the item involved in the movement has been unloaded in the destination area.

[0142] The communication control unit 511 of the camera device 2 sends a move completion notification to the server device 1 indicating that the movement of the item has been completed (step ST67). For example, the communication control unit 511 sends the move completion notification to the server device 1 via the communication I / F 55. The notification includes the name of the item to be moved and a location indicating the location of the shipping berth.

[0143] The communication control unit 411 of server device 1 receives a move completion notification. Next, the storage control unit 412 registers the movement of the item (step ST68) and terminates this process. For example, the storage control unit 412 associates the item name and location included in the move completion notification, erases the status of "in transit," and stores it in the management information 441.

[0144] As described above, the camera device 2 according to this embodiment recognizes symbols such as strings representing the name of an item based on the image captured by the camera, and recognizes that an item has been picked up and put down based on the size change of the symbols in the image. The camera device 2 also acquires location information representing the position where the item was put down and outputs the item name and location information in association.

[0145] As a result, the camera device 2 according to this embodiment can automatically recognize when an item has been picked up and when an item has been put down simply by taking an image with the camera. Furthermore, by recognizing that an item has been picked up, the camera device 2 according to this embodiment can recognize that an item has been moved. Moreover, by recognizing that an item has been put down and acquiring location information representing the location where the item was put down, the camera device 2 according to this embodiment can recognize that an item has been placed. Incidentally, when items are placed in a free-location system, it is important to be aware of when items have been moved and when items have been placed in order to properly manage the items. Since the camera device 2 according to this embodiment can automatically recognize when an item has been picked up and when an item has been put down, workers can be aware of when items have been moved and when items have been placed in order without having to make the judgment themselves. In other words, according to this embodiment, the convenience of item management in a warehouse where items are placed in a free-location system can be improved.

[0146] (Second Embodiment) In the first embodiment described above, a configuration in which a worker operates the forklift 5 to receive and ship goods was explained. In the second embodiment, a configuration in which an unmanned forklift receives and ships goods will be explained.

[0147] In the following, we will mainly describe the differences from the embodiments described above, and will omit detailed explanations of points that are common to those already described. Furthermore, each embodiment described below may be implemented individually or in combination as appropriate.

[0148] Figure 24 shows an example of the overall configuration of the information processing system Sa according to the second embodiment. The information processing system Sa comprises a server device 11, a camera device 21, and a drive control device. The server device 1, the camera device 2, and the worker terminal 3 are connected to each other via a network N so that they can communicate with one another. The configuration of the server device 11 will be described below using Figures 25 and 26.

[0149] First, the hardware configuration of the server device 11 will be described. Figure 25 is a block diagram showing an example of the hardware configuration of the server device 11 according to the second embodiment. The server device 11 has the same configuration as the server device 1 in Figure 2, but differs from the server device 1 in Figure 2 in that it has a CPU 47 instead of a CPU 41.

[0150] Next, the functional configuration of the server device 11 will be described. Figure 26 is a block diagram showing an example of the functional configuration of the server device 11 according to the second embodiment. In this embodiment, the CPU 47 of the server device 1 executes programs stored in the storage 44, etc., to realize various functions of the server device 11.

[0151] The CPU 47 according to this embodiment includes a communication control unit 471, a storage control unit 472, and an instruction unit 473. The storage 44 also includes management information 441. The storage control unit 472 and the instruction unit 473 are the same as the storage control unit 412 and the instruction unit 413 in Figure 3, so their description is omitted.

[0152] The communication control unit 471 controls the transmission and reception of information with the camera device 21. For example, the communication control unit 471 transmits instructions for receiving goods, shipping goods, moving goods, etc., to the camera device 21 via the communication I / F 45. Similarly, the communication control unit 471 receives various notifications from the camera device 21, such as notifications that goods are in motion, that goods have been received, and that goods have been shipped.

[0153] Next, the configuration of the camera device 21 will be described using Figures 27 and 28. Figure 27 is a block diagram showing an example of the hardware configuration of the camera device according to the first embodiment. The camera device 21 has a similar configuration to the camera device 2 in Figure 6, but differs from the camera device 2 in that the controller 50 includes a CPU 58 and a distance measuring sensor 59.

[0154] The distance measuring sensor 59 detects objects in the vicinity of the forklift 5. The distance measuring sensor 59 also detects the distance to the object. The distance measuring sensor 59 is, for example, a sensor device that uses ultrasound to detect objects and measure distance, or a LiDAR (Light Detection and ranging) sensor device that uses laser light to detect objects.

[0155] Next, the functional configuration of the camera device 21 will be described. Figure 28 is a block diagram showing an example of the functional configuration of the camera device 21 according to the second embodiment. In this embodiment, the CPU 58 of the camera device 21 realizes various functions of the camera device 21 by executing a program stored in the storage 54, etc.

[0156] The CPU 58 includes a communication control unit 581, a first acquisition unit 582, a first recognition unit 583, a second recognition unit 584, a third recognition unit 585, a second acquisition unit 586, a search unit 587, a matching unit 588, and a determination unit 589. The first acquisition unit 582, first recognition unit 583, second recognition unit 584, third recognition unit 585, and second acquisition unit 586 are the same as the first acquisition unit 512, first recognition unit 513, second recognition unit 514, third recognition unit 515, and second acquisition unit 516 in Figure 9, so their explanation is omitted.

[0157] The communication control unit 581 controls the transmission and reception of information between the server device 11 and the drive control device 4. For example, the communication control unit 581 receives instructions for receiving goods, instructions for shipping goods, instructions for moving goods, etc., from the server device 1 via the communication interface 55.

[0158] Furthermore, for example, the communication control unit 581 similarly transmits various requests to the drive control device 4, such as a move request requesting that the forklift 5 be moved to a specific location in the warehouse. Also, for example, the communication control unit 511 transmits a notification to the drive control device 4 that an item has been picked up or unloaded.

[0159] The search unit 587 searches the area around the forklift 5. For example, based on the sensing results of the distance sensor 59, the search unit 587 searches the area around the forklift 5 for areas where goods can be stored (hereinafter also referred to as empty shelves). The information of areas recognized as empty shelves as a result of the search by the search unit 587 may be stored in the storage 54 or the like. In this case, the search unit 587 may take this information into consideration when searching for empty shelves.

[0160] Furthermore, for example, the search unit 587 searches for goods. As an example, during the receiving process of goods, when the forklift 5 moves to the receiving berth, the search unit 587 searches for the goods to be received based on the sensing results of the distance measuring sensor 59. Here, the fact that the forklift 5 has moved to the receiving berth can be recognized, for example, by the second acquisition unit 586 acquiring the location in real time.

[0161] As another example, during the shipment or movement of goods, the search unit 587 searches for the goods to be shipped or moved based on the sensing results of the distance measuring sensor 59 when the forklift 5 moves to the location of the goods to be shipped or moved.

[0162] In this embodiment, when the search unit 587 detects the presence of an object that is considered to be an article based on the sensing results of the distance measuring sensor 59, the first recognition unit 583 performs a process to acquire the name of the article.

[0163] The matching unit 588 compares the name of the item instructed to be shipped or moved by the server device 11 with the name of the item found as a result of the search. For example, when the search unit 587 finds an item, the matching unit 588 compares the name of the item obtained by the first recognition unit 583 with the name of the item instructed to be shipped or moved by the server device 11.

[0164] If the item name does not match as a result of the matching, the matching unit 588 may cooperate with the communication control unit 581 to send a request to the drive control device 4 to unload the item. If the forklift 5 unloads the item under the control of the drive control device 4, the search unit 587 performs the item search process again.

[0165] The determination unit 589 determines whether there is enough space to unload the goods. For example, when processing incoming goods, if the forklift 5 moves to a location recognized as an empty shelf, the determination unit 589 determines whether there is enough space to unload the goods based on the sensing results of the distance measuring sensor 59.

[0166] Here, even if the search unit 587 recognizes an area as an empty shelf, it is possible that items may be placed in that area after recognition, or that workers or others may place other objects that are not items to be managed in the warehouse in that area. For this reason, in this embodiment, the determination unit 589 determines whether there is space to unload the items before actually unloading them.

[0167] Returning to Figure 24, let's continue the explanation. The drive control device 4 is a device that controls the drive of the forklift 5. The configuration of the drive control device 4 will be explained below using Figures 29 and 30.

[0168] First, the hardware configuration of the drive control device 4 will be described. Figure 29 is a block diagram showing an example of the hardware configuration of the drive control device 4 according to the second embodiment. The drive control device 4 includes a CPU 81, RAM 82, ROM 83, storage 84, and communication I / F 85. These parts 81 to 85 are connected to each other via a data bus.

[0169] The CPU 81 comprehensively controls the drive control device 4. For example, the CPU 81 uses the RAM 82 as a working area and executes various processes by running programs stored in the ROM 83, storage 84, etc. The storage 84 is a non-volatile memory that allows data to be written to and rewritten. The communication interface 85 is a device for establishing communication with the camera device 21 in a predetermined format.

[0170] Next, the functional configuration of the drive control device 4 will be described. Figure 30 is a block diagram showing an example of the functional configuration of the drive control device 4 according to the second embodiment. In this embodiment, the CPU 81 of the drive control device 4 realizes various functions of the drive control device 4 by executing a program stored in the storage 84, etc. The CPU 81 has a communication control unit 811 and a drive control unit 812.

[0171] The communication control unit 811 controls the transmission and reception of information with the camera device 21. For example, the communication control unit 811 receives various requests from the camera device 21 via the communication interface 85, such as requests to move items to the receiving berth, requests to move items to the shipping berth, and requests to move items to the storage location.

[0172] Furthermore, for example, the communication control unit 811 similarly receives various notifications from the camera device 21, such as notifications that an item has been picked up, notifications that an item has been put down, notifications that an item has been found, and notifications that permission has been granted to put down an item.

[0173] The drive control unit 812 controls the drive of the forklift 5. For example, the drive control unit 813 controls the drive wheels 75 and steering wheels 76 of the forklift 5 in accordance with a request to move to the receiving berth transmitted from the camera device 21, to move the forklift 5 from its current location to the receiving berth.

[0174] For example, a request to move to an incoming berth includes information representing the route from the forklift 5's current location to the incoming berth, and the drive control unit 812 controls the drive wheels 75 and steering wheels 76 so that the forklift 5 moves along this route. The same process is performed when moving to an outgoing berth or to a shelf that is the destination of the movement.

[0175] Furthermore, when the drive control unit 812 receives notification from the camera device 21 that an item has been found, it controls the pickup of the item. For example, the drive control unit 812 inserts the forks 71 into the pallet on which the item is loaded, and controls the movement of the mast 73 to raise the position of the forks 71, thereby picking up the item. In this case, the sensing results of the distance sensor 59 of the camera device 21 may be used to accurately insert the forks 71 into the pallet.

[0176] Furthermore, the drive control unit 812 controls the lowering of the item when it receives a notification from the camera device 21 granting permission to lower the item. For example, the drive control unit 812 lowers the item by controlling the movement of the mast 73 to lower the position of the forks 71, and then controls the forklift 5 to move backward.

[0177] Furthermore, the program that causes the drive control device 4 to execute the processing necessary to realize the above functions can be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, floppy disk, CD-R, or DVD. This program may also be provided or distributed via a network such as the Internet.

[0178] Next, we will describe the processes performed by the information processing system Sa. First, we will describe the process of receiving goods. Figure 31 is a sequence diagram showing an example of the processes performed by the information processing system Sa according to the second embodiment. Figure 31 shows an example of the process when an unmanned, automatically operated forklift 5 performs the receiving process. Steps ST71 and ST72 are the same as steps ST1 and ST2 in Figure 21, so their explanation will be omitted.

[0179] The search unit 587 of the camera device 21 searches for empty shelves (step ST73). For example, the search unit 587 searches for empty shelves around the forklift 5 based on the sensing results of the distance sensor 59.

[0180] In Figure 31, the process of searching for empty shelves is described as step ST73, but the process in step ST73 is to be executed continuously. In addition, information on areas recognized as empty shelves as a result of the search is stored as empty shelf information in storage 55, etc.

[0181] Furthermore, the communication control unit 471 of the server device 11 transmits an item arrival instruction to the camera device 21 (step ST74). For example, the communication control unit 471 transmits the item arrival instruction to the camera device 21 via the communication I / F 45. The communication control unit 581 of the camera device 21 receives the item arrival instruction via the communication I / F 65.

[0182] Next, the communication control unit 581 requests the forklift 5 to move to the receiving berth (step ST75). For example, the communication control unit 581 transmits a request to move to the receiving berth via the communication I / F 55, which includes information representing the route from the forklift 5's current location to the receiving berth. The communication control unit 811 of the drive control device 4 receives this request to move.

[0183] Next, the drive control unit 812 controls the forklift 5 to move to the receiving berth (step ST76). For example, based on the information representing the route from the current location to the receiving berth included in the movement request received from the camera device 21, the drive control unit 812 controls the drive wheels 75 and steering wheels 76 to move the forklift 5 to the receiving berth.

[0184] When forklift 5 moves to the receiving berth, the search unit 587 of the camera device 21 recognizes that forklift 5 has moved to the receiving berth (step ST77). For example, based on the location acquired in real time by the second acquisition unit 586, the search unit 587 recognizes that forklift 5 has moved to the receiving berth.

[0185] Next, the search unit 587 searches for the item (step ST78). For example, the search unit 587 works in cooperation with the drive control device 4 to move the forklift 5 within the receiving berth and searches for the item based on the sensing results of the distance measuring sensor 59.

[0186] If an item is found as a result of the search, the first recognition unit 583 obtains the name of the item from the string of characters written on the paper attached to the item, based on the image captured by the first camera 56. At the same time, the second recognition unit 584 recognizes the size of the item in the image captured by the first camera 56 (step ST79).

[0187] Next, the communication control unit 581 sends a notification to the drive control device 4 via the communication interface 55 that an item has been found (step ST80). The communication control unit 811 of the drive control device 4 receives the notification that an item has been found. Upon receiving this notification, the drive control unit 812 controls the forklift 5 to pick up the item (step ST81).

[0188] For example, the drive control unit 812 moves the forklift 5 in front of the item and then controls the picking up the item by inserting the forks 71 into the pallet on which the item is loaded. At this time, the drive control unit 812 may use the size of the item recognized in step ST79 and the sensing results of the distance sensor 59 to move the forklift 5 to an appropriate position for picking up the item.

[0189] Steps ST82 to ST85 are the same as steps ST4, ST5, ST8, and ST9 in Figure 21, so their explanation is omitted. After step ST85, the communication control unit 581 requests the forklift 5 to move to an empty shelf (step ST86).

[0190] For example, the communication control unit 581 transmits a move request to an empty shelf via the communication interface 55, which includes information representing the route from the forklift 5's current location to the empty shelf. As an example, the communication control unit 581 refers to the empty shelf information and identifies the empty shelf closest to the forklift 5's current location (within the receiving berth). The communication control unit 581 transmits a move request that includes the route from the forklift 5's current location to the empty shelf. The communication control unit 811 of the drive control device 4 receives the move request.

[0191] Next, the drive control unit 812 controls the forklift 5 to move to an empty shelf (step ST87). For example, based on the information representing the route from the current location to the empty shelf included in the movement request received from the camera device 21, the drive control unit 812 controls the drive wheels 75 and steering wheels 76 to move the forklift 5 to an empty shelf.

[0192] When the forklift 5 moves to an empty shelf, the determination unit 589 of the camera device 21 recognizes that the forklift 5 has arrived at the empty shelf (step ST88). For example, the determination unit 589 recognizes that the forklift 5 has arrived at the empty shelf based on the location acquired in real time by the second acquisition unit 586.

[0193] Next, the determination unit 589 determines whether there is space to unload the item from the arriving empty shelf (step ST89). For example, the determination unit 589 determines whether there is space to unload the item based on the size of the item recognized in step ST79 and the sensing result of the distance measuring sensor 59.

[0194] Although not shown in the diagram, if it is determined that there is no space to unload the goods, the communication control unit 581 may refer to the available shelf information and identify the available shelf closest to the current location of the forklift 5. In this case, the communication control unit 581 may send a movement request to the drive control device 4 that includes information representing the route from the current location of the forklift 5 to the identified available shelf.

[0195] If the communication control unit 581 determines in step ST89 that there is space to unload the item, it sends a notification to the drive control device 4 via the communication interface 55 granting permission to unload the item (step ST90). The communication control unit 811 of the drive control device 4 receives the notification.

[0196] Upon receiving the notification, the drive control unit 812 performs control to unload the goods (step ST91). For example, the drive control unit 812 controls the operation of the mast 73 to lower the forks 71 and move the forklift 5 backward, thereby removing the forks 71 from the pallet on which the goods are loaded and unloading the goods.

[0197] Steps ST92 to ST95 are the same as steps ST10, ST11, ST14, and ST15 in Figure 21, so their explanation is omitted.

[0198] Next, the shipping process for goods will be described. Figure 32 is a sequence diagram showing an example of the process performed by the information processing system Sa according to the second embodiment. Figure 32 shows an example of the process when an unmanned, automatically operated forklift 5 performs the shipping process. Steps ST101 to ST103 are the same as steps ST71 to ST73 in Figure 31, so their explanation will be omitted.

[0199] The communication control unit 471 of server device 1 transmits an item shipment instruction to camera device 21 (step ST104). For example, the communication control unit 471 transmits the item shipment instruction to camera device 21 via communication I / F 45. The item shipment instruction includes the name of the item to be shipped and the location where the item is stored. The communication control unit 581 of camera device 21 receives the item shipment instruction via communication I / F 55.

[0200] Next, the communication control unit 581 requests the forklift 5 to move to the storage location of the goods that have been instructed to be shipped (step ST105). For example, the communication control unit 581 transmits a request to move to the storage location of the goods via the communication I / F 55, which includes information representing the route from the current location of the forklift 5 to the storage location of the goods. The communication control unit 811 of the drive control device 4 receives this request to move.

[0201] Next, the drive control unit 812 controls the forklift 5 to move to the storage location of the goods (step ST106). For example, based on the information representing the route from the current location to the storage location of the goods included in the movement request received from the camera device 21, the drive control unit 812 controls the drive wheels 75 and steering wheels 76 to move the forklift 5 to the storage location of the goods.

[0202] When the forklift 5 moves to the storage location of the goods, the search unit 587 of the camera device 21 recognizes that the forklift 5 has moved to the storage location of the goods (step ST107). For example, based on the location acquired in real time by the second acquisition unit 586, the search unit 587 recognizes that the forklift 5 has moved to the storage location of the goods.

[0203] Steps ST108 to ST113 are the same as steps ST78 to ST83 in Figure 31, so their explanation is omitted. After step ST113, the matching unit 588 compares the item name of the item related to the shipping instruction with the item name of the found item (step ST114). For example, the matching unit 613 compares the item name included in the shipping instruction for the item transmitted from the server device 1 with the item name acquired by the first recognition unit 583 during the item search.

[0204] Although not shown in the diagram, if the item names do not match as a result of the comparison, the search unit 587 may perform the item search process in step ST108 until an item name matches.

[0205] If the item names match as a result of the verification, the process proceeds to step ST115. Steps ST115 and ST116 are the same as steps ST84 and ST85 in Figure 31, so their explanation is omitted. After step ST116, the communication control unit 581 requests the forklift 5 to move to the shipping berth (step ST117).

[0206] For example, the communication control unit 581 transmits a move request to an empty shelf via the communication interface 55, which includes information representing the route from the current location of the forklift 5 to the shipping berth. As an example, the communication control unit 581 transmits a move request that includes the route from the current location of the forklift 5 to the shipping berth. The communication control unit 811 of the drive control device 4 receives the move request.

[0207] Next, the drive control unit 812 controls the forklift 5 to move to the shipping berth (step ST118). For example, based on the information representing the route from the current location to the shipping berth included in the movement request received from the camera device 21, the drive control unit 812 controls the drive wheels 75 and steering wheels 76 to move the forklift 5 to the shipping berth.

[0208] When the forklift 5 moves to the shipping berth, the determination unit 589 of the camera device 21 recognizes that the forklift 5 has arrived at the shipping berth (step ST119). For example, the determination unit 589 recognizes that the forklift 5 has arrived at the shipping berth based on the location acquired in real time by the second acquisition unit 586.

[0209] Next, the determination unit 589 determines whether there is space to unload the goods at the arrival shipping berth (step ST120). For example, the determination unit 589 determines whether there is space to unload the goods based on the size of the goods recognized in step ST109 and the sensing results of the distance measuring sensor 59.

[0210] Although not shown in the diagram, if the communication control unit 581 determines that there is no space to unload the goods, it may send a message to the administrator of the server device 11 or the warehouse administrator, etc., indicating that there is no space to place the goods for shipment at the shipping berth. Alternatively, the search unit 587 may work in cooperation with the drive control device 4 to search for space to unload the goods while moving the forklift 5 within the shipping berth.

[0211] Steps ST121 to ST124 are the same as steps ST90 to ST93 in Figure 31, so their explanation is omitted. Also, steps ST125 and ST126 are the same as steps ST47 and ST48 in Figure 22, so their explanation is omitted.

[0212] Next, the process of moving items will be described. Figure 33 is a sequence diagram showing an example of the process performed by the information processing system Sa according to the second embodiment. Figure 33 shows an example of the process when an unmanned, automatically operated forklift 5 performs the movement process. Steps ST131 to ST133 are the same as steps ST101 to ST103 in Figure 32, so their explanation will be omitted.

[0213] The communication control unit 471 of the server device 1 transmits an instruction to move an item to the camera device 21 (step ST134). For example, the communication control unit 471 transmits an instruction to move an item to the camera device 21 via the communication interface 45. The instruction to move an item includes the name of the item to be moved, the location where the item is stored, and the location where the item will be moved. The communication control unit 581 of the camera device 21 receives the instruction to move an item via the communication interface 55.

[0214] Steps ST135 to ST146 are the same as steps ST105 to ST116 in Figure 32, so their explanation is omitted. After step ST146, the communication control unit 581 requests the forklift 5 to move the goods to their destination (step ST147).

[0215] For example, the communication control unit 581 transmits a move request to a destination via the communication interface 55, which includes information representing the route from the current location of the forklift 5 to the destination. As an example, the communication control unit 581 transmits a move request from the current location of the forklift 5, which includes the route to the destination of the item included in the move instruction received from the server device 11. The communication control unit 811 of the drive control device 4 receives this move request.

[0216] Next, the drive control unit 812 controls the forklift 5 to move to the destination of the goods (step ST148). For example, based on the information representing the route from the current location to the destination of the goods included in the movement request received from the camera device 21, the drive control unit 812 controls the drive wheels 75 and steering wheels 76 to move the forklift 5 to the destination of the goods.

[0217] When the forklift 5 moves to the destination of the goods, the determination unit 589 of the camera device 21 recognizes that the forklift 5 has arrived at the destination of the goods (step ST149). For example, the determination unit 589 recognizes that the forklift 5 has arrived at the destination of the goods based on the location acquired in real time by the second acquisition unit 586.

[0218] Next, the determination unit 589 determines whether there is space to unload the item at the destination of the arrived item (step ST150). For example, the determination unit 589 determines whether there is space to unload the item based on the size of the item recognized in step ST139 and the sensing result of the distance measuring sensor 59.

[0219] Although not shown in the diagram, if the communication control unit 581 determines that there is no space to unload the items, it may send a message to the administrator of the server device 11 or the warehouse manager, etc., indicating that there is no space to place the items in the area designated as the destination.

[0220] Steps ST151 to ST154 are the same as steps ST121 to ST124 in Figure 32, so their explanation is omitted. Also, steps ST155 and ST156 are the same as steps ST67 and ST68 in Figure 23, so their explanation is omitted.

[0221] As described above, in the information processing system Sa according to this embodiment, the receiving, shipping, and moving of goods are performed by an unmanned, automatically operated forklift 5.

[0222] For example, when transporting goods using unmanned, automatically operated forklifts, etc., and placing the goods in a free-location manner, the actual location where the goods are placed may differ from the recorded location. This is thought to occur because the system may recognize that goods have been unloaded when they have not actually been unloaded, or vice versa. To prevent this, one possible solution is for a human to make the judgment regarding whether goods have been picked up and unloaded. However, this would negate the full benefits of automated goods transport, as it would require human judgment.

[0223] In contrast, the camera device 21 according to this embodiment can automatically recognize from the image captured by the camera that an item has been picked up and that an item has been put down. Therefore, according to the camera device 21 according to this embodiment, even if the transport of items is automated and items are placed in a free-location manner, the placement location (storage location) of the items can be accurately recorded without requiring human judgment. In other words, according to this embodiment, the convenience of item management in a warehouse where items are placed in a free-location manner can be improved.

[0224] Furthermore, the first and second embodiments described above can be modified and implemented as appropriate by changing the configuration or some of the functions of each device in the information processing system S(Sa). Therefore, several modifications of the above embodiments will be described below as other embodiments. In the following, we will mainly describe the differences from the embodiments described above, and will omit detailed explanations of points that are common with what has already been described. Also, the modifications described below may be implemented individually or in combination as appropriate.

[0225] (Variation 1) In the first and second embodiments described above, a method for acquiring location information was described based on a second camera 57 and a two-dimensional code for location recognition. In this modified example, a method for acquiring location information using a sensor such as a gyro sensor will be described.

[0226] In this modified example, the forklift 5 is equipped with a gyro sensor instead of the second camera 57. An example of the installation location of the gyro sensor in the forklift 5 will be described below using Figures 34 and 35.

[0227] Figures 34 and 35 are schematic diagrams showing an example of the camera installation position on the forklift 5 according to Modification 1. Figure 34 shows an example of the forklift 5 according to Modification 1 viewed from the side. Figure 35 shows an example of the forklift 5 according to Modification 1 viewed from the front.

[0228] As shown in Figures 34 and 35, the forklift 5 according to this modified example is equipped with a gyro sensor 70 on the head guard 74. Also, unlike the forklift 5 according to the first and second embodiments, the second camera 57 is not provided on the mast 73. The gyro sensor 70 is a sensor that measures rotational angular velocity. The gyro sensor 70 is an example of a position measurement sensor.

[0229] In this modified example, the second acquisition unit 516 (586) acquires position information representing the current location of the forklift 5 based on the sensing results of the gyro sensor 70, and acquires the position of the target item from this position information. For example, the second acquisition unit 516 (586) identifies the current location of the forklift 5 by detecting a change in the direction of travel of the forklift 5 based on the sensing results of the gyro sensor 70.

[0230] In this modified example, an offset value is pre-prepared for converting position information representing the location where the gyro sensor 70 is installed into position information representing the tip of the fork 71. The second acquisition unit 516 uses this offset value to convert the position information representing the current location of the forklift 5, identified by the gyro sensor 70, into position information representing the position of the tip of the fork 71. The second acquisition unit 516 (586) then acquires the converted position information as the position information of the target item.

[0231] In this modified example, the second acquisition unit 516(586) acquires position information representing the current location of the forklift 5 at the time the second recognition unit 514(584) recognizes that the item has been unloaded. The second acquisition unit 516(586) also converts this position information into position information of the tip of the fork 71 using a pre-prepared offset value. The second acquisition unit 516(586) then acquires this converted position information as position information representing the location of the item at the time the second recognition unit 514(584) recognizes that the item has been unloaded.

[0232] The communication control unit 511(581) transmits the location information, along with the item name acquired by the first recognition unit 513(583), to the server device 1(11).

[0233] Furthermore, the second acquisition unit 516(586) may continuously acquire location information representing the current location of the forklift 5, regardless of whether the second recognition unit 514(584) has recognized that the goods have been unloaded. This allows the administrator of the server device 1(11) or the warehouse manager to always know the current location of the forklift 5.

[0234] According to this modified version, location information indicating the position where an item was taken off can be obtained without installing barcodes or QR codes for location recognition on the shelf.

[0235] In this modified example, a gyro sensor is used to obtain location information representing the current location of the forklift 5. However, location information representing the current location of the forklift 5 may also be obtained using sensors other than a gyro sensor, such as an accelerometer or GPS sensor.

[0236] (Modification 2) In the first embodiment, the second embodiment, and Modification 1 described above, the mobile body for transporting the goods is a forklift, but the mobile body for transporting the goods is not limited to a forklift. For example, the mobile body for transporting the goods may be a crane. As an example, if the mobile body for transporting the goods is a crane, at least one camera is provided near the crane's hook. A sensor for position measurement may also be provided along with the camera.

[0237] According to this modified example, even when transporting goods using conveying equipment other than forklifts, the convenience of goods management in warehouses where goods are placed in a free location can be improved.

[0238] Although embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. The novel embodiments described above can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. Embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0239] 1.11 Server equipment 2.21 Camera Devices 3. Worker terminals 4. Drive control device 5 Forklift 41, 47, 51, 58, 61, 81 CPU 411, 471, 511, 581, 611, 811 Communication Control Unit 412, 472 Memory Control Unit 413, 473 Instruction section 414 Screen generation section 512, 582 1st acquisition part 513, 583 1st recognition part 514, 584 2nd recognition part 515, 585 3rd recognition part 516, 586 2nd acquisition part 587 Search Department 588, 613 Verification section 589 Judgment section 612 Display Control Unit 812 Drive Control Unit [Prior art documents] [Patent Documents]

[0240] [Patent Document 1] Japanese Patent Publication No. 2020-19647

Claims

1. A first acquisition unit that acquires images from a camera mounted on a mobile vehicle that transports goods, A first recognition unit recognizes a symbol attached to the article from the image and obtains identification information that allows the article to be identified from the symbol. A second recognition unit recognizes the action performed by the moving object on the item based on the size change of the symbol recognized by the first recognition unit from within the image, A second acquisition unit that acquires positional information of the article when it recognizes the movement of the moving body, An output unit outputs correspondence information that associates the identification information of the item indicated by the symbol with the position information of the item, in accordance with the operation content recognized by the second recognition unit. An information processing device equipped with the following features.

2. The second recognition unit recognizes that a first action of picking up the item has been performed when the size of the symbol is continuously enlarged for a predetermined time, and recognizes that a second action of unloading the item has been performed when the size of the symbol is continuously reduced from the enlarged state. The output unit outputs the corresponding information when at least one of the first operation and the second operation is recognized. The information processing apparatus according to claim 1.

3. The second recognition unit determined that the enlarged state had continued for a predetermined time, (a) The size of the symbol on the image is a predetermined size. (b) The paper could be recognized as an object when the symbol was printed on paper and attached to the article. (c) The symbol is located in a predetermined position on the image. (d) The first operation is recognized as having been performed when at least one of the following conditions is met: the distance to the item measured by the distance measuring sensor is within a predetermined range. The information processing apparatus according to claim 2.

4. The aforementioned articles are received into the warehouse where they are stored, or shipped out from the warehouse. The warehouse is equipped with a code symbol that encodes information representing its location within the warehouse. When the second acquisition unit recognizes that the moving body has performed an action on the item, it acquires the location within the warehouse, which is identified based on the reading result of the code symbol included in the image, as the location information of the item. The information processing apparatus according to claim 1.

5. The second acquisition unit acquires the position information of the article from the position of the moving body, which is determined based on the measurement results of a position measurement sensor mounted on the moving body, when the second recognition unit recognizes that the moving body has performed an action on the article. The information processing apparatus according to claim 1.

6. It comprises an information processing device and a server device, The aforementioned information processing device is A first acquisition unit that acquires images from a camera mounted on a mobile vehicle that transports goods, A first recognition unit recognizes a symbol attached to an article within the aforementioned image and obtains identification information that allows the article to be identified from the symbol. A second recognition unit recognizes the action performed by the moving object on the item based on the size change of the symbol recognized by the first recognition unit from within the image, A second acquisition unit that acquires positional information of the article when it recognizes the movement of the moving body, An output unit outputs correspondence information that associates the identification information of the item indicated by the symbol with the position information of the item, in accordance with the operation content recognized by the second recognition unit. Equipped with, The server device is A receiving unit that receives the aforementioned correspondence information, A storage control unit that stores the aforementioned correspondence information in a storage device, An information processing system equipped with the following features.

7. Computers, A first acquisition unit that acquires images from a camera mounted on a mobile vehicle that transports goods, A first recognition unit recognizes a symbol attached to the article from the image and obtains identification information that allows the article to be identified from the symbol. A second recognition unit recognizes the action performed by the moving object on the item based on the size change of the symbol recognized by the first recognition unit from within the image, A second acquisition unit that acquires positional information of the article when it recognizes the movement of the moving body, An output unit outputs correspondence information that associates the identification information of the item indicated by the symbol with the position information of the item, in accordance with the operation content recognized by the second recognition unit. A program that makes it function as such.

Images