Information processing device, information processing system, information processing method and program

Abstract

To provide an information processing device, an information processing system, an information processing method, and a program for notifying a user of an abnormality occurring in a moving object when the abnormality affects the management of the object's location information.SOLUTION: An information processing device that manages the location information of objects moved by a moving body based on images acquired via a network includes an abnormality determination unit that determines, based on the images, whether an abnormality has occurred in the moving body that will affect the management of the object's location information, a notification generation unit that generates a display indicating that an abnormality has occurred when the abnormality determination unit determines that an abnormality has occurred, and an output unit that outputs the display generated by the notification generation unit to an external terminal.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there has been known an information processing device that manages position information of objects such as cargo or pallets that are moved by a moving object such as a forklift.

[0003] Patent Document 1 discloses a technology for classifying the operating status of a moving body that moves an object in more detail. Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, if an abnormality occurs in a mobile object that affects the management of object location information without the user's knowledge, the object's location information cannot be obtained correctly, and unrealistic object location information may be registered without the user's knowledge. This can have adverse effects, such as the object not being found at the location registered in the system, resulting in a wasteful search for the object.

[0005] The present invention has been made in consideration of the above, and aims to notify a user that an abnormality has occurred in a moving object when an abnormality occurs in the moving object that affects the management of the object's location information. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the present invention provides an information processing device that manages location information of objects moved by a moving body based on images acquired via a network, characterized in that it comprises an abnormality determination unit that determines, based on the images, whether an abnormality that affects the management of the object's location information has occurred in the moving body, a notification generation unit that generates a display indicating that an abnormality has occurred when the abnormality determination unit determines that an abnormality has occurred, and an output unit that outputs the display generated by the notification generation unit to an external terminal. [Effects of the Invention]

[0007] According to the present invention, when an abnormality occurs in a moving object that affects the management of the object's location information, it is possible to notify the user that an abnormality has occurred in the moving object. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of position information of an object in a warehouse. [Figure 2] FIG. 2 is a diagram illustrating an example of the overall configuration of an information processing system according to the embodiment. [Figure 3] FIG. 3 is a block diagram illustrating an example of a hardware configuration of an on-premise server. [Figure 4] FIG. 4 is a block diagram illustrating an example of a functional configuration of the information processing system. [Figure 5] FIG. 5 is a flowchart illustrating an example of processing by the on-premise server. [Figure 6] FIG. 6 is a diagram showing an example of a display screen for a worker (forklift operator). [Figure 7] FIG. 7 is a diagram showing an example of a display screen for an administrator. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of an information processing device, an information processing system, an information processing method, and a program will be described in detail with reference to the accompanying drawings.

[0010] An information processing device according to an embodiment processes position information of an object moved by a moving body. For example, the moving body is a forklift, and the object includes a pallet. The information processing device according to an embodiment processes position information of the object, such as the pallet, moved by the forklift, and recognizes and tracks the movement of the object.

[0011] Here, Fig. 1 is a diagram for explaining an example of position information of an object in a warehouse. Fig. 1 shows the interior of a warehouse 100 and the periphery of the warehouse 100 as viewed from above (the ceiling side).

[0012] Warehouse 100 is a terminal warehouse (a warehouse established at a transit point in transportation). This terminal warehouse is a type of warehouse known as a cross-docking type. In a cross-docking type warehouse, multiple pallets for each product are received from a factory or wholesaler, and temporarily stored in the warehouse. Then, at the time of shipment, multiple types of pallets are combined while still packed on the same pallet, and shipped to the respective retail stores.

[0013] In Fig. 1, a truck yard 200 is located around a warehouse 100. Fig. 1 shows that the truck yard 200 has detached containers 300 transported by trailers and truck beds connected to the warehouse. A forklift 10 removes a pallet 31 from at least one of the beds of trucks that have arrived at the truck yard 200 or the containers 300 transported by trailers.

[0014] Thereafter, the forklift 10 carries the pallet 31 to the temporary storage location 40 and stores it there temporarily. Thereafter, at the time of shipping, the forklift 10 carries the pallet 31 to a location close to the truck yard 200 in the warehouse 100, arranges the items, and then loads the pallet 31 onto the bed of a truck or into a container 300.

[0015] In order to ensure flexible space for the daily changes in the types and quantities of goods coming in and out, temporary storage locations 40 often do not have designated sections for each product. However, because multiple workers temporarily store pallets 31 in arbitrary locations, when shipping, it is necessary to search for the desired pallet from among the multiple temporarily stored pallets.

[0016] To efficiently perform this search work, there is a need for an information processing system that can effectively utilize space by not specifying temporary storage locations for the pallets 31, while recognizing and tracking the movement of the pallets 31 within the warehouse 100 to visualize it. As an example, the information processing device according to the embodiment is used in such an information processing system.

[0017] An information processing system including an information processing device according to an embodiment will be described below.

[0018] (Example of overall configuration of information processing system 1) Fig. 2 is a diagram showing an example of the overall configuration of an information processing system 1 according to an embodiment. As shown in Fig. 2, the information processing system 1 includes a forklift 10, an omnidirectional camera 20, and an on-premise server 50. These are communicably connected via a network 400 such as a LAN (Local Area Network). Note that devices other than those described above, such as an external server or an image forming device, may also be communicably connected to the network 400.

[0019] The forklift 10 is an example of a mobile body that transports the pallet 31 and the cargo 32 by holding the cargo 32 placed on the pallet 31 and transporting it while holding it. Transport by a mobile body is an example of movement by a mobile body. The pallet 31 and the cargo 32 are each an example of an object. In the following, the pallets 31 and the cargo 32 will be collectively referred to as the object 30 unless they are particularly distinguished from one another. The forklift 10 is a generic term for multiple forklifts, the pallet 31 is a generic term for multiple pallets, and the cargo 32 is a generic term for multiple cargoes.

[0020] The forklift 10 may transport the object 30 in response to the driving operation of an operator, or may transport the object 30 by automatic driving without the intervention of an operator.

[0021] The omnidirectional camera 20 is an example of an imaging unit provided in the forklift 10. The omnidirectional camera 20 is a camera that can capture images in all directions of 360 degrees around the omnidirectional camera 20. The direction 20a indicates the direction in which the omnidirectional camera 20 can capture images.

[0022] The omnidirectional image (all-directional image) captured by the omnidirectional camera 20 is one example of an image. However, the imaging unit is not limited to the omnidirectional camera 20, and may be any device that can capture an image of the area around the forklift 10. Furthermore, the captured image does not necessarily have to be a omnidirectional image.

[0023] The spherical image includes an image capturing a scene in the conveying direction 11 of the object 30 as seen from the forklift 10, and a scene in the vertically upward direction 12 as seen from the forklift 10. In other words, the conveying direction 11 is in front of the forklift 10, and the vertically upward direction 12 is above the forklift 10. Since the spherical camera 20 can capture images in all directions, it can capture an image including both the front and the above of the forklift 10 in a single image. The conveying direction 11 is an example of a moving direction.

[0024] The spherical camera 20 is preferably attached to the roof of the forklift 10 or to a support member 22 that supports the forks 21. This ensures a good field of view for capturing images in front of and above the forklift 10. Here, the forks 21 are an example of a holding portion provided on the forklift 10.

[0025] The omnidirectional camera 20 has a wireless communication function and transmits the captured omnidirectional image to the on-premise server 50 via the network 400.

[0026] The cargo 32 is provided with a barcode 33, which is an example of identification information that identifies the cargo 32. Such a barcode may be provided on the pallet 31 and used as identification information that identifies the pallet 31. The barcode 33 is read by a reader such as a barcode reader, and the identification information resulting from the reading is transmitted to the on-premise server 50 via the network 400. Note that the identification information is not limited to a barcode, and may be a QR code (registered trademark), an ID (identifier) ​​number, or the like.

[0027] The on-premise server 50 is an example of an information processing device that is installed in the warehouse 100 and processes the position information of the object 30 transported by the forklift 10. The on-premise server 50 can be replaced with a cloud server (a computing device installed outside the warehouse environment). This can reduce the introduction cost and running cost.

[0028] The on-premise server 50 processes the position information of the object 30 based on the omnidirectional image received via the network 400 and the identification information indicating the object 30. In addition, by using the omnidirectional images captured by the multiple omnidirectional cameras 20, the position information of the object 30 held by the multiple forklifts 10 can be acquired.

[0029] Furthermore, the on-premise server 50 executes a self-position recognition calculation process for the forklift 10 based on the omnidirectional image received via the network 400. The on-premise server 50 performs the self-position recognition calculation process for the multiple omnidirectional cameras 20, thereby grasping the current positions of the multiple forklifts 10.

[0030] The forklift 10 includes, for example, a single-board computer. The forklift 10 has the omnidirectional camera 20 connected to the single-board computer via a wired connection. The self-position recognition calculation process is performed on the single-board computer, and the results are transmitted and aggregated by wireless communication to the on-premise server 50 via the network 400. This allows the on-premise server 50 to grasp the current position of each forklift 10.

[0031] The on-premise server 50 reads the ID, location information, video image information, etc. from the HD 504 (see Figure 3) within the on-premise server 50, identifies the final location information of the forklift 10 and pallet 31, and visualizes it by outputting it to a terminal (smartphone, etc.).

[0032] As described above, the information processing system 1 includes the on-premise server 50 outside the forklift 10, and the processing unit is provided in the on-premise server 50. This makes it possible to reduce the power consumption of the single-board computer (vehicle-mounted edge) included in the forklift 10. Specifically, what previously required approximately 40 W can now be reduced to approximately 10 W.

[0033] However, since the forklift 10 moves around, there is a possibility that communication between the forklift 10 (vehicle-mounted edge) and the on-premise server 50 may be interrupted. Therefore, in this embodiment, the forklift 10 (vehicle-mounted edge) and the on-premise server 50 each have a communication status confirmation unit, which will be described in detail later. This allows the connection status on the network 400 to be monitored. When communication is interrupted, the data that was not transmitted at the time of the interruption will be sent all at once after the connection is re-established.

[0034] (Example of hardware configuration of on-premise server 50) 3 is a block diagram showing an example of a hardware configuration of the on-premise server 50. The on-premise server 50 is constructed by a computer.

[0035] 3 , the on-premise server 50 includes a central processing unit (CPU) 501, a read-only memory (ROM) 502, a random access memory (RAM) 503, a hard disk (HD) 504, a hard disk drive (HDD) controller 505, and a display 506. The on-premise server 50 also includes an external device connection interface (I / F) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a digital versatile disk rewritable (DVD-RW) drive 514, and a media I / F 516.

[0036] Of these, the CPU 501 controls the overall operation of the on-premise server 50. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501.

[0037] The HD 504 stores various data such as programs, etc. The HDD controller 505 controls reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, menus, windows, characters, or images.

[0038] The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using the network 400. The bus line 510 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 501 shown in FIG. 3.

[0039] The keyboard 511 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515, such as a flash memory.

[0040] (Example of functional configuration of information processing system 1) Fig. 4 is a block diagram showing an example of the functional configuration of the information processing system 1. As shown in Fig. 4, the on-premise server 50 includes a receiving unit 51, a mobile object position acquiring unit 52, a retained information acquiring unit 53, an identification information acquiring unit 54, a time acquiring unit 55, an object position acquiring unit 56, an output unit 57, a storage unit 58, a cargo attachment / detachment position / height estimating unit 59, an abnormality determining unit 60, and a notification generating unit 61.

[0041] Each of these units is a function or means for performing a function that is realized when any of the components shown in FIG. 3 operates in response to an instruction from CPU 501 in accordance with a program loaded from HD 504 onto RAM 503.

[0042] The forklift 10 also includes an omnidirectional camera 20, a transmitter 101, and a recording unit 102. The omnidirectional camera 20 acquires a video of the surroundings (a omnidirectional image). The recording unit 102 records the omnidirectional image acquired by the omnidirectional camera 20 on a recording medium. The transmitter 101 transmits the omnidirectional image acquired by the omnidirectional camera 20 to the on-premise server 50 via the network 400.

[0043] The functions of the transmitting unit 101 and the recording unit 102 can be realized by an electric circuit provided in either the forklift 10 or the omnidirectional camera 20, or by software (CPU). Alternatively, they may be realized by a plurality of circuits or a plurality of pieces of software.

[0044] The on-premise server 50 acquires the position information of the object 30 based on the position information of the forklift 10 acquired based on the omnidirectional image captured by the omnidirectional camera 20 and the holding information indicating whether the object 30 is being held or not being held by the forklift 10. The acquired position information of the object 30 can then be output to the outside via the output unit 57.

[0045] Receiving unit 51 receives, via network 400, the omnidirectional image captured by omnidirectional camera 20 and transmitted via transmitting unit 101, and outputs the omnidirectional image to moving object position acquiring unit 52 and retained information acquiring unit 53. Receiving unit 51 also receives, via network 400, identification information read by a reader such as a barcode reader, and outputs the identification information to identification information acquiring unit 54.

[0046] The mobile object position acquisition unit 52 acquires the position information of the forklift 10 by calculation based on the input spherical image, and outputs the acquired information to the object position acquisition unit 56. The process of acquiring the position information of the forklift 10 (self-position estimation process) can apply a technology for simultaneously creating a map and recognizing its own position (SLAM: Simultaneous Localization and Mapping) (see, for example, "Commentary: The Current State and Future Prospects of SLAM," by Tomono Masahiro and Hara Yoshitaka, Systems / Control / Information, Vol. 64, No. 2, 2020, pp. 45-50, https: / / www.jstage.jst.go.jp / article / isciesci / 64 / 2 / 64_45 / _article / -char / ja / ).

[0047] The holding information acquisition unit 53 acquires, by calculation, holding information indicating whether the object 30 is being held or not held by the forklift 10 based on the input spherical image, and outputs the information to the object position acquisition unit 56.

[0048] The identification information acquisition unit 54 acquires the identification information by inputting the identification information from the receiving unit 51, and outputs the identification information to the object position acquisition unit 56. However, acquisition of the identification information by the identification information acquisition unit 54 is not limited to via the network 400. For example, the identification information acquisition unit 54 may acquire identification information input by a user such as an administrator using the keyboard 511 or the pointing device 512 in FIG. 3, or may acquire identification information stored in advance in the storage unit 58, or may acquire identification information via the external device connection I / F 508. The administrator is the administrator of the information processing system 1 or the warehouse 100.

[0049] The time acquisition unit 55 acquires information indicating the time when the receiving unit 51 received the omnidirectional image and the identification information, and outputs the information to the object position acquisition unit 56 .

[0050] The object position acquisition unit 56 acquires the position information of the object 30 based on the position information of the forklift 10 and the holding information. The object position acquisition unit 56 also associates the position information of the object 30 with the identification information indicating the object 30 and time information, and outputs the information via the output unit 57. The output destination of the output unit 57 is an external device such as a PC (Personal Computer), a display device such as the display 506, a storage device such as the HD 504, or the like.

[0051] The cargo attachment / detachment position / height estimation unit 59 estimates the attachment / detachment position and height of the object 30 based on the input spherical image, and acquires the attachment / detachment position and height of the object 30.

[0052] The storage section 58 can store identifying information indicative of an object 30 such as a pallet 31 or cargo 32 .

[0053] The abnormality determination unit 60 determines that an abnormality has occurred based on the output of the mobile object position acquisition unit 52, the output of the cargo attachment / detachment position / height estimation unit 59, or the state of the network 400.

[0054] If a problem occurs in the network 400 or the forklift 10 moves to an area outside the communication range and video transmission to the on-premise server 50 is not possible, the spherical image and the like will be resent after reconnecting to the network 400. However, if the capacity of the storage unit of the forklift 10 (vehicle-mounted edge) is exceeded, data may be overwritten before the spherical image is processed, or a delay in video transmission may cause the loss of position information of the object 30 for a certain period of time, which may result in delays in the positioning of the forklift, detection of the gripping position, and detection of the height. A network problem is an example of an abnormality that affects the management of object position information.

[0055] Furthermore, if the available network bandwidth of network 400 is narrow, the quality of the spherical image may be reduced (the amount of data may be reduced) and video transmission may be performed so that the spherical image is not interrupted. In this case, the quality of the video reaching on-premise server 50 may be poor, which may cause image-based positioning, attachment / detachment detection, and height detection to fail, making it impossible to record the position information of object 30. Failure of image-based positioning, attachment / detachment detection, and height detection is an example of an abnormality that affects the management of object position information.

[0056] The notification generation unit 61 generates an appropriate notification based on the abnormality determination by the abnormality determination unit 60. When an abnormality in work is detected, the notification generated by the notification generation unit 61 is used to notify appropriate relevant parties via the output unit 57. The notification by the notification generation unit 61 is provided in an appropriate form, such as a message, an alert, or notification to a remote device. This allows the work environment to be monitored efficiently, and when an abnormality is detected, a prompt and appropriate response can be taken.

[0057] (Example of processing by on-premise server 50) 5 is a flowchart showing an example of processing by the on-premise server 50. Fig. 5 shows processing triggered by the timing at which the on-premise server 50 receives an operation to start acquiring location information of the object 30. The operation to start acquiring location information of the object 30 is performed by a user such as an administrator using the pointing device 512 in Fig. 3 or the like.

[0058] First, in step S1, the receiving unit 51 receives a spherical image and identification information via the network 400. The spherical image received via the network 400 includes, in addition to the spherical image, a timestamp when the image was acquired by the forklift 10, a radio wave intensity value at the time of transmission, and other metadata (acceleration sensor, moving object ID, etc.).

[0059] Next, in step S2, the moving object position acquisition unit 52 acquires the position information of the forklift 10 by calculation based on the input spherical image. Furthermore, the cargo attachment / detachment position / height estimation unit 59 estimates the attachment / detachment position and height of the object 30 based on the input spherical image, and acquires the attachment / detachment position and height of the object 30. This provides data for tracking the position of the object 30 moved by the forklift 10.

[0060] Note that the detection of the attachment / detachment position of the object 30 and the detection of the height of the object 30 are output when the forklift 10 is holding the object 30, and therefore this output is not included in all frames of the omnidirectional image.

[0061] Next, in step S3, the abnormality determination unit 60 determines whether an abnormality has occurred in each module (mobile body position acquisition unit 52, cargo attachment / detachment position / height estimation unit 59) based on the latest results of each module (mobile body position acquisition unit 52, cargo attachment / detachment position / height estimation unit 59) and the result history.

[0062] Here, several methods for detecting abnormalities in positioning by the mobile object position acquisition unit 52 will be described.

[0063] (Feature consistency verification) First, we will explain how to detect anomalies in positioning by verifying feature consistency. Visual SLAM generally detects and tracks feature points and feature quantities. If positioning is accurate, the same object or feature points should be detected consistently across different images. Therefore, the anomaly determination unit 60 verifies the consistency of feature points and determines that an anomaly exists if one is found.

[0064] (Integration of motion model and observation model) Next, we will explain how to detect positioning anomalies by integrating the motion model and observation model. Using the robot's motion model and the observation model of the environment, we can verify the consistency between the estimated position and the map. The anomaly determination unit 60 determines that the positioning is abnormal if the motion model is inaccurate or the observation model is inconsistent.

[0065] (Loop Closing) Next, detection of an abnormality in positioning by loop closing will be described. Loop closing is a process of verifying whether the forklift 10 can return to the same location. If the forklift 10 cannot reach the same location again, the abnormality determination unit 60 determines that an abnormality in positioning has occurred.

[0066] (Using outlier detection methods) Next, we will explain how to detect anomalies in positioning using an outlier detection method. By using an anomaly detection method or an outlier detection method, the anomaly determination unit 60 detects unexpected anomalies in positioning. In addition, statistical methods and machine learning methods may also be combined.

[0067] (Error Modeling) Next, we will explain how to detect abnormalities in positioning using error modeling. Errors and noise from cameras and sensors are modeled, and the reliability of the positioning results is evaluated based on the model. If the positioning results do not fall within the modeled error range, the abnormality determination unit 60 considers them to be abnormal.

[0068] The abnormality determination unit 60 can determine an abnormality in the positioning results of Visual SLAM by using these methods or a combination of these methods.

[0069] Here, several methods for detecting abnormalities in position detection in the cargo attachment / detachment position / height estimation unit 59 will be described.

[0070] In specific implementations, an appropriate method will be selected depending on the system requirements and environment. Here, an abnormality detection method will be described in which a marker is attached to a moving part of the forklift 10 and the height of the claws of the forklift 10 gripping the object 30 is estimated by detecting the displacement of the marker from a spherical image.

[0071] (Monitoring marker detection accuracy) First, we will explain how to detect abnormalities in position detection by monitoring the marker detection accuracy. It monitors whether the AR marker is correctly detected in the image. The abnormality determination unit 60 checks whether the position and direction of the detected marker are within the expected range, and if they are outside the expected range, it determines that an abnormality has occurred.

[0072] (Verification of tracking stability) Next, we will explain how to detect abnormalities in position detection by verifying tracking stability. Verification of tracking stability involves checking whether marker tracking is stable. The abnormality determination unit 60 monitors the tracking stability in consecutive frames and detects whether there is an abnormality. If tracking is unstable, the estimation results may also be inaccurate.

[0073] (Evaluation of projective transformation accuracy) Next, we will explain how to detect abnormalities in position detection by evaluating the accuracy of projective transformation. We evaluate whether the transformation (projective transformation) from the camera image to the real-world coordinate system is accurate. The abnormality determination unit 60 uses the characteristic pattern of the AR marker to check the relationship between the coordinates on the camera image and the coordinates in the real-world coordinate system and detects abnormalities.

[0074] (Checking physical constraints on distance and direction) Next, we will explain how to detect abnormalities in position detection by checking physical constraints on distance and orientation. If there are physical constraints on the distance or orientation to the target, the abnormality determination unit 60 will verify the estimation results taking those constraints into account and determine whether there is an abnormality. For example, there are cases where the AR marker cannot be moved more than a certain distance away from the target, or the angle is limited.

[0075] (Using anomaly detection techniques) Next, we will explain how to detect anomalies in position detection using anomaly detection methods. In image processing, anomaly detection methods and outlier detection methods are used. The anomaly determination unit 60 uses the anomaly detection method and outlier detection method to detect whether the estimated distance or direction is outside the predicted range and determine whether an anomaly has occurred.

[0076] By using these methods or a combination of these methods, the abnormality determination unit 60 can detect abnormalities in the estimation results of the distance to the target and the direction using the AR marker and take appropriate action.

[0077] Subsequently, in step S4, the abnormality determination unit 60 determines whether an abnormality has occurred in the network 400.

[0078] The abnormality determination unit 60, for example, compares the timestamp assigned to each frame of the spherical image with the current time, and determines that an abnormality has occurred if a delay of a certain level or more is observed. The abnormality determination unit 60 also obtains, for example, a history of the timestamps assigned to each frame of the spherical image, and determines that an abnormality has occurred. Furthermore, the abnormality determination unit 60 calculates, for example, a packet loss ratio, and determines that an abnormality has occurred if a packet loss of a certain level or more has occurred.

[0079] Subsequently, if the abnormality determination unit 60 determines that at least one or more abnormalities have occurred in step S3 or step S4 (Yes in step S5), the notification generation unit 61 generates a display for the worker (forklift operator) indicating that an abnormality has occurred (step S6), and also generates a display for the manager indicating that an abnormality has occurred (step S7). Thereafter, the output unit 57 transmits the display generated by the notification generation unit 61 to each external terminal used by each user (worker (forklift operator), manager) (step S8).

[0080] The output unit 57 may send the display generated by the notification generation unit 61 via a cloud service or email via an internet line. Even when the output unit 57 sends the display generated by the notification generation unit 61 via the network 400 where a network problem is occurring, the amount of data is much smaller than that of a video image, which has a large data size, so there is a possibility that the display can be sent (notified) to the worker.

[0081] FIG. 6 is a diagram showing an example of a display screen for a worker (forklift operator). FIG. 6 is a screen D1 showing an example of an abnormality notification (fault notification) to be displayed on the in-vehicle terminal for the worker (forklift operator). As shown in FIG. 6, the screen D1 includes a display X showing the locations of surrounding access points, where the locations of access points are registered in advance. Furthermore, the screen D1 does not display abnormality notifications (fault notifications) for forklifts 10 other than the forklift 10 occupied by the forklift operator who is the target of the notification.

[0082] For example, screen D1 displays "type of fault and severity." When an abnormality (failure) occurs, the "type of fault and severity" is displayed, showing the specific type of abnormality (failure) (e.g., communication failure, sensor failure, low battery, etc.) and its severity (urgency and impact). Screen D1 shown in FIG. 6 displays "Positioning results are unstable." and "Communication has been unstable for a long time."

[0083] For example, the screen D1 displays the "location of the fault." The display of the "location of the fault" indicates the specific location or area where the abnormality (fault) occurred, allowing the operator of the forklift 10 to quickly take action. In the screen D1 shown in Fig. 6, the vehicle (forklift) in operation is highlighted.

[0084] For example, screen D1 displays "Recovery Procedures." The "Recovery Procedures" display includes abnormality (failure) recovery procedures and basic troubleshooting guides, supporting forklift operators in quickly and accurately resolving abnormalities (failures). Screen D1 shown in Figure 6 displays "Please click the Cancel button." By notifying users of the means to recover from abnormalities in this way, it is possible to minimize productivity losses due to equipment downtime.

[0085] For example, screen D1 displays "contact information." The "contact information" display includes information about contact points for abnormalities (failures) and support personnel, so that support can be received as needed.

[0086] Fig. 7 is a diagram showing an example of a display screen for the administrator, which is a screen D2 showing an example of an abnormality notification (fault notification) to be displayed on the terminal for the administrator.

[0087] For example, screen D2 displays a "Comprehensive Fault Summary." The "Comprehensive Fault Summary" display shows a comprehensive overview of all abnormalities (faults) that have occurred in the warehouse, allowing the manager to grasp the overall situation. Screen D2 shown in Figure 7 displays "Cargo was placed at a time when the positioning results were unstable." and "Forklift No. 2 has been out of range for an extended period of time."

[0088] For example, screen D2 displays "Failure frequency and trends." When a similar abnormality (failure) occurs repeatedly, the display of "Failure frequency and trends" indicates the frequency and trends, and provides information for taking measures.

[0089] For example, screen D2 displays "Recovery status and progress information." The "Recovery status and progress information" display displays the recovery status and progress information for each abnormality (failure), allowing you to understand which abnormalities (failures) have been resolved and which are still being addressed.

[0090] For example, screen D2 displays "Corrections and Improvement Suggestions." The "Corrections and Improvement Suggestions" display presents corrections for abnormalities (failures) and future improvement suggestions, helping to improve the efficiency of the entire warehouse and reduce risks.

[0091] For example, screen D2 displays "Statistical Information." The "Statistical Information" display provides statistical information such as the frequency of occurrence of abnormalities (failures) and their causes, which is useful for future predictions and planning.

[0092] As described above, according to this embodiment, an abnormality is detected in the location management device for objects such as pallets and cargo, and a user (the operator and manager of the forklift 10) is notified by a display indicating that an abnormality has occurred. In this way, when an abnormality that affects the management of object location information occurs in a moving object, the user can be notified that an abnormality has occurred in the moving object.

[0093] In the present embodiment, a forklift is used as an example of a moving object, but the moving object is not limited to this. For example, the moving object may be an automated guided vehicle, a drone, or the like.

[0094] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0095] Note that the information processing device is not limited to the on-premise server 50, as long as it is a device with a communication function. The information processing device may be, for example, an image forming device, a PJ (Projector), an IWB (Interactive White Board: a white board with an electronic blackboard function that allows mutual communication), an output device such as digital signage, a HUD (Head Up Display) device, industrial machinery, an imaging device, a sound collection device, medical equipment, a network home appliance, an automobile (Connected Car), a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, or the like.

[0096] For example, aspects of the present invention are as follows. <1> An information processing device that manages position information of an object moved by a moving body based on an image acquired via a network, an abnormality determination unit that determines, based on the image, whether an abnormality that affects management of position information of the object has occurred in the moving object; a notification generating unit that generates a display indicating that an abnormality has occurred when the abnormality determining unit determines that an abnormality has occurred; an output unit that outputs the display generated by the notification generation unit to an external terminal; An information processing device comprising: <2> a moving object position acquisition unit that acquires position information of the moving object by calculation based on the image; a cargo attachment / detachment position / height estimation unit that estimates the attachment / detachment position and height of the object based on the image; Equipped with The abnormality determination unit determines that an abnormality has occurred based on any one of the outputs of the moving body position acquisition unit, the cargo attachment / detachment position / height estimation unit 5, and the network status output. Characterized by <1> The information processing device described in <3> the notification generation unit generates a display for an operator who drives the mobile object and a display for a manager other than the operator, based on the abnormality found by the abnormality determination unit. Characterized by <1> or <2> The information processing device described in <4> The display generated by the notification generation unit includes a method for recovering from the abnormality. Characterized by <3> The information processing device described in <5> A moving object and an imaging unit that captures an image; <1> Or <4> and an information processing device according to any one of the above. An information processing system comprising: <6> 1. An information processing method in an information processing device that manages position information of an object moved by a moving body based on an image acquired via a network, comprising: an abnormality determination step of determining, based on the image, whether an abnormality that affects management of position information of the object has occurred in the moving object; a notification generating step of generating a display indicating that an abnormality has occurred when it is determined that an abnormality has occurred by the abnormality determining step; an output step of outputting the display generated by the notification generating step to an external terminal; An information processing method comprising: <7> a computer that manages position information of an object moved by a moving body based on images acquired via a network; an abnormality determination unit that determines, based on the image, whether an abnormality that affects management of position information of the object has occurred in the moving object; a notification generating unit that generates a display indicating that an abnormality has occurred when the abnormality determining unit determines that an abnormality has occurred; an output unit that outputs the display generated by the notification generation unit to an external terminal; A program that functions as a [Explanation of symbols]

[0097] 1. Information Processing Systems 10 Mobile 20 Imaging unit 50 Information processing equipment 52 Mobile object position acquisition unit 57 Output section 59 Cargo detachment position and height estimation unit 60 Abnormality determination section 61 Notification generator 400 Network [Prior art documents] [Patent documents]

[0098] [Patent Document 1] Patent No. 6588123

Claims

1. An information processing device that manages position information of an object moved by a moving body based on an image acquired via a network, an abnormality determination unit that determines, based on the image, whether an abnormality that affects management of position information of the object has occurred in the moving object; a notification generating unit that generates a display indicating that an abnormality has occurred when the abnormality determining unit determines that an abnormality has occurred; an output unit that outputs the display generated by the notification generation unit to an external terminal; An information processing device comprising:

2. a moving object position acquisition unit that acquires position information of the moving object by calculation based on the image; a cargo attachment / detachment position / height estimation unit that estimates the attachment / detachment position and height of the object based on the image; Equipped with the abnormality determination unit determines that an abnormality has occurred based on any one of the outputs of the moving body position acquisition unit, the cargo attachment / detachment position / height estimation unit, and the network status output.

2. The information processing apparatus according to claim 1, wherein:

3. the notification generation unit generates a display for an operator who drives the mobile object and a display for a manager other than the operator, based on the abnormality found by the abnormality determination unit.

2. The information processing apparatus according to claim 1, wherein:

4. The display generated by the notification generation unit includes a method for recovering from the abnormality.

4. The information processing apparatus according to claim 3,

5. A moving object and an imaging unit that captures an image; An information processing device according to any one of claims 1 to 4; An information processing system comprising:

6. 1. An information processing method in an information processing device that manages position information of an object moved by a moving body based on an image acquired via a network, comprising: an abnormality determination step of determining, based on the image, whether an abnormality that affects management of position information of the object has occurred in the moving object; a notification generating step of generating a display indicating that an abnormality has occurred when it is determined that an abnormality has occurred by the abnormality determining step; an output step of outputting the display generated by the notification generating step to an external terminal; An information processing method comprising:

7. a computer that manages position information of an object moved by a moving body based on images acquired via a network; an abnormality determination unit that determines, based on the image, whether an abnormality that affects management of position information of the object has occurred in the moving object; a notification generating unit that generates a display indicating that an abnormality has occurred when the abnormality determining unit determines that an abnormality has occurred; an output unit that outputs the display generated by the notification generation unit to an external terminal; A program that functions as a

Images