Information processing method, information processing system, and trained model

Abstract

To provide an information processing method, an information processing system, and a trained model for determining whether a forklift is stacking cargo highly. [Solution] A computer determines the height of the load being carried by the forklift TG1 shown in the video and the height of the forklift TG1 based on a time-series video of images captured by an imaging unit 21 installed at a predetermined position capable of capturing the path of travel of the forklift TG1, and the height of the forklift TG1, and calculates the ratio between the height of the load and the height of the forklift TG1 using the determined height of the load and the height of the forklift TG1.If the calculated ratio is equal to or greater than a predetermined threshold, it determines that a violation event that violates the high stacking condition has occurred.

Description

[Technical Field]

[0001] The present invention relates to an information processing method, an information processing system, and a trained model. [Background technology]

[0002] Conventionally, in forklifts, technologies for preventing high stacking of loads on the forks have been known, as this can cause the vehicle body to tilt forward, etc. Patent Document 1 discloses a technology that detects the hydraulic pressure of the lift cylinder to detect the load of the load on the forks, and then displays the allowable load percentage calculated by back-calculating the rotation moment based on the detected value on a display panel (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 6-247698 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, when the load is light but large, even if the display panel indicates that the load is within the allowable load, the load may actually be piled high. Therefore, with conventional technology, it has been difficult to determine whether the forklift is piled high. [Means for solving the problem]

[0005] An information processing method for achieving the above-mentioned object is characterized in that a computer identifies the height of the load being carried by the forklift and the height of the forklift shown in a video of consecutive images captured by an imaging unit installed at a predetermined position capable of capturing the path of the forklift, based on the video; calculates the ratio between the height of the load and the height of the forklift using the identified height of the load and the height of the forklift; and if the calculated ratio is equal to or greater than a predetermined threshold, determines that a violation event that violates high stacking conditions has occurred.

[0006] With this configuration, it is possible to determine whether the forklift is stacked high. In an information processing method for achieving the above-mentioned object, when the images constituting the video are input, a trained model outputs the position of the load and the position of the forklift, respectively, and the height of the load and the height of the forklift are identified based on the information output by inputting each of the images constituting the video.

[0007] With this configuration, the ratio can be calculated while identifying the height of the load and the height of the forklift using a simple method that uses a trained model. In an information processing method that achieves the above-mentioned objective, the trained model, when input with the images that make up the video, outputs bounding boxes indicating the position of the load and the position of the forklift, respectively, and by inputting the images that make up the video into the trained model, it is possible to cause the trained model to output bounding boxes indicating the position of the load and the position of the forklift.

[0008] According to this configuration, the ratio can be calculated while specifying the height of the load and the height of the forklift using the bounding box. In an information processing method for achieving the above object, it is preferable to determine whether the forklift is moving based on the video, and only if it is determined that the forklift is moving, to perform processing to determine whether a violation event that violates the high stacking condition has occurred based on the video.

[0009] According to this configuration, the determination process can be omitted if the determination is not necessary. In the information processing method for achieving the above object, it may be determined that the forklift shown in the video is traveling when the forklift continues to move for a predetermined period of time or more.

[0010] In an information processing method for achieving the above-mentioned object, the forward direction and orientation of the forklift shown in the video are identified based on the video, and based on the identified forward direction and orientation of the forklift, it is identified whether the forklift is moving forward or backward, the specified threshold values ​​include a forward threshold value used when the forklift is moving forward and a reverse threshold value used when the forklift is moving backward, the forward threshold value is smaller than the reverse threshold value, and if it is identified that the forklift is moving forward, the forward threshold value is used to determine whether the high stacking condition is violated, and if it is identified that the forklift is moving backward, the reverse threshold value is used to determine whether the high stacking condition is violated.

[0011] With this configuration, it is possible to appropriately determine whether or not the high stacking condition is violated based on the behavior of the forklift. In an information processing method for achieving the above-mentioned object, the predetermined threshold value includes a turning threshold value used when the forklift is turning, the turning threshold value being smaller than the reverse threshold value, and when the forklift is turning, the turning threshold value may be used to perform processing to determine whether or not the high stacking condition is violated.

[0012] In an information processing method for achieving the above-mentioned object, for each of the images constituting the video, a first position, which is the position of the balance weight of the forklift on the image, and a second position, which is the position of the mast of the forklift on the image, may be identified, and the orientation of the forklift may be identified based on the positional relationship between the first position and the second position.

[0013] According to this configuration, the orientation of the forklift can be identified based on the characteristic points of the forklift. In an information processing method for achieving the above-mentioned object, the first position and the second position may be identified based on information output by inputting the images constituting the video into a trained model that outputs the first position and the second position when the images constituting the video are input.

[0014] With this configuration, the positions of the feature points of the forklift can be identified using a simple method that uses a trained model. In an information processing method that achieves the above-mentioned objective, when it is determined that the violation event has occurred, a violation video may be generated by extracting a portion of the video that includes the image of the scene where the violation event occurred.

[0015] With this configuration, a violation video can be generated that shows a scene in which the high stacking condition is violated. In the information processing method for achieving the above object, when it is determined that the violation event has occurred, determination result information indicating a determination result corresponding to the violation event may be generated.

[0016] According to this configuration, by referring to the determination result information, detailed information about the violation event can be grasped for each event. In an information processing method for achieving the above-mentioned object, the forklift involved in the violation event may be identified based on identification information corresponding to the forklift, and the judgment result information may be generated by further including moving object identification information that can identify the identified forklift.

[0017] According to this configuration, by referring to the determination result information, it is possible to refer to information about the moving body that has violated the law. In an information processing method that achieves the above-mentioned object, the judgment result information may include imaging unit identification information that can identify the imaging unit, and a confirmation screen may be generated that displays the occurrence status of the violation event for each imaging unit on the map image based on the judgment result information and a map image that shows the predetermined position where the imaging unit corresponding to the imaging unit identification information is installed and the movement route.

[0018] According to this configuration, the location where a violation of a regulation occurred on the travel route can be presented in an intuitive and easy-to-understand manner. In the information processing method for achieving the above object, when it is determined that the violation event has occurred, the forklift may be notified that the violation event has occurred.

[0019] According to this configuration, it is possible to warn a moving object that is present within or near the detection range by notifying the moving object that a violation event has occurred. In the information processing method for achieving the above object, it may be determined that a compliance event in which the high pile condition is complied with has occurred, based on the calculated ratio, if the ratio is less than a predetermined threshold value.

[0020] With this configuration, it is possible to determine whether the forklift is stacking items too high. In an information processing method that achieves the above-mentioned object, when it is determined that the compliance event has occurred, a compliance video may be generated by extracting a portion of the video that includes the image of the scene where the compliance event occurred.

[0021] With this configuration, it is possible to generate a compliance video showing a scene in which the high stacking conditions are observed. In the information processing method for achieving the above object, judgment result information indicating a judgment result corresponding to the compliance event may be generated.

[0022] According to this configuration, by referring to the determination result information, detailed information about the compliance event shown in the compliance video can be grasped for each event. An information processing system that achieves the above-mentioned object comprises an imaging unit provided at a predetermined position capable of imaging the path of a forklift, one or more management devices that store a video of a chronological sequence of images generated by the imaging unit imaging the path of the forklift in a server device on the cloud, and an information processing device that performs various processes based on the video stored in the server device, wherein the information processing device comprises: a position identification unit that identifies the height of the load carried by the forklift and the height of the forklift based on the input video; a ratio calculation unit that calculates the ratio between the height of the load and the height of the forklift using the height of the load and the height of the forklift; and a high-stacking determination unit that determines that a violation event that violates high-stacking conditions has occurred if the ratio calculated by the ratio calculation unit is equal to or greater than a predetermined threshold.

[0023] According to this configuration, the same effects as those of the above-described information processing method can be obtained. An information processing system that achieves the above-mentioned object comprises an imaging unit provided at a predetermined position capable of imaging the path of a forklift, one or more management devices that store in the information processing device a video of a chronological sequence of images generated by the imaging unit imaging the path of the forklift, and an information processing device that performs various processes based on the video, wherein the information processing device comprises: a position identification unit that identifies the height of the load carried by the forklift and the height of the forklift based on the input video; a ratio calculation unit that calculates the ratio between the height of the load and the height of the forklift using the height of the load and the height of the forklift; and a high-stacking determination unit that determines that a violation event that violates high-stacking conditions has occurred if the ratio calculated by the ratio calculation unit is equal to or greater than a predetermined threshold.

[0024] According to this configuration, the same effects as those of the above-described information processing method can be obtained. The trained model that achieves the above objective is characterized in that when an imaging unit capable of capturing an image of a forklift's travel path inputs an image generated by capturing the travel path, it outputs the position of the load being loaded on the forklift and the position of the forklift.

[0025] With this configuration, the position of the load and the position of the forklift can be identified based on the image. [Effects of the Invention]

[0026] According to the present invention, it is possible to determine whether a forklift is carrying a load too high. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of an information processing device. [Figure 3] FIG. 3 is a diagram showing an example of a forklift shown in an image. [Figure 4] FIG. 4 is a diagram used to explain the processing of the angle determination unit. [Figure 5] FIG. 5 is a diagram used to explain the processing of the angle calculation unit. [Figure 6] FIG. 6 is a diagram showing an example of the contents of the determination result information. [Figure 7] FIG. 7 is a diagram showing an example of the confirmation screen. [Figure 8] FIG. 8 is a flowchart illustrating an example of processing by the information processing device. [Figure 9] FIG. 9 is a flowchart illustrating an example of processing by the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0028] <Embodiment> Hereinafter, a specific embodiment of the information processing system 1 will be described with reference to the drawings. [Overall configuration] 1, the information processing system 1 includes an information processing device 10, a management device 20, and a server device 30. The information processing device 10, the management device 20, and the server device 30 are connected to each other via a network NW so that they can send and receive information. The network NW includes, for example, the Internet, a wide area network (WAN), a local area network (LAN), a provider terminal, a wireless communication network, a wireless base station, a dedicated line, and the like. Note that the network NW may include a local network in part.

[0029] The management device 20 includes a plurality of imaging units 21. The management device 20 and the imaging units 21 are connected so as to be able to send and receive information. Hereinafter, it is assumed that the management device 20 and the imaging units 21 are directly connected. The imaging units 21 are provided at predetermined positions within the premises of a factory, warehouse, or the like where cargo is transported, where they are able to capture images of the travel path of a forklift TG1 moving within the premises. The plurality of imaging units 21 are provided at different positions, and each captures an image of the forklift TG1 traveling along the travel path.

[0030] The information processing system 1 executes the processes from step ST1 to step ST4. Each step ST will be explained below. In procedure ST1, the management device 20 transmits the video MV generated by the imaging unit 21 to the server device 30 via the network NW. The video MV is a video in which images IMa are generated in chronological order by the imaging unit 21 capturing images of the travel path. The video MV is generated, for example, every predetermined time, and is transmitted to the server device 30 by the management device 20 as soon as it is generated. The predetermined time is, for example, several minutes to several tens of minutes. The video MV also includes information indicating the date and time when the video MV was captured and generated, and imaging unit identification information that can identify the imaging unit 21 that captured and generated the video MV.

[0031] The server device 30 is realized by a cloud server including a non-transitory storage medium such as a hard disk drive (HDD), flash memory, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), or random access memory (RAM). The server device 30 stores the moving image MV received from the management device 20 via the network NW in a storage unit.

[0032] In step ST2, the information processing device 10 reads the moving image MV from the storage unit of the server device 30 via the network NW. In step ST3, the information processing device 10 determines, based on the acquired information, whether the load carried by the forklift TG1 satisfies the high-stacking condition when the forklift TG1 travels along the travel route. Specifically, the information processing device 10 determines whether the load carried by the forklift TG1 is stacked to an appropriate height when the forklift TG1 travels along the travel route. If the information processing device 10 determines that the high-stacking condition has been violated, it generates a violation video MV1 showing the scene where the violation event EV1 occurred, assuming that a violation event EV1 has occurred. If the information processing device 10 determines that the high-stacking condition has been complied with, it generates a compliance video MV2 showing the scene where the compliance event EV2 occurred, assuming that a compliance event EV2 has occurred.

[0033] In step ST4, the information processing device 10 provides a screen prompting the user to check information related to the violation event EV1 and the compliance event EV2. Details of the various configurations and processes used by the information processing system 1 to realize the processes of steps ST1 to ST4 will be described below.

[0034] [Configuration of information processing device 10] 2, the information processing device 10 includes, for example, a communication unit 11, a control unit 100, and a storage unit 150. The communication unit 11 includes, for example, a communication circuit or a communication module, and transmits and receives various data to and from the management device 20, the server device 30, and other devices via the network NW. The communication unit 11 performs communication via wired or wireless communication. In the case of wireless communication, the communication mode of the communication unit 11 may be short-range wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0035] The control unit 100 is realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (not shown) that includes a non-transitory storage medium such as an HDD or flash memory provided in the storage unit 150.

[0036] The storage unit 150 may be realized by the various storage devices described above, or an EEPROM, a ROM, a RAM, etc. The storage unit 150 stores, for example, a trained model 151, a violation event video folder 153, a compliance event video folder 154, and judgment result information 155. These will be described in detail later.

[0037] The control unit 100 includes a ratio calculation unit 101, a loading determination unit 102, a position identification unit 103, a stop determination unit 104, an angle calculation unit 105, an angle determination unit 106, an orientation identification unit 107, a movement direction identification unit 108, a behavior identification unit 109, a high stacking determination unit 110, an animation generation unit 111, a determination result generation unit 112, and a presentation unit 113.

[0038] Based on the image IMa, the ratio calculation unit 101 calculates the ratio between the height of the load and the height of the forklift TG1 using the height of the load and the height of the forklift TG1. Based on the video MV, the loading determination unit 102 determines whether the forklift TG1 is loaded. The position identification unit 103 identifies the height of the load loaded by the forklift TG1 and the height of the forklift TG1 in the image IMa. The stop determination unit 104 determines whether the forklift TG1 is traveling or stopped. The angle calculation unit 105 calculates the angle of the forklift TG1 in each image IMa that constitutes the video MV. The angle determination unit 106 determines whether the angle of the forklift TG1 calculated by the angle calculation unit 105 is an angle suitable for determining whether the forklift TG1 is heavily loaded.

[0039] The orientation identification unit 107 identifies the orientation of the forklift TG1 based on feature points of the forklift TG1 shown in the video MV. The movement direction identification unit 108 identifies the traveling direction of the forklift TG1 and, based on the identified traveling direction and orientation of the forklift TG1, identifies whether the forklift TG1 is moving forward or backward. The behavior identification unit 109 identifies the behavior of the forklift TG1 based on the video MV. The high-stacking determination unit 110 determines whether the forklift TG1 violates the high-stacking conditions. The video generation unit 111 generates a violation video MV1 including scenes in which the forklift TG1 violates the high-stacking conditions. The video generation unit 111 also generates a compliance video MV2 including scenes in which the forklift TG1 complies with the high-stacking conditions. The judgment result generation unit 112 generates judgment result information 155 including a judgment result corresponding to the violation video MV1 and a judgment result corresponding to the compliance video MV2. The presentation unit 113 generates an image used to present the judgment result information 155 to the user and presents the image to the user.

[0040] [Regarding the position identification unit 103] The detailed processing of each unit will be described below. The position identification unit 103 identifies an image IMa that shows the forklift truck TG1 from among the images IMa that make up the video MV read from the server device 30. This identification process is performed by the trained model 151. The position identification unit 103 also inputs, to the trained model 151, an image IMa that makes up the video MV that shows the forklift truck TG1 from among the images IMa that make up the video MV. When the trained model 151 receives an image IMa that makes up the video MV, it outputs information indicating the position of the body of the forklift truck TG1 on the image IMa. When the trained model 151 receives an image IMa that makes up the video MV, it outputs information indicating the position of the load carried by the forklift truck TG1 on the image IMa. When the trained model 151 receives an image IMa that makes up the video MV, it outputs information indicating the position of the mast and balance weight of the forklift truck TG1. The body position of the forklift truck TG1 is the position of the body of the forklift truck TG1 excluding the forks and mast. Therefore, when the trained model 151 receives an image IMa that constitutes the video MV, it outputs the position of the load and the position of the forklift TG1.

[0041] Specifically, for example, when image IMa is input, the trained model 151 identifies the area on image IMa where the main body of the forklift TG1 is located by using a bounding box BXa. Therefore, the trained model 151 outputs the coordinates of the bounding box BXa as the partial area where the forklift TG1 is located, as information indicating the position of the main body of the forklift TG1. In other words, when image IMa constituting the video MV is input, the trained model 151 outputs a bounding box BXa that indicates the position of the forklift TG1 on image IMa. Furthermore, when image IMa is input, the trained model 151 identifies the area on image IMa where the load carried on the forks of the forklift TG1 is located by using a bounding box BXb. Therefore, the trained model 151 outputs the coordinates of the bounding box BXb as the partial area where the load is located, as information indicating the position of the load carried by the forklift TG1. That is, when the trained model 151 receives an image IMa that constitutes the video MV, it outputs a bounding box BXb that indicates the position of the load on the image IMa. Therefore, when the trained model 151 receives an image IMa that constitutes the video MV, it outputs a bounding box BX that indicates the position of the load and the position of the forklift TG1. In other words, the position identification unit 103 identifies the height of the load carried by the forklift TG1 and the height of the forklift TG1 based on the input video MV.

[0042] The coordinates of the bounding boxes BXa and BXb are indicated by, for example, the coordinates of two diagonal points. As described above, the bounding box BXa indicates the main body position of the forklift TG1 shown in image IMa, excluding the forks and mast, as the position of the forklift TG1. In the following description, the main body position of the forklift TG1, excluding the forks and mast, will also be simply referred to as the main body position of the forklift TG1.

[0043] Furthermore, when an image IMa is input, the trained model 151 outputs a first position, which is the position of the balance weight on the image IMa, and a second position, which is the position of the mast on the image IMa, of the forklift TG1. The position identification unit 103 inputs each of the images IMa constituting the video MV to the trained model 151, and acquires, for each image IMa, information indicating the first position and the second position shown in the image IMa.

[0044] 5 shows image IMa2, one of the images IMa constituting the video MV, showing a scene in which a forklift truck TG1 is loading a load onto its forks. Image IMa2 shows a bounding box BXc indicating the position of the balance weight of the forklift truck TG1 and a bounding box BXd indicating the position of the mast of the forklift truck TG1.

[0045] That is, for example, when image IMa is input, the trained model 151 identifies the area on image IMa where the balance weight of the forklift TG1 is located by using a bounding box BXc. Furthermore, when image IMa is input, the trained model 151 identifies the area on image IMa where the mast of the forklift TG1 is located by using a bounding box BXd. Therefore, the trained model 151 outputs the coordinates of the bounding box BXc as a partial area that identifies the balance weight of the forklift TG1 as information indicating the first position. Furthermore, the trained model 151 outputs the coordinates of the bounding box BXd as a partial area that identifies the mast of the forklift TG1 as information indicating the second position. The coordinates of the bounding boxes BXc and BXd are indicated, for example, by the coordinates of two diagonal points. Therefore, the trained model 151 outputs the coordinates of the main body position, the load, the balance weight, and the mast of the forklift TG1.

[0046] [Regarding the ratio calculation unit 101] The ratio calculation unit 101 calculates a ratio based on the position of the forklift TG1 and the position of the load identified by the position identification unit 103 on the image IMa. The ratio calculation unit 101 identifies the height dimension (length of the vertical axis) of the bounding box BXa from the coordinates of two diagonal points indicating the coordinates of the bounding box BXa as the height of the forklift TG1. The ratio calculation unit 101 also identifies the height dimension (length of the vertical axis) of the bounding box BXb from the coordinates of two diagonal points indicating the coordinates of the bounding box BXb as the height of the load loaded on the forklift TG1. The ratio calculation unit 101 then calculates the ratio between the height of the load and the height of the forklift TG1 using the identified height of the load and the height of the forklift TG1. Note that if the trained model 151 outputs the height of the bounding box BX (length of the vertical axis), the ratio calculation unit 101 can calculate the ratio without identifying the respective heights. Specifically, the ratio calculation unit 101 calculates the ratio by dividing the height of the load by the height of the forklift TG1.

[0047] 3 shows image IMa1, one of the images IMa constituting the video MV, showing a scene in which a forklift truck TG1 is loading a load onto its forks. Image IMa1 shows a bounding box BXa indicating the position of the forklift truck TG1's body and a bounding box BXb indicating the position of the load on the forks.

[0048] [Regarding the loading determination unit 102] The loading determination unit 102 determines whether the forklift TG1 shown in the image IMa is carrying a load on its fork, based on the load identified on the image IMa by the position identification unit 103. Specifically, the loading determination unit 102 determines that the forklift TG1 is carrying a load when the trained model 151 identifies the load by a bounding box BXb.

[0049] [Regarding the stop determination unit 104] The stop determination unit 104 determines, based on the video MV, whether the forklift TG1 shown in the video MV is stopped, i.e., whether the forklift TG1 is traveling. Specifically, the stop determination unit 104 determines the speed of the forklift TG1 based on the fluctuation of the bounding box BXa representing the forklift TG1 output by the trained model 151. First, the stop determination unit 104 determines the distance traveled by the forklift TG1 within a predetermined time period after the forklift TG1 appears in the image IMa constituting the video MV (i.e., after the forklift TG1 enters the angle of view of the imaging unit 21), based on the fluctuation of the bounding box BXa output by the trained model 151. The predetermined time period is, for example, several seconds to several tens of seconds. More specifically, the stop determination unit 104 determines the speed of the forklift TG1 based on the distance from the position where the bounding box BXa representing the forklift TG1 appears in the image IMa to the position of the bounding box BXa after the predetermined time period has elapsed. The stop determination unit 104 determines that the forklift TG1 is stopped when the identified speed is less than a predetermined speed or is zero. The predetermined speed is, for example, a speed close to zero. Furthermore, the stop determination unit 104 determines that the forklift TG1 is not stopped when the identified speed remains equal to or greater than the predetermined speed for a predetermined period of time or longer. In other words, the stop determination unit 104 determines that the forklift TG1 is traveling when the forklift TG1 shown in the video MV continues to move for a predetermined period of time or longer.

[0050] [Angle calculation unit 105 and angle determination unit 106] The angle calculation unit 105 calculates the angle of the forklift TG1 in each image IMa shown in the video MV based on the video MV. The angle determination unit 106 determines whether the angle of the forklift TG1 calculated by the angle calculation unit 105 is an angle suitable for determining whether the loads carried by the forklift TG1 are piled high. Here, if the forklift TG1 shown in the image IMa is facing the imaging unit 21 directly, the height of the loads carried by the forklift TG1 may not be determined appropriately. Therefore, in order to determine whether the height of the loads carried by the forklift TG1 is piled high, it is preferable that the forklift TG1 be shown sideways in the image IMa.

[0051] As shown in FIG. 4, when the forklift TG1 is viewed from above, the angle of the forklift TG1 is defined as 0 degrees when the imaging unit 21 is located behind the forklift TG1 with the forklift TG1 at the center. When the forklift TG1 is viewed from above, the angle of the forklift TG1 increases counterclockwise around the forklift TG1. Examples of ranges in which the forklift TG1 is facing sideways include ranges of 45 degrees to 135 degrees and 225 degrees to 315 degrees. In other words, when the angle of the forklift TG1 shown in the image IMa calculated by the angle calculation unit 105 is within the ranges of 45 degrees to 135 degrees and 225 degrees to 315 degrees, the angle determination unit 106 determines that the angle is appropriate for determining whether the forklift TG1 is stacked high.

[0052] As shown in FIG. 5, the angle calculation unit 105 identifies a center point TP1 of the area on the image IMa identified as the balance weight based on the coordinates of the bounding box BXc output from the trained model 151. The center point TP1 is, for example, the position of the center of gravity of the bounding box BXc. Furthermore, the angle calculation unit 105 identifies a center point TP2 of the area on the image IMa identified as the mast based on the coordinates of the bounding box BXd output from the trained model 151. The center point TP2 is, for example, the position of the center of gravity of the bounding box BXd.

[0053] The screen coordinate system, which is the coordinate system when the image IMa is displayed on the screen, is a coordinate system based on the upper left corner of the screen, with the left-right direction of the screen defined as the X-axis direction and the up-down direction of the screen defined as the Y-axis direction. The angle calculation unit 105 determines a first lift direction DRa, which indicates the direction from the center point TP1 to the center point TP2, in the X-axis direction of the screen coordinate system of the image IMa. The direction from the center point TP1 to the center point TP2 is the direction from the balance weight to the mast. If the direction from the center point TP1 to the center point TP2 is defined as the orientation of the forklift TG1, the first lift direction DRa coincides with the X-component of the orientation of the forklift TG1. The processed images AL1 and AL2 in FIG. 5 are diagrams schematically illustrating the processing of the angle calculation unit 105. In the example shown in the processed image AL1, the angle calculation unit 105 determines the leftward direction, which is the direction from the center point TP1 to the center point TP2, as the first lift direction DRa in the X-axis direction of the screen coordinate system of the image IMa.

[0054] The angle calculation unit 105 identifies the bottom endpoints of the four sides representing the bounding boxes BXc and BXd on the side of the first lift direction DRa. In the example shown in processed image AL2, the angle calculation unit 105 identifies the lower left endpoint TP3 of the bounding box BXc and the lower left endpoint TP4 of the bounding box BXd. Next, the angle calculation unit 105 calculates, as the angle of the forklift TG1, the angle AG formed by a line LN1 extending in the Y-axis direction from the identified bottom endpoint (lower left endpoint TP3) of the bounding box BXc and a line segment LN2 connecting the identified two bottom endpoints (lower left endpoint TP3 and TP4), the line segment LN2 extending clockwise from the line segment LN2 with the identified bottom endpoint (lower left endpoint TP3) of the bounding box BXc as the center.

[0055] As shown in processed image AL1, the angle calculation unit 105 may calculate, as the angle of the forklift TG1, the angle AG' between a straight line LN1' extending from the center point TP1 in the Y-axis direction and a line segment LN2' connecting the center points TP1 and TP2, without identifying the bottom end points of the four sides of the bounding boxes BXc and BXd on the first lift direction DRa side. In this case, the angle AG can be calculated while suppressing the influence of fluctuations in the coordinates of the bounding boxes BXc and BXd.

[0056] The angle determination unit 106 determines whether the forklift TG1 is facing sideways based on the angle AG calculated by the angle calculation unit 105. If the angle AG is within a range indicating a sideways orientation, the angle determination unit 106 determines that the forklift TG1 is facing sideways. If the angle AG is outside the range indicating a sideways orientation, the angle determination unit 106 determines that the forklift TG1 is not facing sideways.

[0057] [Regarding the orientation identification unit 107] The orientation identification unit 107 identifies the midpoint at the bottom end of the bounding box BXc in the area on the image IMa identified as the balance weight, based on the coordinates of the bounding box BXc output from the trained model 151. Furthermore, the orientation identification unit 107 identifies the midpoint at the bottom end of the bounding box BXd in the area on the image IMa identified as the mast, based on the coordinates of the bounding box BXd output from the trained model 151.

[0058] The orientation determination unit 107 determines the orientation of the forklift TG1, which indicates the direction from the midpoint at the bottom of the bounding box BXc to the midpoint at the bottom of the bounding box BXd in the X-axis direction in the screen coordinate system of the image IMa. Furthermore, the direction from the midpoint at the bottom of the bounding box BXc to the midpoint at the bottom of the bounding box BXd is the direction from the balance weight to the mast. Therefore, the orientation determination unit 107 determines the orientation of the forklift TG1 based on feature points of the forklift TG1 shown in the video MV. More specifically, for each image IMa constituting the video MV, the orientation determination unit 107 determines a first position, which is the position of the balance weight of the forklift TG1 on the image IMa, and a second position, which is the position of the mast of the forklift TG1 on the image IMa, and determines the orientation of the forklift TG1 based on the positional relationship between the first position and the second position.

[0059] In addition, instead of identifying the midpoint at the bottom end of the bounding box BX, the orientation determination unit 107 may identify the bottom end point (bottom left end point or bottom right end point) on the first lift direction DRa side, and determine the orientation of the forklift TG1 using the direction from the bottom end point of the bounding box BXc to the bottom end point of the bounding box BXd.

[0060] [Regarding the moving direction identification unit 108] The movement direction identification unit 108 identifies whether the forklift TG1 is moving forward or backward based on the identified traveling direction and orientation of the forklift TG1. First, the movement direction identification unit 108 identifies the traveling direction of the forklift TG1 based on changes over time in the coordinates of a bounding box BXa indicating the forklift TG1 identified by the trained model 151. That is, the movement direction identification unit 108 identifies the traveling direction of the forklift TG1 shown in the video MV based on the video MV. Note that the traveling direction of the forklift TG1 may also be identified based on changes over time in the coordinates of a bounding box BXc, which is information indicating a first position identified by the trained model 151, or the coordinates of a bounding box BXd, which is information indicating a second position.

[0061] Next, the movement direction identification unit 108 determines whether the identified traveling direction and orientation of the forklift TG1 match. Specifically, the movement direction identification unit 108 determines whether the angle between the identified traveling direction and the orientation of the forklift TG1 identified by the orientation identification unit 107 is within a predetermined angle. If the angle between the identified traveling direction and orientation of the forklift TG1 is within the predetermined angle, it is determined that the identified traveling direction and orientation of the forklift TG1 match, that is, the forklift TG1 is moving forward. If the angle between the identified traveling direction and orientation of the forklift TG1 is greater than the predetermined angle, it is determined that the identified traveling direction and orientation of the forklift TG1 do not match, that is, the forklift TG1 is moving backward. The predetermined angle is, for example, 50 degrees.

[0062] For example, if the direction from the midpoint at the bottom end of the balance weight bounding box BXc to the midpoint at the bottom end of the mast bounding box BXd is rightward, the direction of the forklift TG1 identified by the direction identification unit 107 is determined to be rightward. If the traveling direction of the forklift TG1 identified by the movement direction identification unit 108 is rightward, the forklift TG1 is determined to be moving forward. Also, if the direction from the midpoint at the bottom end of the balance weight bounding box BXc to the midpoint at the bottom end of the mast bounding box BXd is leftward, the direction of the forklift TG1 identified by the direction identification unit 107 is determined to be leftward. If the traveling direction of the forklift TG1 identified by the movement direction identification unit 108 is rightward, the forklift TG1 is determined to be moving backward.

[0063] [About the behavior identification unit 109] The behavior identification unit 109 identifies the behavior of the forklift TG1 shown in the video MV based on the video MV. The behavior identification unit 109 identifies the behavior of the forklift TG1 based on, for example, changes over time in the position of the bounding box BX identified by the trained model 151. Hereinafter, the behavior of the forklift TG1 is assumed to include the speed, maximum speed, minimum speed, acceleration, and duration of time during which the speed of the forklift TG1 remains equal to or less than the speed threshold value of the forklift TG1. Note that the behavior of the forklift TG1 is merely an example and is not limited thereto, and may include some of these indicators or other indicators.

[0064] [About the high-piling determination unit 110] The high-stacking determination unit 110 determines whether the forklift TG1 violates the high-stacking condition. For example, when the loading determination unit 102 determines that a load is loaded and the stop determination unit 104 determines that the forklift TG1 is not stopped and is facing sideways, the high-stacking determination unit 110 determines whether the forklift TG1 violates the high-stacking condition. On the other hand, when the loading determination unit 102 determines that no load is loaded and the stop determination unit 104 determines that the forklift TG1 is stopped and is not facing sideways, the high-stacking determination unit 110 does not determine whether the forklift TG1 violates the high-stacking condition.

[0065] When it is determined that the forklift TG1 is loaded with a load, that the forklift TG1 is not stopped, and that the forklift TG1 is facing sideways, the high-piling determination unit 110 determines whether the ratio calculated by the ratio calculation unit 101 is equal to or greater than a predetermined threshold. Therefore, the high-piling determination unit 110 determines whether a violation event EV1 has occurred based on the video MV only when it is determined that the forklift TG1 is traveling. When the ratio is equal to or greater than the predetermined threshold, the high-piling determination unit 110 determines that a violation event EV1 has occurred, in which the forklift TG1 has violated the high-piling condition. Therefore, when the ratio calculated by the ratio calculation unit 101 is equal to or greater than the predetermined threshold, the high-piling determination unit 110 determines that a violation event EV1 has occurred, in which the high-piling condition has been violated.

[0066] Here, the predetermined threshold value varies depending on the behavior of the forklift truck TG1. In the following description, the predetermined threshold value when the forklift truck TG1 is moving forward will be referred to as the forward threshold value, the predetermined threshold value when the forklift truck TG1 is moving backward will be referred to as the reverse threshold value, and the predetermined threshold value when the forklift truck TG1 is turning will be referred to as the turning threshold value. In other words, the forward threshold value is a threshold value used when the forklift truck TG1 is moving forward. The reverse threshold value is a threshold value used when the forklift truck TG1 is moving backward. The turning threshold value is a threshold value used when the forklift truck TG1 is turning. The above-mentioned predetermined threshold values ​​include the forward threshold value, the reverse threshold value, and the turning threshold value. The forward threshold value is smaller than the reverse threshold value, and the turning threshold value is smaller than the reverse threshold value.

[0067] The high stacking determination unit 110 applies a forward threshold or a reverse threshold based on the traveling direction identified by the movement direction identification unit 108. Furthermore, the high stacking determination unit 110 applies a turning threshold when the stop determination unit 104 determines that the forklift TG1 is not stopped, but the traveling direction of the forklift TG1 is not identified by the movement direction identification unit 108, or the identified traveling direction changes appropriately.

[0068] Furthermore, when it is determined that the forklift TG1 is loaded with a load, that the forklift TG1 is not stopped, and that the forklift TG1 is facing sideways, the high-stacking determination unit 110 determines whether the ratio calculated by the ratio calculation unit 101 is less than a predetermined threshold value. If the ratio is less than the predetermined threshold value based on the calculated ratio, the high-stacking determination unit 110 determines that the forklift TG1 has complied with the high-stacking condition, and a compliance event EV2 has occurred.

[0069] [About the video generation unit 111] When the high pile determination unit 110 determines that a violation event EV1 has occurred, the video generation unit 111 generates a violation video MV1 by extracting a portion of the video MV that includes an image IMa of the scene where the violation event EV1 has occurred from the video MV. Also, when the high pile determination unit 110 determines that a compliance event EV2 has occurred, the video generation unit 111 generates a compliance video MV2 by extracting a portion of the video MV that includes an image IMa of the scene where the compliance event EV2 has occurred from the video MV.

[0070] Here, the image IMa of the scene where the violation event EV1 occurred is, for example, the image IMa of the image MV at the timing when the violation event EV1 is determined to have occurred by the high-piling determination unit 110. Also, the image IMa of the screen where the compliance event EV2 occurred is, for example, the image IMa of the image MV at the timing when the compliance event EV2 is determined to have occurred by the high-piling determination unit 110.

[0071] Furthermore, the video generation unit 111 identifies the position of the forklift TG1 shown in the violation video MV1 based on the coordinates of the bounding box BX identified by the trained model 151 for each image IMa constituting the violation video MV1. Then, the video generation unit 111 generates, as the violation video MV1, a video in which the bounding box BX indicating the identified position of the forklift TG1 is superimposed on each image IMa constituting the violation video MV1. That is, the violation video MV1 shows the forklift TG1 and the bounding box BX indicating the position of the forklift TG1 when the violation event EV1 occurred. The video generation unit 111 also performs the same process on the compliance video MV2. That is, the compliance video MV2 shows the forklift TG1 and the bounding box BX indicating the position of the forklift TG1 when the compliance event EV2 occurred.

[0072] Hereinafter, the video generator 111 will generate as a violation video MV1 a portion of the video MV from when the forklift TG1, for which it has been determined that a violation event EV1 has occurred, is displayed in image IMa until a predetermined time has elapsed or the forklift TG1 moves out of the range displayed in image IMa. Also, the video generator 111 will generate as a compliance event EV2 a portion of the video MV from when the forklift TG1, for which it has been determined that a compliance event EV2 has occurred, is displayed in image IMa until a predetermined time has elapsed or the forklift TG1 moves out of the range displayed in image IMa.

[0073] The video generation unit 111 stores the generated violation video MV1 in a violation event video folder 153 of the storage unit 150, while storing the generated compliance video MV2 in a compliance event video folder 154 of the storage unit 150, within the storage area of ​​the storage unit 150. The violation event video folder 153 is an example of a first storage location, and the compliance event video folder 154 is an example of a second storage location.

[0074] [Regarding the judgment result generation unit 112 and the judgment result information 155] The judgment result generating unit 112 generates / updates judgment result information 155 indicating the judgment result corresponding to the violation video MV1 and the judgment result corresponding to the compliance video MV2.

[0075] 6, the determination result information 155 includes one or more records in which the following information is associated with each other: the date and time when the violation event EV1 or the compliance event EV2 occurred, the playback time of the violation video MV1 or the compliance video MV2 corresponding to that time, the event number, image capture unit identification information that can identify the image capture unit 21, the name of the forklift TG1, moving object identification information that can identify the forklift TG1, the determination type, the determination result, the determination reason, the average speed of the forklift TG1, the maximum speed of the forklift TG1, the minimum speed of the forklift TG1, the duration for which the forklift TG1 continued to be equal to or less than the speed threshold, and the speed threshold. The determination result information 155 is realized, for example, by a data file in CSV (Comma Separated Values) format.

[0076] Each time the high-piling judgment unit 110 judges that a violation event EV1 has occurred or that a compliance event EV2 has occurred, the judgment result information 155 generates / updates a record and adds the generated record to the judgment result information 155. In other words, each time the video generation unit 111 generates a violation video MV1 or a compliance video MV2, the judgment result information 155 generates / updates a record.

[0077] The judgment result generation unit 112 identifies the date and time when the violation event EV1 or the compliance event EV2 occurred for the moving image MV that is the target of judgment by the high pile judgment unit 110, based on information indicating the date and time included in the moving image MV. Specifically, the judgment result generation unit 112 identifies the date and time a predetermined time before the high pile judgment unit 110 judges that the violation event EV1 or the compliance event EV2 occurred. The predetermined time is, for example, several seconds (specifically, about 3 seconds). The judgment result generation unit 112 also identifies the playback time of the moving image MV on the date and time when the violation event EV1 or the compliance event EV2 occurred.

[0078] Furthermore, the judgment result generating unit 112 identifies the event number assigned to the violation event EV1 or the compliance event EV2. The event number is, for example, a serial number that is common to the violation event EV1 and the compliance event EV2. Therefore, the judgment result generating unit 112 identifies the number next to the event number of the violation event EV1 or compliance event EV2 that occurred immediately before as the event number of the violation event EV1 or compliance event EV2 that occurred next. Note that the event number may be a serial number for each violation event EV1 and each compliance event EV2.

[0079] The determination result generation unit 112 identifies the imaging unit identification information included in the moving image MV for the moving image MV that is the determination target of the high-piling determination unit 110. In this example, the determination result generation unit 112 identifies all moving object names as forklift TG1.

[0080] The judgment result generation unit 112 identifies the moving object identification information based on the corresponding violation video MV1 or compliance video MV2. Here, the forklift TG1 is each assigned with the moving object identification information. Specifically, vehicles such as the forklift TG1 are assigned with the moving object identification information indicated by a barcode or the like in a position such as the top surface that can be imaged by the imaging unit 21. The judgment result generation unit 112 identifies the moving object identification information of the forklift TG1 being judged by performing general image processing on the violation video MV1 or compliance video MV2.

[0081] The judgment result generation unit 112 identifies the judgment type based on the processing performed by the high pile judgment unit 110. In this example, the high pile judgment unit 110 makes a judgment regarding high piles, and therefore the judgment type is high piles in all cases. The judgment result generation unit 112 identifies the judgment result based on the processing performed by the high pile judgment unit 110. In this example, if the high pile judgment unit 110 determines that a violation event EV1 has occurred, the judgment result is "NG," and if the high pile judgment unit 110 determines that a compliance event EV2 has occurred, the judgment result is "OK." The judgment result generation unit 112 identifies the judgment reason based on the processing performed by the high pile judgment unit 110. In this example, if the high pile judgment unit 110 determines that a violation event EV1 has occurred, the judgment reason is "high piles present," and if the high pile judgment unit 110 determines that a compliance event EV2 has occurred, the judgment result is "no high piles present."

[0082] The determination result generation unit 112 identifies the average speed of the forklift TG1 based on the processing performed by the behavior identification unit 109. The determination result generation unit 112 also identifies the maximum speed of the forklift TG1 based on the processing performed by the behavior identification unit 109. The determination result generation unit 112 also identifies the minimum speed of the forklift TG1 based on the processing performed by the behavior identification unit 109. The determination result generation unit 112 also identifies the time during which the speed of the forklift TG1 continues to be equal to or less than the speed threshold value based on the processing performed by the behavior identification unit 109.

[0083] The judgment result generating unit 112 generates / updates the judgment result information 155 by generating a record that associates the various pieces of information identified by the above-mentioned processing with each other for each corresponding violation video MV1 and compliance video MV2.

[0084] Furthermore, the video generation unit 111 may determine the title of the violation video MV1 corresponding to the violation event EV1 and the title of the compliance video MV2 corresponding to the compliance event EV2 based on the judgment result information 155 generated / updated by the judgment result generation unit 112. The video generation unit 111 generates the violation video MV1 and the compliance video MV2 by giving them a title such as "(event number)_(judgment name)_(moving object name)_(judgment result).mp4", for example.

[0085] [About presentation unit 113] The presentation unit 113 generates a confirmation screen IMb so as to present the determination result information 155 to the user in an easy-to-understand manner. The confirmation screen IMb is a screen that shows the occurrence status of the violation event EV1 for each imaging unit 21. The presentation unit 113 presents the generated confirmation screen IMb to the user of the management device 20. Specifically, the presentation unit 113 makes the generated confirmation screen IMb available for reference by the management device 20 via the network NW.

[0086] As shown in FIG. 7, the confirmation screen IMb includes a designation image IMc, a map image IMd, and a slider bar image IMe. The designation image IMc includes, for example, pull-down lists PD1 to PD4. The pull-down list PD1 is a pull-down list used to designate the start of a period of time desired to be confirmed among the records included in the determination result information 155. The pull-down list PD2 is a pull-down list used to designate the end of a period of time desired to be confirmed among the records included in the determination result information 155. The pull-down list PD3 is a pull-down list used to select a determination type of time desired to be confirmed among the records included in the determination result information 155. The pull-down list PD4 is a pull-down list used to select an imaging unit 21 of time desired to be confirmed among the records included in the determination result information 155.

[0087] The map image IMd is, for example, an image that schematically shows a premises or the like in which the imaging unit 21 is installed. Furthermore, icons ICN are shown at positions on the map image IMd that correspond to positions where the imaging unit 21 is installed on the travel route of the schematically shown premises or the like. In this example, seven imaging units 21, namely imaging units 21a to 21g, are installed in the premises or the like. Therefore, seven icons ICNa to ICNg are shown at corresponding positions on the map image IMd. The map image IMd is generated in advance for each premises to be determined and stored in the storage unit 150. Furthermore, of the icons ICN that represent the imaging units 21, a symbol image SY is shown superimposed on a specific icon ICN. The symbol image SY is, for example, an image that represents the occurrence status of a violation event EV1. Details of the specific icon ICN on which the symbol image SY is superimposed will be described later.

[0088] The slider bar image IMe includes, for example, a slider bar SB, a start point slider CS1, and an end point slider CS2. The slider bar SB represents the entire period specified by the pull-down lists PD1 and PD2. In this example, since a single day is specified by the pull-down lists PD1 and PD2, the slider bar SB represents the period from midnight to midnight. On the other hand, if multiple days are specified by the pull-down lists PD1 and PD2, the slider bar SB represents the multiple days. If several hours are specified by the pull-down lists PD1 and PD2, the slider bar SB represents the several hours. The start point slider CS1 and the end point slider CS2 are used to specify a specific period to be checked within the period indicated by the slider bar SB. In the initial state, the start point slider CS1 and the end point slider CS2 are located at both ends of the slider bar SB so as to specify the entire period of the pull-down lists PD1 and PD2.

[0089] The user of the management device 20 operates the pull-down lists PD1 to PD4 and the start point slider CS1 and end point slider CS2 of the slider bar SB based on the confirmation screen IMb presented by the presentation unit 113. The presentation unit 113 acquires information indicating the operation by the user of the management device 20.

[0090] The presentation unit 113 identifies a record to be displayed as a map image IMd from records included in the determination result information 155, based on information specified using the pull-down list PD and the slider bar SB on the confirmation screen IMb. First, the presentation unit 113 searches the determination result information 155 from among the records included in the determination result information 155, using a date within the period specified by the pull-down lists PD1 to PD2 as a search key. The presentation unit 113 identifies and narrows down records associated with dates matching the specified date. Next, the presentation unit 113 searches for records in the identified determination result information 155, using a date and time within the period specified by the start point slider CS1 and the end point slider CS2 as a search key. The presentation unit 113 identifies and narrows down records matching the date and time. Furthermore, the presentation unit 113 searches for records in the identified determination result information 155, using a record of the determination type specified by the pull-down list PD3 and the target imaging unit 21 specified by the pull-down list PD4 as a search key. The presentation unit 113 identifies and narrows down the records that match the specified determination type and match the imaging unit identification information of the specified imaging unit 21.

[0091] The presentation unit 113 refers to the identified record and identifies information indicating the image capture unit identification information and the judgment result from the information included in the record. If the identified information indicating the judgment result indicates “NG,” the presentation unit 113 generates a map image IMd and updates the confirmation screen IMb so that a symbol image SY is superimposed on the icon ICN corresponding to the image capture unit identification information associated with the “NG” judgment result. The symbol image SY also includes a number image indicating the number of identified records for which the information indicating the judgment result in the image capture unit 21 corresponding to the superimposed icon ICN was “NG.” As shown in FIG. 7 , in this example, the symbol image SY superimposed on the icon ICNa indicates that seven “NG” judgments were made during a specified period at the position captured by the image capture unit 21a corresponding to the icon ICNa.

[0092] The presenting unit 113 also changes the display mode of the symbol image SY based on the number of occurrences of the violation event EV1 indicated in the judgment result information 155. For example, the presenting unit 113 may enlarge, compared to its normal size, the symbol image SY superimposed on the icon ICN of the imaging unit 21 capturing an image of a regulated location where a large number of violation events EV1 occurred during a period designated by the pull-down lists PD1 to PD2. As shown in FIG. 7, in this example, the symbol image SY superimposed on the icon ICNb indicates that one "NG" judgment was made during the designated period at the position captured by the imaging unit 21b corresponding to the icon ICNb. Therefore, the number of "NG" judgments made by the imaging unit 21b is smaller than the number of "NG" judgments made by the imaging unit 21a. Accordingly, the presenting unit 113 generates a map image IMd and updates the confirmation screen IMb so that the symbol image SY superimposed on the icon ICNa is displayed larger than the symbol image SY superimposed on the icon ICNb.

[0093] [Operation of information processing device 10] The process of the information processing device 10 in step ST3 will be described in detail below with reference to Fig. 8. The process of the flowchart shown in Fig. 8 is executed for a moving image MV every time the management device 20 transmits the moving image MV to the server device 30, for example.

[0094] First, the position identification unit 103 reads and acquires the video MV from the storage unit of the server device 30 via the network NW (step S200). The position identification unit 103 inputs the image IMa of the video MV to the trained model 151. For each image IMa, the trained model 151 outputs information indicating the position of the main body of the forklift TG1, the position of the load, the position of the balance weight, and the position of the mast, as coordinates. The position identification unit 103 acquires the output coordinates (step S202). Furthermore, the trained model 151 identifies a bounding box BX.

[0095] Next, the loading determination unit 102 determines whether the forklift TG1 shown in the image IMa constituting the moving image MV identified by the position identification unit 103 is carrying a load (step S204). If the position of the load has been identified by the trained model 151, that is, if the area on the image IMa where the load is present on the fork is identified by the bounding box BXb, the loading determination unit 102 determines that the forklift TG1 is carrying a load. If the loading determination unit 102 determines that the forklift TG1 is not carrying a load (step S204; NO), there is no need to determine whether the high stacking condition is violated, and the series of processes ends.

[0096] If the loading determination unit 102 determines that the forklift TG1 is loaded (step S204; YES), the angle calculation unit 105 calculates the angle of the forklift TG1 in each image IMa shown in the video MV based on the video MV (step S206). Specifically, the angle calculation unit 105 identifies a center point TP1 based on the coordinates of the bounding box BXc output from the trained model 151. Furthermore, the angle calculation unit 105 identifies a center point TP2 of the area on the image IMa identified as the mast based on the coordinates of the bounding box BXd output from the trained model 151. Next, the angle calculation unit 105 identifies a first lift direction DRa from the identified center points TP1 and TP2. Next, the angle calculation unit 105 identifies a lower left corner point TP3 of the bounding box BXc and a lower left corner point TP4 of the bounding box BXd. Next, the angle calculation unit 105 calculates the angle AG formed by the straight line LN1 and the line segment LN2 as the angle of the forklift TG1.

[0097] Next, the angle determination unit 106 determines whether the forklift TG1 is sideways based on the angle AG calculated by the angle calculation unit 105 (step S208). If the angle AG is within a range indicating a sideways orientation, the angle determination unit 106 determines that the forklift TG1 is sideways (step S208; YES). If the angle AG is outside the range indicating a sideways orientation, the angle determination unit 106 determines that the forklift TG1 is not sideways. If the angle determination unit 106 determines that the forklift TG1 is not sideways (step S208; NO), the forklift TG1 shown in the moving image MV acquired in step S200 is not appropriate as a moving image MV for determining whether or not the high stacking condition is violated, and therefore the series of processes is terminated.

[0098] Next, the stop determination unit 104 determines whether the forklift TG1 shown in the video MV is stopped, based on the video MV (step S210). The stop determination unit 104 identifies the speed of the forklift TG1 based on the change over time in the bounding box BXa representing the forklift TG1 output by the trained model 151. If the identified speed is less than a predetermined speed or is zero, the stop determination unit 104 determines that the forklift TG1 is stopped (step S210; YES). The predetermined speed is, for example, a speed close to zero. Furthermore, if the identified speed is equal to or greater than the predetermined speed, the stop determination unit 104 determines that the forklift TG1 is not stopped (step S210; NO). If the stop determination unit 104 determines that the forklift TG1 is stopped (step S210; YES), there is no need to determine whether the stopped forklift TG1 violates the high-stacking condition, and the process ends.

[0099] Next, for the moving image MV acquired in step S200, the ratio calculation unit 101 identifies the height of the forklift TG1 from the coordinates of the bounding box BXa output from the trained model 151, and identifies the height of the load from the coordinates of the bounding box BXb for each image IMa constituting the moving image MV. The ratio calculation unit 101 calculates the ratio between the height of the load and the height of the forklift TG1 using the identified heights of the load and the forklift TG1 (step S212).

[0100] Next, the orientation identification unit 107 identifies the orientation of the forklift TG1 based on the coordinates of the bounding boxes BXc and BXd output from the trained model 151 (step S214). Next, the movement direction identification unit 108 identifies the traveling direction of the forklift TG1 based on the change over time in the coordinates of the bounding box BXa identified by the trained model 151 (step S216).

[0101] Next, the movement direction specifying unit 108 specifies whether the moving forklift TG1 is moving forward or backward based on the specified orientation and traveling direction of the forklift TG1 (step S218).

[0102] The high stacking determination unit 110 determines whether the movement direction identification unit 108 has identified the forklift TG1 as moving forward (step S220). If the forklift TG1 has been identified as moving forward (step S220; YES), the high stacking determination unit 110 applies a forward movement threshold when determining whether the high stacking condition is violated (step S222). If the movement direction identification unit 108 has not identified the forklift TG1 as moving forward (step S220; NO), the high stacking determination unit 110 determines whether the forklift TG1 has been identified as moving backward (step S224). If the forklift TG1 has been identified as moving backward (step S224; YES), the high stacking determination unit 110 applies a reverse movement threshold when determining whether the high stacking condition is violated (step S226). If the forklift TG1 has not been identified as moving backward (step S224; NO), the high stacking determination unit 110 applies a turning threshold when determining whether the high stacking condition is violated (step S228).

[0103] Next, the behavior identification unit 109 identifies the behavior of the forklift TG1 shown in the video MV based on the video MV (step S230). Specifically, the behavior identification unit 109 identifies the speed, maximum speed, minimum speed, acceleration, and duration of time during which the speed of the forklift TG1 remains equal to or less than the speed threshold value, of the forklift TG1 based on the video MV.

[0104] Next, the high-stacking determination unit 110 determines whether a violation event EV1 has occurred in which the forklift TG1 has violated the high-stacking condition (step S232). If the ratio calculated by the ratio calculation unit 101 in step S212 is equal to or greater than a threshold value specified in any one of steps S222, S225, and S228, the high-stacking determination unit 110 determines that the violation event EV1 has occurred. If the ratio calculated by the ratio calculation unit 101 in step S212 is less than a threshold value specified in any one of steps S222, S226, and S228, the high-stacking determination unit 110 determines that the violation event EV1 has not occurred (i.e., a compliance event EV2 has occurred).

[0105] When the high-piling determination unit 110 determines that the violation event EV1 has occurred (step S232; YES), the video generation unit 111 cuts out a portion of the video MV that includes an image IMa of the scene where the violation event EV1 has occurred, to generate a violation video MV1 (step S234). Specifically, the video generation unit 111 generates, as the violation video MV1, a portion from when the forklift TG1, in which it has been determined that the violation event EV1 has occurred, is displayed in the image IMa until a predetermined time has elapsed or the forklift TG1 moves out of the range displayed in the image IMa. Next, the video generation unit 111 generates, as the violation video MV1, a bounding box BX indicating the position of the forklift TG1 identified by the trained model 151, by superimposing it on each image IMa of the cut-out video. Next, the video generation unit 111 stores the generated violation video MV1 in the violation event video folder 153 of the storage unit 150 (step S236).

[0106] When the high-piling determination unit 110 determines that the compliance event EV2 has occurred (step S232; NO), the video generation unit 111 generates a compliance video MV2 by cutting out a portion of the video MV that includes the image IMa of the screen where the compliance event EV2 has occurred (step S238). Specifically, the video generation unit 111 generates, as the compliance video MV2, a portion from when the forklift truck TG1, for which it has been determined that the compliance event EV2 has occurred, is displayed in the image IMa until a predetermined time has elapsed or the forklift truck TG1 moves out of the range displayed in the image IMa. Next, the video generation unit 111 generates the compliance video MV2 by superimposing a bounding box BX indicating the position of the forklift truck TG1 identified by the trained model 151 on each image IMa of the cut-out video. Next, the video generation unit 111 stores the generated compliance video MV2 in the compliance event video folder 154 of the storage unit 150 (step S240).

[0107] Next, the judgment result generating unit 112 generates / updates the judgment result information 155 indicating the judgment result corresponding to the violation video MV1 and the judgment result corresponding to the compliance video MV2 (step S242).

[0108] The process of the information processing device 10 in procedure ST4 will be described in detail below with reference to Fig. 9. The process of the flowchart shown in Fig. 9 is executed, for example, at the timing when the user of the management device 20 wishes to check the determination result information 155.

[0109] First, the presentation unit 113 generates a confirmation screen IMb and presents it to the user of the management device 20 (step S300). Next, the presentation unit 113 determines whether the user of the management device 20 has specified a period of time for confirmation using the pull-down lists PD1 to PD2, the start point slider CS1, and the end point slider CS2 on the confirmation screen IMb (step S302). If the pull-down lists PD1 to PD2, the start point slider CS1, and the end point slider CS2 have not been operated, the presentation unit 113 determines that no period has been specified (step S302; NO), and proceeds to step S306. If the pull-down lists PD1 to PD2, the start point slider CS1, and the end point slider CS2 have been operated to specify a period (step S302; YES), the presentation unit 113 acquires information indicating the period specified by the operation and identifies a record of the determination result information 155 that matches the period (step S304). The presentation unit 113 searches the determination result information 155 using the dates and times of the period specified by the pull-down lists PD1 to PD2 and the start point slider CS1 and end point slider CS2 as search keys. The presentation unit 113 identifies and narrows down records that match the dates of the period.

[0110] Next, the presentation unit 113 determines whether the user of the management device 20 has specified a judgment type to be confirmed using the pull-down list PD3 on the confirmation screen IMb (step S306). If the pull-down list PD3 has not been operated, the presentation unit 113 determines that no judgment type has been specified (step S306; NO), and proceeds to step S310. If the pull-down list PD3 has been operated to specify a judgment type (step S306; YES), the presentation unit 113 acquires information indicating the judgment type specified by the operation and identifies a record of the judgment result information 155 (step S308). The presentation unit 113 uses the judgment type specified by the pull-down list PD3 as a search key to search for the record of the judgment result information 155 specified in step S304, or searches for the judgment result information 155 if no record has been specified yet. The presentation unit 113 identifies and narrows down the records associated with the judgment type.

[0111] Next, the presentation unit 113 determines whether a target imaging unit desired to be confirmed has been designated by the user of the management device 20 using the pull-down list PD4 on the confirmation screen IMb (step S310). If the pull-down list PD4 has not been operated, the presentation unit 113 determines that a target imaging unit has not been designated (step S310; NO), and proceeds to step S314. If the pull-down list PD4 has been operated to designate a target imaging unit (step S310; YES), the presentation unit 113 acquires information indicating the target imaging unit designated by the operation and identifies a record of the determination result information 155 (step S312). The presentation unit 113 searches for the record of the determination result information 155 identified in step S304 or step S308, using the target imaging unit designated by the pull-down list PD4 as a search key, or searches for the determination result information 155 if no record has yet been identified. The presentation unit 113 identifies and narrows down the records associated with the target imaging unit.

[0112] Next, if a period is acquired in step S304, the presentation unit 113 generates a map image IMd representing records for that period, or if a period is not acquired, generates records for the entire period (step S314). The presentation unit 113 updates the confirmation screen IMb including the map image IMd generated in step S314 and presents it to the user of the management device 20 (step S316), and then ends the series of processes.

[0113] [Effects of the embodiment] According to the above embodiment, the following effects can be obtained. (1) The loading determination unit 102 determines whether the forklift TG1 is loaded based on a video MV, which is a chronological sequence of images IMa generated by capturing images of the travel path using the imaging unit 21. When the forklift TG1 is loaded, the ratio calculation unit 101 calculates the ratio between the height of the load and the height of the forklift TG1 based on the position of the load output by inputting the images IMa into the trained model 151 and the position of the forklift TG1. Based on the calculated ratio, if the ratio is equal to or greater than a predetermined threshold, the high-piling determination unit 110 determines that a violation event EV1, in which the high-piling condition is violated, has occurred. Furthermore, based on the calculated ratio, if the ratio is less than the predetermined threshold, the high-piling determination unit 110 determines that a compliance event EV2, in which the high-piling condition is complied with, has occurred.

[0114] Here, it is time-consuming for a person to check the video MV and determine whether the forklift TG1 traveling along a route within a premises or the like is violating the high-stacking conditions regarding high-stacking of cargo. With this configuration, the high-stacking determination unit 110 can determine, based on the video MV, that a violation event EV1 has occurred in which the forklift TG1 has violated the high-stacking conditions.

[0115] (2) When the trained model 151 receives an input of each image IMa that constitutes the video MV, it outputs the position of the load and the position of the forklift TG1. Then, based on the information output by the trained model 151, the ratio calculation unit 101 calculates the ratio. Specifically, when the trained model 151 receives an input of each image IMa that constitutes the video MV, it outputs a bounding box BXb that indicates the position of the load and a bounding box BXa that indicates the position of the forklift TG1. Then, the ratio calculation unit 101 uses the coordinates of the bounding boxes BXa and BXb to determine the height and calculate the ratio.

[0116] According to this configuration, the ratio calculation unit 101 can calculate the ratio while identifying the height of the load and the height of the forklift TG1 by a simple method using the trained model 151.

[0117] (3) The stop determination unit 104 determines whether the forklift TG1 is traveling based on the video MV. When the stop determination unit 104 determines that the forklift TG1 is not traveling, the high-piling determination unit 110 does not perform processing to determine whether the high-piling condition is violated for the forklift TG1 that is not traveling. Furthermore, the high-piling determination unit 110 performs processing to determine whether the high-piling condition is violated for the traveling forklift TG1 only when the stop determination unit 104 determines that the forklift TG1 is traveling. With this configuration, the high-piling determination unit 110 can omit the determination processing when a determination is not necessary.

[0118] (4) The direction identification unit 107 identifies the direction of the forklift TG1 based on the video MV. Furthermore, the movement direction identification unit 108 identifies the traveling direction of the forklift TG1 based on the body position or feature points of the forklift TG1 shown in the video MV. Then, the movement direction identification unit 108 identifies whether the forklift TG1 is moving forward or backward based on the identified direction and traveling direction of the forklift TG1. When the movement direction identification unit 108 identifies that the forklift TG1 is moving forward, the high-stacking determination unit 110 performs processing to determine whether the high-stacking condition is violated using a forward-stacking threshold value. When the movement direction identification unit 108 identifies that the forklift TG1 is moving backward, the high-stacking determination unit 110 performs processing to determine whether the high-stacking condition is violated using a reverse-stacking threshold value. When the movement direction identification unit 108 has not identified the traveling direction of the forklift TG1 or the identified traveling direction changes as needed, the high-stacking determination unit 110 performs processing to determine whether the high-stacking condition is violated using a turning threshold value.

[0119] The forward threshold is smaller than the reverse threshold, for example. This is because the driver has higher visibility when the forklift TG1 is reversing than when the forklift TG1 is moving forward or turning, allowing for more stable driving. With this configuration, the high-stacking determination unit 110 can appropriately determine whether the high-stacking condition is violated depending on the behavior of the forklift TG1.

[0120] (5) For each image IMa constituting the violation video MV1, the position identifying unit 103 identifies a first position, which is the position of the balance weight of the forklift TG1 on the image IMa, and a second position, which is the position of the mast of the forklift TG1 on the image IMa. The orientation identifying unit 107 identifies the orientation of the forklift TG1 based on the positional relationship between the first position and the second position.

[0121] According to this configuration, the position identification unit 103 can identify a first position, which is the position of the balance weight of the forklift TG1, and a second position, which is the position of the mast, by a simple method using the trained model 151, while the orientation identification unit 107 can identify the orientation of the forklift TG1 based on the information obtained by the simple method.

[0122] (6) When it is determined that a violation event EV1 has occurred, the video generation unit 111 may generate a violation video MV1 by extracting a portion of the video MV that includes an image IMa of the scene where the violation event EV1 has occurred from the video MV. Furthermore, when it is determined that a compliance event EV2 has occurred, the video generation unit 111 may generate a compliance video MV2 by extracting a portion of the video MV that includes an image IMa of the scene where the compliance event EV2 has occurred from the video MV. With this configuration, the video generation unit 111 can generate a violation video MV1 that shows a scene where the high pile condition is violated. Furthermore, the video generation unit 111 can generate a compliance video MV2 that shows a scene where the high pile condition is complied with.

[0123] (7) The judgment result generating unit 112 generates judgment result information 155 indicating the judgment result corresponding to the violation video MV1. The judgment result generating unit 112 also generates judgment result information 155 indicating the judgment result corresponding to the compliance video MV2. With this configuration, by referencing the judgment result information 155, detailed information about the violation event EV1 indicated by the violation video MV1 can be ascertained for each event. With this configuration, by referencing the judgment result information 155, detailed information about the compliance event EV2 indicated by the compliance video MV2 can be ascertained for each event.

[0124] (8) The judgment result generating unit 112 identifies the forklift TG1 involved in the violation event EV1 based on the moving object identification information attached to the forklift TG1, and generates judgment result information 155 by further including the identified moving object identification information. Also, the judgment result generating unit 112 identifies the forklift TG1 involved in the compliance event EV2 based on the moving object identification information attached to the forklift TG1, and generates judgment result information 155 by further including the identified moving object identification information.

[0125] According to this configuration, the determination result generating unit 112 can refer to information about the forklift TG1 that violates the regulations by referring to the determination result information 155. Furthermore, the determination result generating unit 112 can refer to information about the forklift TG1 that complies with the regulations by referring to the determination result information 155.

[0126] (9) The determination result generating unit 112 generates determination result information 155 including imaging unit identification information that can identify the imaging unit 21. The presenting unit 113 generates a confirmation screen IMb based on the determination result information 155 and a map image IMd that indicates a predetermined position where the imaging unit 21 corresponding to the imaging unit identification information is provided and a travel route. With this configuration, the presenting unit 113 can intuitively and easily present the position where a violation of the high stacking condition has occurred or the position where the high stacking condition has been complied with.

[0127] (10) The video generation unit 111 stores the violation event EV1 in the violation event video folder 153 and the compliance event EV2 in the compliance event video folder 154 of the storage unit 150 that stores the judgment result information 155. This configuration allows easier access to the violation video MV1 and the compliance video MV2 than when the violation video MV1 and the compliance video MV2 are mixed in the same storage location.

[0128] The above-described embodiments may be modified as follows: The above-described embodiments and the following modifications may be combined with each other within the scope of technical compatibility. The information processing system 1 may include a plurality of management devices 20. In this case, the information processing device 10 executes the above-described process for each management device 20. Furthermore, the management device 20 may be connected to one imaging unit 21.

[0129] In the case where moving images MV are generated at intervals of several minutes to several tens of minutes as described above, when the moving image generating unit 111 generates a violation moving image MV1 or a compliance moving image MV2 from a part of the moving image MV, a violation event EV1 or a compliance event EV2 may occur in a gap in the moving image MV. In this case, the moving image generating unit 111 generates the violation moving image MV1 and the compliance moving image MV2 using the moving image MV used for the determination and a moving image MV that is chronologically consecutive to the moving image MV.

[0130] The information included in the determination result information 155 is merely an example and is not limited to this. The determination result information 155 may include information other than that described above, or may include some of the information described above.

[0131] A barcode is one example of a method for realizing the moving object identification information, and is not limited to this. The moving object identification information may be realized, for example, by RFID (Radio Frequency Identification) or a beacon. In this case, a receiving device for RFID or a beacon is provided near the image capture unit 21, and the receiving device acquires the moving object identification information by receiving the RFID or beacon attached to the forklift TG1 traveling near the image capture unit 21. The management device 20 transmits information to the server device 30 that associates the moving object identification information with the date and time when the moving object identification information was read. Based on this information, the determination result generation unit 112 identifies the moving object identification information of the forklift TG1 corresponding to the violation video MV1 or the forklift TG1 corresponding to the compliance video MV2.

[0132] Although the case where the management device 20 transmits the video MV to the server device 30 has been described, this is not limiting. The management device 20 may transmit the video MV to the information processing device 10. Furthermore, if the information processing device 10 has a storage unit that the server device 30 has, the information processing system 1 does not need to have the server device 30. The information processing device 10 stores the received video MV in the storage unit 150 or another storage unit that has the same function as the storage unit that the server device 30 has. Each unit included in the control unit 100 performs various processes based on the video MV received by the information processing device 10.

[0133] The video MV may be constantly transmitted by the management device 20 to the server device 30 and / or the information processing device 10. The processing of the flowchart in Fig. 8 may be executed at a frequency other than every time the management device 20 transmits a moving image MV to the server device 30 or the information processing device 10. The processing of the flowchart in Fig. 8 may be executed, for example, at predetermined time intervals (for example, once a day). In this case, the processing of the flowchart shown in Fig. 8 is repeatedly executed for each moving image MV newly acquired from the server device 30.

[0134] When the moving image MV includes a plurality of forklifts TG1 moving, the information processing device 10 executes the processes of steps S202 to S240 of the flowchart shown in FIG. 8 for each of the forklifts TG1.

[0135] Of the processes shown in FIG. 8, the processes of steps S214 to S218 and step S230 may be executed in any order after step S202. The pull-down list PD displayed on the confirmation screen IMb is an example and is not limited to this. For example, the pull-down list PD may be provided in a number corresponding to the number of selectable pieces of information included in the determination result information 155.

[0136] The confirmation screen IMb does not have to include the pull-down lists PD1 to PD2 and the slider bar SB. In this case, the presentation unit 113 omits the processes of steps S302 and S304 in the flowchart shown in FIG.

[0137] Of the processes shown in FIG. 9, the processes of steps S302 to S304, steps S306 to S308, and steps S310 to S312 may be executed in any order. The position identification unit 103 may identify the body position of the forklift TG1 using a method other than the bounding box BXa. The position identification unit 103 may also identify the position of the load using a method other than the bounding box BXb. For example, when the position identification unit 103 receives the image IMa, it may identify the position of the forklift TG1 or the load using a trained model that segments the entire area on the image IMa of the forklift TG1 or the load. In this case, the position identification unit 103 inputs each of the images IMa that make up the video MV into the trained model, and acquires, for each image IMa, information indicating the area of ​​the forklift TG1 or the load shown in the image IMa.

[0138] The position identification unit 103 may identify a first position, which is the position of the balance weight on the image IMa, and a second position, which is the position of the mast on the image IMa, by a method other than the bounding boxes BXc and BXd. For example, when the image IMa is input, the position identification unit 103 may identify the first position and the second position using a trained model that segments the entire area on the image IMa of the balance weight and the entire area on the image IMa of the mast. In this case, the position identification unit 103 inputs each of the images IMa that constitute the video MV into the trained model, and acquires information indicating the area on the image IMa of the balance weight and the area on the image IMa of the mast that is shown in each image IMa.

[0139] If the orientation of the forklift truck TG1 can be identified by defining a position behind the forklift truck TG1 as the first position and a position in front of the forklift truck TG1 as the second position, positions other than the balance weight and the mast may be used as feature points of the forklift truck TG1. In this case, the position identifying unit 103 identifies the positions of the feature points other than the balance weight and the mast.

[0140] The forklift TG1 may be provided with feature point identification information represented by a barcode or the like, for example, on a position on the top surface or the like that can be imaged by the imaging unit 21. The feature point identification information is, for example, information that enables identification of the front and rear of the forklift TG1. Furthermore, moving object identification information may be used as the feature point identification information. The angle determination unit 106 and the orientation identification unit 107 may identify the angle and orientation of the target forklift TG1 by performing general image processing on the image IMa to recognize the feature point identification information.

[0141] The high-stacking determination unit 110 may perform processing to determine whether or not the high-stacking condition is violated even when the stop determination unit 104 determines that the forklift TG1 is stopped. With this configuration, the high-stacking determination unit 110 can perform determination processing even for a forklift TG1 that is currently violating the high-stacking condition but is stopped and therefore safe, but which would violate the high-stacking condition if it started to move. In this case, the processing of step S210 in the flowchart shown in FIG. 8 is omitted. In this case, the control unit 100 does not need to be equipped with the stop determination unit 104.

[0142] The high-piling determination unit 110 may further determine the degree of violation of the high-piling condition in the violation event EV1 based on the magnitude of the ratio calculated by the ratio calculation unit 101. For example, the high-piling determination unit 110 may make a two-level determination, such as "bad" or "very bad," depending on the degree of violation of the high-piling condition. In this case, "bad" indicates that the high-piling condition is violated and the degree of violation is light, while "very bad" indicates that the high-piling condition is violated and the degree of violation is severe. For example, the high-piling determination unit 110 determines that the degree of violation is "very bad" as the value of the ratio calculated by the ratio calculation unit 101 increases.

[0143] The determination result generating unit 112 generates the determination result information 155 by further including information indicating the degree of violation in the determination result information 155. With this configuration, the determination result generating unit 112 can refer to the determination result information 155 to further refer to the degree of violation of the forklift TG1 that violated the high stacking condition.

[0144] The high-stacking determination unit 110 may further determine the degree of compliance with the high-stacking conditions in the compliance event EV2 based on the magnitude of the ratio calculated by the ratio calculation unit 101. For example, the high-stacking determination unit 110 may make a two-level determination, such as “good” or “excellent,” depending on the degree of compliance with the regulations. In this case, “good” indicates that the high-stacking conditions are complied with but to a low degree, while “excellent” indicates that the high-stacking conditions are complied with and to a high degree. For example, the high-stacking determination unit 110 determines that the degree of compliance is “excellent” as the value of the ratio calculated by the ratio calculation unit 101 decreases. The determination result generation unit 112 generates the determination result information 155 by, for example, further including information indicating the degree of compliance in the determination result information 155. With this configuration, the determination result generation unit 112 can further refer to the degree of compliance of the forklift TG1 that complies with the high-stacking conditions by referring to the determination result information 155.

[0145] The degree of violation may be expressed in three stages, four stages or more stages other than two stages. Furthermore, the high stacking determination unit 110 may determine the degree of compliance in multiple stages instead of (or in addition to) the degree of violation.

[0146] Instead of (or in addition to) changing the size of the symbol image SY depending on the number of times the violation event EV1 has occurred, the presentation unit 113 may flash the symbol image SY superimposed on the icon ICN of the imaging unit 21 that has captured an image of a regulated location where the number of violation events EV1 that have occurred is high.

[0147] The presentation unit 113 may change the display mode of the symbol image SY based on the number of times the compliance event EV2 has occurred instead of (or in addition to) the number of times the violation event EV1 has occurred. For example, the presentation unit 113 may enlarge, compared to a normal size, the symbol image SY to be superimposed on the icon ICN of the imaging unit 21 that has captured an image of a regulated location where a large number of compliance events EV2 have occurred during a period specified by the pull-down lists PD1 to PD2. Furthermore, the presentation unit 113 may blink the symbol image SY to be superimposed on the icon ICN of the imaging unit 21 that has captured an image of a regulated location where a large number of compliance events EV2 have occurred.

[0148] According to this configuration, the presentation unit 113 can present information indicating the number of violations or the degree of violation to the user viewing the confirmation screen IMb in an intuitive and easy-to-understand manner. The presentation unit 113 may change the display mode of the symbol image SY based on the degree of violation indicated in the determination result information 155. For example, on the confirmation screen IMb representing a record of the violation event EV1, the presentation unit 113 may increase the size of the symbol image SY superimposed on the icon ICN of the image capture unit 21 corresponding to the record as the information indicating the degree of violation of the record is "extremely bad" or the proportion of "extremely bad" is high. Furthermore, on the confirmation screen IMb representing a record of the violation event EV1, the presentation unit 113 may blink the symbol image SY superimposed on the icon ICN of the image capture unit 21 corresponding to the record as the information indicating the degree of violation of the record is "extremely bad" or the proportion of "extremely bad" is high.

[0149] Although the above description has been given of the case where the control unit 100 includes the presenting unit 113, the presenting unit 113 may not be included. In the above description, the control unit 100 includes the stop determination unit 104, the angle calculation unit 105, the angle determination unit 106, the orientation identification unit 107, the movement direction identification unit 108, and the behavior identification unit 109. However, the control unit 100 does not necessarily have to include these units. That is, the control unit 100 of the information processing device 10 may be configured with the position identification unit 103, the ratio calculation unit 101, and the high-piling determination unit 110. In this case, steps S206 to S210 and steps S214 to S230 are omitted from the processing shown in FIG. 8 .

[0150] In the above description, the control unit 100 is described as including the video generating unit 111, but this is not limiting. The control unit 100 does not have to include the video generating unit 111. In this case, the information processing device 10 does not generate the violation video MV1 and the compliance video MV2 based on the determination result of the high-piling determination unit 110. Furthermore, steps S234 to S240 of the process shown in FIG. 8 are omitted.

[0151] Although the above description has been given of the case where the information processing device 10 generates the determination result information 155 based on the determination result of the high-piling determination unit 110, this is not limitative. The information processing device 10 does not need to generate the determination result information 155. In this case, the control unit 100 does not include the determination result generation unit 112, and the process of step S242 in the process shown in FIG. 8 may be omitted.

[0152] Although the above description has been given of the case where the control unit 100 includes the presenting unit 113, this is not limiting. The control unit 100 does not have to include the presenting unit 113. In this case, the information processing device 10 does not provide the user of the management device 20 with a confirmation screen IMb based on the determination result information 155. Furthermore, the control unit 100 does not execute the process shown in FIG. 9.

[0153] When the information processing device 10 determines that a violation event EV1 has occurred, it may notify the forklift truck TG1 that the violation event EV1 has occurred. In this case, a notification unit is provided near the imaging unit 21. The notification unit may be, for example, a speaker that notifies by sound or a rotating light that notifies by light. The control unit 100 also includes a notification control unit. When the high stacking determination unit 110 determines that a violation event EV1 has occurred, the notification control unit identifies the imaging unit 21 near the location of the forklift truck TG1 that is the subject of the determination. Specifically, the notification control unit identifies the imaging unit 21 that generated the video MV that served as the source data for the determination process of the high stacking determination unit 110. The notification control unit notifies the forklift truck TG1 that the violation event EV1 has occurred via the notification unit corresponding to the identified imaging unit 21. With this configuration, the notification control unit notifies the forklift truck TG1 that is located near the imaging unit 21 that the violation event EV1 has occurred, thereby alerting the occupant or other people if the forklift truck TG1 is a vehicle.

[0154] The notification unit may be provided in the forklift TG1 instead of (or in addition to) being provided near the imaging unit 21. Specifically, in the forklift TG1, the notification unit may be provided in, for example, the driver's seat. In this case, when the high stacking determination unit 110 determines that a violation event EV1 has occurred, the notification control unit identifies the moving object identification information of the forklift TG1 being determined. The process of identifying the moving object identification information is similar to the process executed by the determination result generation unit 112 described above, and therefore, description thereof will be omitted. Based on the identified moving object identification information, the notification control unit notifies the forklift TG1 associated with the identified moving object identification information that the violation event EV1 has occurred. With this configuration, the notification control unit notifies the forklift TG1 that caused the violation event EV1 that the violation event EV1 has occurred, thereby alerting the occupant or other people, if the forklift TG1 is a vehicle.

[0155] When the notification control unit performs the notification, the video MV is constantly transmitted by the management device 20 to the server device 30 or the information processing device 10. The processing of the flowchart shown in FIG. 8 is executed immediately when the server device 30 or the information processing device 10 receives the video MV. The processing by the notification control unit is executed between step S232 and step S234 or S238 of the processing shown in FIG. 8. With this configuration, the notification control unit can perform the notification at a timing when the forklift truck TG1 that caused the violation event EV1 is closer to the imaging unit 21. When the video MV is constantly transmitted to the information processing device 10, the information processing device 10 includes a storage unit.

[0156] Instead of generating the violation video MV1 and the compliance video MV2, the determination result generating unit 112 may generate link information that associates the date and time when the violation event EV1 occurred with the playback time of the video MV at that date and time, and include the link information in the determination result information 155. With this configuration, the user of the management device 20 can easily check the video at the time when the violation event EV1 occurred by checking the determination result information 155.

[0157] The management device 20 may extract a portion of the video MV that shows the forklift TG1 and transmit it to the server device 30 or the information processing device 10. With this configuration, when the position identification unit 103 identifies the position of the forklift TG1 from the video MV, the number of images IMa that make up the video MV is reduced, thereby reducing the processing load.

[0158] The information processing device 10 may perform at least one of a process related to a violation of a high-pile condition and a process related to compliance with the high-pile condition. Although the case where the trained model 151 is stored in the storage unit 150 has been described, this is not limiting. The storage unit 150 may store trained models other than the trained model 151. For example, when the trained model 151 receives an image IMa, it may output information indicating the position of the main body of the forklift TG1 on the image IMa and information indicating the position of the load on the image IMa. When the image IMa is received, another trained model may output a first position that is the position of the balance weight of the forklift TG1 on the image IMa and a second position that is the position of the mast of the forklift TG1 on the image IMa.

[0159] The technical ideas that can be understood from the above-described embodiment and modified examples will be described. [Mode 1] An information processing method in which a computer determines the height of the load being carried by the forklift and the height of the forklift shown in a time-series video generated by capturing images of the forklift's travel path using an imaging unit installed at a predetermined position capable of capturing the image of the forklift's travel path, calculates the ratio between the height of the load and the height of the forklift using the determined height of the load and the height of the forklift, and if the calculated ratio is equal to or greater than a predetermined threshold, determines that a violation event that violates high stacking conditions has occurred.

[0160] [Aspect 2] An information processing method as described in [Aspect 1], in which the height of the load and the height of the forklift are identified based on the information output by inputting each of the images that make up the video into a trained model that outputs the position of the load and the position of the forklift, when the images that make up the video are input.

[0161] [Aspect 3] The trained model, when input with the images that make up the video, outputs bounding boxes indicating the position of the load and the position of the forklift, respectively, and by inputting the images that make up the video into the trained model, it outputs bounding boxes indicating the position of the load and the position of the forklift, in the information processing method described in [Aspect 2].

[0162] [Aspect 4] An information processing method described in any one of [Aspect 1] to [Aspect 3], which determines whether the forklift is moving based on the video, and only if it is determined that the forklift is moving, performs a process of determining whether a violation event that violates the high stacking condition has occurred based on the video.

[0163] [Aspect 5] The information processing method according to [Aspect 4], wherein if the forklift shown in the video continues to move for a predetermined period of time or more, it is determined that the forklift is traveling.

[0164] [Aspect 6] An information processing method described in any one of [Aspect 1] to [Aspect 5], wherein the direction of travel and orientation of the forklift shown in the video are identified based on the video, and whether the forklift is moving forward or backward is identified based on the identified direction of travel and orientation of the forklift, the predetermined thresholds include a forward threshold used when the forklift is moving forward and a reverse threshold used when the forklift is moving backward, the forward threshold being smaller than the reverse threshold, and if it is identified that the forklift is moving forward, the forward threshold is used to determine whether the high stacking condition has been violated, and if it is identified that the forklift is moving backward, the reverse threshold is used to determine whether the high stacking condition has been violated.

[0165] [Aspect 7] The predetermined threshold value includes a turning threshold value used when the forklift is turning, the turning threshold value being smaller than the reverse threshold value, and when the forklift is turning, the turning threshold value is used to perform a process to determine whether the high stacking condition is violated.This is an information processing method described in [Aspect 6].

[0166] [Aspect 8] An information processing method as described in [Aspect 6] or [Aspect 7], which identifies, for each image constituting the video, a first position, which is the position of the balance weight of the forklift on the image, and a second position, which is the position of the mast of the forklift on the image, and identifies the orientation of the forklift based on the positional relationship between the first position and the second position.

[0167] [Aspect 9] An information processing method as described in [Aspect 8], in which the first position and the second position are identified based on information output by inputting the images that make up the video into a trained model that outputs the first position and the second position when the images that make up the video are input.

[0168] [Aspect 10] An information processing method described in any one of [Aspect 1] to [Aspect 9], in which, when it is determined that the violation event has occurred, a violation video is generated by extracting a portion of the video that includes the image of the scene in which the violation event occurred.

[0169] [Aspect 11] An information processing method described in any one of [Aspect 1] to [Aspect 10], in which, when it is determined that the violation event has occurred, judgment result information indicating the judgment result corresponding to the violation event is generated.

[0170] [Aspect 12] An information processing method as described in [Aspect 11], which identifies the forklift related to the violation event based on identification information corresponding to the forklift, and generates the judgment result information by further including moving object identification information that can identify the identified forklift.

[0171] [Aspect 13] An information processing method described in [Aspect 11] or [Aspect 12], wherein the judgment result information includes imaging unit identification information that can identify the imaging unit, and a confirmation screen is generated that displays the occurrence status of the violation event for each imaging unit on the map image based on a map image showing the specified position where the imaging unit corresponding to the imaging unit identification information is installed and the movement route, and the judgment result information.

[0172] [Aspect 14] An information processing method described in any one of [Aspect 1] to [Aspect 13], which notifies the forklift that the violation event has occurred if it is determined that the violation event has occurred.

[0173] [Aspect 15] An information processing method described in any one of [Aspect 1] to [Aspect 14], in which, based on the calculated ratio, if the ratio is less than a predetermined threshold, it is determined that a compliance event in which the high stacking condition is complied with has occurred.

[0174] [Aspect 16] An information processing method described in [Aspect 15], in which, when it is determined that the compliance event has occurred, a compliance video is generated by extracting a portion of the video that includes the image of the scene where the compliance event occurred.

[0175] [Aspect 17] An information processing method according to [Aspect 16], which generates judgment result information indicating a judgment result corresponding to the compliance event. [Aspect 18] An information processing system comprising an imaging unit installed at a predetermined position capable of imaging the path of a forklift, one or more management devices that store a video of a chronological sequence of images generated by the imaging unit imaging the path of the forklift on a server device on the cloud, and an information processing device that performs various processes based on the video stored on the server device, wherein the information processing device comprises: a position identification unit that identifies the height of the load carried by the forklift and the height of the forklift based on the input video; a ratio calculation unit that calculates the ratio between the height of the load and the height of the forklift using the height of the load and the height of the forklift; and a high-stacking determination unit that determines that a violation event that violates high-stacking conditions has occurred if the ratio calculated by the ratio calculation unit is equal to or greater than a predetermined threshold.

[0176] [Aspect 19] An information processing system comprising an imaging unit installed at a predetermined position capable of imaging the path of a forklift, one or more management devices that store in an information processing device a video of a chronological sequence of images generated by the imaging unit imaging the path of the forklift, and an information processing device that performs various processes based on the video, wherein the information processing device comprises: a position identification unit that identifies the height of the load carried by the forklift and the height of the forklift based on the input video; a ratio calculation unit that calculates the ratio between the height of the load and the height of the forklift using the height of the load and the height of the forklift; and a high-stacking determination unit that determines that a violation event that violates high-stacking conditions has occurred if the ratio calculated by the ratio calculation unit is equal to or greater than a predetermined threshold.

[0177] [Aspect 19] When an image capturing unit capable of capturing an image of a forklift's travel path captures and generates an image of the travel path, the unit outputs the position of the load being loaded by the forklift and the position of the forklift. Trained model. [Explanation of symbols]

[0178] 1...information processing system, 10...information processing device, 20...management device, 21, 21a, 21b, 21c, 21d, 21e, 21f, 21g...imaging unit, 30...server device, 101...ratio calculation unit, 103...position identification unit, 110...height stacking judgment unit, 151...trained model, 155...judgment result information, BX...bounding box, BXa, BXb, BXc, BXd...bounding box, EV1...violation event, EV2...compliance event, IMa, IMa1, IMa2...image, IMb...confirmation screen, IMd...map image, MV...video, MV1...violation video, MV2...compliance video, TG1...forklift.

Claims

1. The computer The system identifies the height of the load being loaded by the forklift and the height of the forklift, based on a time-series video of images generated by an imaging unit installed at a predetermined position capable of imaging the forklift's travel path, and calculates the ratio between the height of the load and the height of the forklift using the identified height of the load and the height of the forklift, and if the calculated ratio is equal to or greater than a predetermined threshold, determines that a violation event that violates the high stacking condition has occurred. Information processing methods.

2. a trained model that outputs the position of the load and the position of the forklift when the images constituting the video are input, and then specifies the height of the load and the height of the forklift based on the information output by inputting the images constituting the video; The information processing method according to claim 1 .

3. The trained model, when inputting the images constituting the video, outputs bounding boxes indicating the position of the load and the position of the forklift, The images constituting the video are input to the trained model, and a bounding box indicating the position of the load and the position of the forklift is output. The information processing method according to claim 2 .

4. determining whether the forklift is traveling based on the video; Only when it is determined that the forklift is traveling, a process is performed to determine whether or not the violation event that violates the high stacking condition has occurred based on the video.

3. The information processing method according to claim 1.

5. If the forklift shown in the video continues to move for a predetermined period of time or more, it is determined that the forklift is traveling. The information processing method according to claim 4.

6. Identifying the direction of travel and orientation of the forklift shown in the video based on the video; Identifying whether the forklift is moving forward or backward based on the identified traveling direction and orientation of the forklift; The predetermined thresholds include a forward movement threshold used when the forklift is moving forward and a reverse movement threshold used when the forklift is moving backward, the forward threshold is smaller than the reverse threshold, When it is determined that the forklift is moving forward, a process is performed using the forward threshold value to determine whether or not the high stacking condition is violated, and when it is determined that the forklift is moving backward, a process is performed using the backward threshold value to determine whether or not the high stacking condition is violated.

3. The information processing method according to claim 1.

7. The predetermined threshold value includes a turning threshold value used when the forklift is turning, The turning threshold is smaller than the reverse threshold, When the forklift is turning, a process is performed using the turning threshold value to determine whether or not the high stacking condition is violated. The information processing method according to claim 6.

8. for each of the images constituting the video, a first position which is a position of a balance weight of the forklift on the image and a second position which is a position of a mast of the forklift on the image are identified, and the orientation of the forklift is identified based on a positional relationship between the first position and the second position. The information processing method according to claim 6.

9. Identifying the first position and the second position based on information output by inputting the images constituting the video into a trained model that outputs the first position and the second position when the images constituting the video are input. The information processing method according to claim 8.

10. generating a violation video by extracting a part of the video that includes the image of the scene where the violation event occurred, when it is determined that the violation event has occurred; 3. The information processing method according to claim 1.

11. generating determination result information indicating a determination result corresponding to the violation event when it is determined that the violation event has occurred; 3. The information processing method according to claim 1.

12. Identifying the forklift involved in the violation event based on identification information corresponding to the forklift; generating the determination result information by further including moving object identification information that can identify the specified forklift; The information processing method according to claim 11.

13. the determination result information includes imaging unit identification information that can identify the imaging unit, generating a confirmation screen that displays the occurrence status of the violation event for each imaging unit on the map image based on a map image that indicates the predetermined position where the imaging unit corresponding to the imaging unit identification information is provided and the movement route, and the determination result information; The information processing method according to claim 11.

14. If it is determined that the violation event has occurred, notify the forklift that the violation event has occurred.

3. The information processing method according to claim 1.

15. and determining, based on the calculated ratio, that a compliance event in which the high stacking condition is complied with has occurred if the ratio is less than a predetermined threshold value.

3. The information processing method according to claim 1.

16. generating a compliance video by extracting a part of the video including the image of the scene where the compliance event occurred, when it is determined that the compliance event has occurred; The information processing method according to claim 15.

17. generating judgment result information indicating a judgment result corresponding to the compliance event; The information processing method according to claim 16.

18. An information processing system comprising: an imaging unit provided at a predetermined position capable of capturing an image of a travel path of a forklift; one or more management devices that store, in a server device on a cloud, a video in which images generated by the imaging unit capturing the travel path are successively displayed in time series; and an information processing device that executes various processes based on the video stored in the server device, The information processing device includes: a position specifying unit that specifies the height of the load loaded on the forklift and the height of the forklift based on the input video; a ratio calculation unit that calculates a ratio between the height of the load and the height of the forklift using the height of the load and the height of the forklift; a high-piling determination unit that determines that a violation event that violates a high-piling condition has occurred when the ratio calculated by the ratio calculation unit is equal to or greater than a predetermined threshold value, Information processing system.

19. An information processing system comprising: an imaging unit provided at a predetermined position capable of capturing an image of a travel path of a forklift; one or more management devices that store, in an information processing device, a video in which images generated by the imaging unit capturing the travel path are successively displayed in time series; and an information processing device that executes various processes based on the video, The information processing device includes: a position specifying unit that specifies the height of the load loaded on the forklift and the height of the forklift based on the input video; a ratio calculation unit that calculates a ratio between the height of the load and the height of the forklift using the height of the load and the height of the forklift; a high-piling determination unit that determines that a violation event that violates a high-piling condition has occurred when the ratio calculated by the ratio calculation unit is equal to or greater than a predetermined threshold value, Information processing system.

20. an imaging unit capable of capturing an image of a movement path of a forklift, when an image generated by capturing the movement path is input, outputs a position of a load being loaded on the forklift and a position of the forklift; Trained model.

Images