Information processing method, information processing system, and trained model

Abstract

To provide an information processing method, an information processing system, and a trained model for determining whether a forklift is in proper compliance with regulations. [Solution] The computer identifies the detection range of the forklift TG1 shown in the image generated by the imaging unit 21 and the direction of travel and orientation of the forklift TG1 shown in the video, identifies the main body of the forklift TG1 excluding the forks, and based on the identified orientation and direction of travel of the forklift TG1, identifies whether the forklift TG1 is moving forward or backward, applies the forward detection range to the image identified as moving forward, identifies the behavior of the forklift TG1 shown in the video, and determines that a violation event has occurred if the forklift TG1 is not performing the specified behavior within the detection range or forward detection range.

Description

[Technical Field]

[0001] The present invention relates to an information processing method, an information processing system, and a trained model. [Background technology]

[0002] BACKGROUND ART Conventionally, there is known a technique for determining whether or not a vehicle has stopped appropriately at an intersection based on an image generated by an imaging unit capturing an image of the intersection (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-348469 Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, forklifts used within premises where loads are transported may be subject to regulations such as stop signs and slow-down orders based on signs posted along the route of travel within the premises. For example, when a forklift must stop at a stop line, the forklift is required to stop so that the tips of its forks are positioned at the stop line when it is moving forward if it is not carrying a load, and so that the tips of the load are positioned at the stop line when it is carrying a load. When an image of a forklift is captured, the appearance of the forks differs depending on whether the forklift is loaded or not. Specifically, when the forklift is loaded, the position of the tips of the load may be difficult to identify in the image. Therefore, the technology disclosed in Patent Document 1 may not be able to determine whether the forklift is properly complying with regulations. [Means for solving the problem]

[0005] An information processing method for achieving the above object involves a computer setting in advance a detection range corresponding to a restriction position of the forklift shown in an image generated by capturing an image of the movement path of the forklift using an imaging unit provided at a predetermined position capable of capturing the image of the movement path of the forklift, identifying the direction of travel and orientation of the forklift shown in the video based on a time-series video of the images, identifying the main body of the forklift excluding the forks shown in the images constituting the video, and identifying whether the forklift moving toward the restriction position is moving forward or backward based on the identified orientation and direction of travel of the forklift, and identifying that the forklift is moving forward. an information processing method for determining whether a violation event has occurred when the forklift is moving forward and whether a violation event has occurred when the forklift is not performing a predetermined behavior in the video from when at least a part of the main body enters the forward detection range to when it exits the forward detection range, when the forklift is moving forward and whether a violation event has occurred when the forklift is not performing a predetermined behavior in the video from when at least a part of the main body enters the forward detection range to when it exits the forward detection range.

[0006] With this configuration, it is possible to determine whether the forklift is properly complying with regulations. In the information processing method for achieving the above object, the predetermined behavior may include stopping the forklift or slowing down.

[0007] According to this configuration, it is possible to determine whether or not a predetermined behavior corresponding to stopping or slowing down has been performed. In an information processing method that achieves the above-mentioned object, a reference direction associated with the detection range and the forward detection range may be set in advance, and when the traveling direction entering the detection range matches the reference direction of the detection range, it may be determined whether or not the violation event has occurred, and when the traveling direction entering the forward detection range matches the reference direction, it may be determined whether or not the violation event has occurred.

[0008] According to this configuration, it is possible to determine whether a violation event has occurred only when a regulation is applied. In an information processing method that achieves the above-mentioned object, a plurality of detection ranges may be determined on the movement route, and a reference direction may be determined that corresponds to each of the plurality of detection ranges, and it may be determined whether or not a violation event has occurred for each of the plurality of detection ranges.

[0009] According to this configuration, it is possible to make determinations regarding a plurality of regulations based on an image generated by one imaging unit. In an information processing method for achieving the above-mentioned object, when the traveling direction and the reference direction match, if the time during which the speed of the forklift in the detection range or the forward detection range remains below a speed threshold is less than a predetermined time, it may be determined that the violation event has occurred.

[0010] In an information processing method for achieving the above-mentioned object, if the traveling direction and the reference direction match and the speed of the forklift does not reach a speed threshold in the detection range or the forward detection range, it may be determined that the forklift is not stopping or slowing down.

[0011] According to this configuration, it is possible to determine whether a moving object is not behaving in a prescribed manner based on regulations, based on simple indicators such as time and speed. In an information processing method for achieving the above-mentioned object, the reference direction is a direction connecting the detection range and an entry range into which the forklift enters after passing through the detection range, and when the identified position of the forklift indicates that it has moved from the detection range to the entry range, it may be determined that the direction of travel and the reference direction match.

[0012] According to this configuration, the reference direction can be determined without relying on user operation, and forklifts whose traveling direction matches the reference direction can be identified as those to which regulations apply.

[0013] In the information processing method for achieving the above object, the forward detection range may be a range obtained by moving the detection range by a predetermined distance in a direction opposite to the reference direction. According to this configuration, the forward detection range based on the detection range can be applied.

[0014] In an information processing method for achieving the above-mentioned object, for each of the images constituting the video, a first position, which is the position of the balance weight of the forklift on the image, and a second position, which is the position of the mast of the forklift on the image, may be identified, and the orientation of the forklift may be identified based on the positional relationship between the first position and the second position.

[0015] According to this configuration, the orientation of the forklift can be identified based on the characteristic points of the forklift. In an information processing method for achieving the above-mentioned object, the first position and the second position may be identified based on information output by inputting the images constituting the video into a trained model that outputs the first position and the second position when the images constituting the video are input.

[0016] With this configuration, the positions of the feature points of the forklift can be identified using a simple method that uses a trained model. In an information processing method that achieves the above-mentioned objective, when it is determined that the violation event has occurred, a violation video may be generated by extracting a portion of the video that includes the image of the scene where the violation event occurred.

[0017] According to this configuration, a violation video showing a scene in which a regulation is violated can be generated. In the information processing method for achieving the above object, when it is determined that the violation event has occurred, determination result information indicating a determination result corresponding to the violation event may be generated.

[0018] According to this configuration, by referring to the determination result information, detailed information about the violation event can be grasped for each event. In an information processing method for achieving the above-mentioned object, the forklift involved in the violation event may be identified based on identification information corresponding to the forklift, and the judgment result information may be generated by further including moving object identification information that can identify the identified forklift.

[0019] According to this configuration, by referring to the determination result information, it is possible to refer to information about a moving object that has violated a regulation. In an information processing method that achieves the above-mentioned object, the judgment result information may include imaging unit identification information that can identify the imaging unit, and a confirmation screen may be generated that displays the occurrence status of the violation event for each imaging unit on the map image based on the judgment result information and a map image that shows the predetermined position where the imaging unit corresponding to the imaging unit identification information is installed and the movement route.

[0020] According to this configuration, the location where a violation of a regulation occurred on the travel route can be presented in an intuitive and easy-to-understand manner. In the information processing method for achieving the above object, when it is determined that the violation event has occurred, the forklift may be notified that the violation event has occurred.

[0021] According to this configuration, it is possible to warn a moving object that is present within or near the detection range by notifying the moving object that a violation event has occurred. In an information processing method for achieving the above-mentioned object, it is also possible to determine that a compliance event has occurred if at least a part of the forklifts behaves in a predetermined manner within the detection range after entering the detection range based on the identified position of the forklifts, and to determine that the compliance event has occurred if at least a part of the forklifts behaves in a predetermined manner within the forward detection range after entering the forward detection range.

[0022] With this configuration, it is possible to determine whether a forklift to which regulations apply has performed a predetermined behavior pursuant to the regulations. In an information processing method that achieves the above-mentioned object, when it is determined that the compliance event has occurred, a compliance video may be generated by extracting a portion of the video that includes the image of the scene where the compliance event occurred.

[0023] With this configuration, for a forklift to which a regulation applies, it is possible to determine whether or not the regulation is complied with, and to generate a compliance video showing a scene in which the regulation is complied with. In the information processing method for achieving the above object, judgment result information indicating a judgment result corresponding to the compliance event may be generated.

[0024] According to this configuration, by referring to the determination result information, detailed information about the compliance event shown in the compliance video can be grasped for each event. In the information processing method for achieving the above object, the at least part of the forklift may be a front end of the forklift in the traveling direction on the image.

[0025] In an information processing method for achieving the above-mentioned object, when the images constituting the video are input, the position of the forklift is identified based on the information output by inputting each of the images constituting the video into a trained model that outputs the position of the forklift using a bounding box, and at least a portion of the forklift may be the lower front end in the direction of travel of the bounding box that represents the forklift on the image.

[0026] With this configuration, it is possible to determine whether the forklift has violated regulations while identifying the location of the forklift using a simple method that uses a trained model. In the information processing method for achieving the above object, the severity of the violation event may be further determined based on a movement pattern of the forklift in the detection range or the forward detection range.

[0027] With this configuration, it is possible to determine the degree of violation of the forklift regulations. An information processing system that achieves the above object includes an imaging unit provided at a predetermined position capable of imaging a path of travel of a forklift, one or more management devices that store a video in a server device on a cloud, the video being a chronological sequence of images generated by the imaging unit capturing the path of travel, and the information processing system that executes various processes based on the video stored on the server device, wherein the information processing system includes a traveling direction specifying unit that specifies the traveling direction of the forklift shown in the video based on the video, a position specifying unit that specifies the main body excluding the forks of the forklift shown in the images that constitute the video, an orientation specifying unit that specifies the orientation of the forklift based on the video, and a direction specifying unit that determines whether the forklift moving toward a regulating position is moving forward or backward based on the specified orientation of the forklift and the traveling direction of the forklift. an application unit that sets a forward detection range corresponding to the regulated position and different from the predetermined detection range for the images constituting the video in which the forklift is identified as moving forward; a behavior identification unit that identifies the behavior of the forklift shown in the video based on the video; and a determination unit that, when the forklift is moving backward, determines that a violation event has occurred if the forklift does not perform a predetermined behavior in the video from when at least a part of the main body enters the forward detection range to when it exits the forward detection range, and, when the forklift is moving forward, determines that the violation event has occurred if the forklift does not perform a predetermined behavior in the video from when at least a part of the main body enters the forward detection range to when it exits the forward detection range.

[0028] According to this configuration, the same effects as those of the above-described information processing method can be obtained. An information processing system that achieves the above object includes an imaging unit provided at a predetermined position capable of imaging a path of travel of a forklift, one or more management devices that store in the information processing device a video in which images of the path of travel are captured by the imaging unit and generated in chronological order, and an information processing device that executes various processes based on the video, wherein the information processing device includes a traveling direction specifying unit that specifies the traveling direction of the forklift shown in the video based on the video, a position specifying unit that specifies the main body, excluding the forks, of the forklift shown in the images that make up the video, an orientation specifying unit that specifies the orientation of the forklift based on the video, and a direction specifying unit that specifies whether the forklift moving toward a regulating position is moving forward or backward based on the specified orientation of the forklift and the traveling direction of the forklift. The video system includes a movement direction identification unit, an application unit that sets a forward detection range that corresponds to the regulated position and is different from the predetermined detection range for the images that constitute the video in which the forklift is identified as moving forward, a behavior identification unit that identifies the behavior of the forklift shown in the video based on the video, and a determination unit that, when the forklift is moving backward, determines that a violation event has occurred if the forklift does not perform a predetermined behavior in the video from when at least a part of the main body enters the detection range to when it exits the detection range, and, when the forklift is moving forward, determines that the violation event has occurred if the forklift does not perform a predetermined behavior in the video from when at least a part of the main body enters the forward detection range to when it exits the forward detection range.

[0029] According to this configuration, the same effects as those of the above-described information processing method can be obtained. The trained model that achieves the above objective is characterized in that, when an imaging unit capable of capturing an image of a forklift's travel path inputs an image generated by capturing the travel path, it outputs a first position, which is the position of the forklift's balance weight on the image, and a second position, which is the position of the forklift's mast on the image.

[0030] According to this configuration, it is possible to identify the characteristic points of the forklift based on the image. [Effects of the Invention]

[0031] According to the present invention, it is possible to determine whether a forklift is in proper compliance with regulations. [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of an information processing device. [Figure 3] FIG. 3 is a diagram used to explain the preset image. [Figure 4] FIG. 4 is a diagram used to explain the processing of the position specifying unit and the processing of the orientation specifying unit. [Figure 5] FIG. 5 is a diagram showing an example of the contents of the determination result information. [Figure 6] FIG. 6 is a diagram showing an example of the confirmation screen. [Figure 7] FIG. 7 is a flowchart illustrating an example of processing by the information processing device. [Figure 8] FIG. 8 is a flowchart illustrating an example of processing by the information processing device. [Figure 9] FIG. 9 is a flowchart illustrating an example of processing by the information processing device. [Figure 10] FIG. 10 is a diagram used to explain the approach range. [Figure 11] FIG. 11 is a diagram used to explain another example of the detection range. DETAILED DESCRIPTION OF THE INVENTION

[0033] <Embodiment> Hereinafter, a specific embodiment of the information processing system 1 will be described with reference to the drawings. [Overall configuration] 1, the information processing system 1 includes an information processing device 10, a management device 20, and a server device 30. The information processing device 10, the management device 20, and the server device 30 are connected to each other via a network NW so that they can send and receive information. The network NW includes, for example, the Internet, a wide area network (WAN), a local area network (LAN), a provider terminal, a wireless communication network, a wireless base station, a dedicated line, and the like. Note that the network NW may include a local network in part.

[0034] The management device 20 includes a plurality of imaging units 21. The management device 20 and the imaging units 21 are connected so as to be able to send and receive information. Hereinafter, it is assumed that the management device 20 and the imaging units 21 are directly connected. The imaging units 21 are provided at predetermined positions within the premises of a factory, warehouse, or the like where cargo is transported, where they can capture images of the travel path of the forklift TG1 moving within the premises. Specifically, the imaging units 21 are provided at predetermined positions within the travel path where they can capture images of stop positions LN defined on the travel path. Furthermore, the plurality of imaging units 21 are provided at different positions, and each captures an image of a different stop position LN.

[0035] The information processing system 1 executes the processes from step ST1 to step ST6. Each step ST will be explained below. In procedure ST1, the management device 20 transmits the video MV generated by the imaging unit 21 to the server device 30 via the network NW. The video MV is a video in which images IMa are generated in chronological order by the imaging unit 21 capturing images of the travel path. The video MV is generated, for example, every predetermined time, and is transmitted to the server device 30 by the management device 20 as soon as it is generated. The predetermined time is, for example, several minutes to several tens of minutes. The video MV also includes information indicating the date and time when the video MV was captured and generated, and imaging unit identification information that can identify the imaging unit 21 that captured and generated the video MV.

[0036] The server device 30 is realized by a cloud server including a non-transitory storage medium such as a hard disk drive (HDD), flash memory, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), or random access memory (RAM). The server device 30 stores the moving image MV received from the management device 20 via the network NW in a storage unit.

[0037] In step ST2, the information processing device 10 reads the moving image MV from the storage unit of the server device 30 via the network NW. In step ST3, the information processing device 10 presents a preset image IM1 that prompts the user of the management device 20 to determine a detection range AR, a reference direction DR, and a predetermined distance d based on the moving image MV acquired in step ST2 and the image IMa constituting the moving image MV. As shown in FIGS. 3 and 4 , the detection range AR is a range corresponding to a stop position LN defined on the travel path and is set to determine whether the forklift truck TG1 moving backward has stopped appropriately before the stop position LN. The reference direction DR is a direction associated with the detection range AR and indicates the direction of travel on the travel path that is subject to restriction, i.e., stopping at the stop position LN. The predetermined distance d is a distance used to set the forward detection range AR′. The forward detection range AR′ is a range set to determine whether the forklift truck TG1 moving forward has stopped appropriately before the stop position LN. Specifically, the forward detection range AR′ is a range obtained by moving the detection range AR by the predetermined distance d in the direction opposite to the reference direction DR. The stop position LN is an example of a restriction position.

[0038] As shown in FIG. 1 , in step ST4, the information processing device 10 acquires information indicating the detection range AR and information indicating the reference direction DR from the management device 20. In step ST5, the information processing device 10 determines, based on the acquired information, whether the forklift TG1 has stopped appropriately within the detection range AR or the forward detection range AR' defined on the travel path. If the information processing device 10 determines that the forklift TG1 has not stopped appropriately, it determines that a violation event EV1 has occurred and generates a violation video MV1 showing the scene where the violation event EV1 occurred. If the information processing device 10 determines that the forklift TG1 has stopped appropriately, it determines that a compliance event EV2 has occurred and generates a compliance video MV2 showing the scene where the compliance event EV2 occurred.

[0039] In step ST6, the information processing device 10 provides a screen prompting the user to check information related to the violation event EV1 and the compliance event EV2. Details of the various configurations and processes used by the information processing system 1 to realize the processes of steps ST1 to ST6 will be described below.

[0040] [Configuration of information processing device 10] 2, the information processing device 10 includes, for example, a communication unit 11, a control unit 100, and a storage unit 150. The communication unit 11 includes, for example, a communication circuit or a communication module, and transmits and receives various data to and from the management device 20, the server device 30, and other devices via the network NW. The communication unit 11 performs communication via wired or wireless communication. In the case of wireless communication, the communication mode of the communication unit 11 may be short-range wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0041] The control unit 100 is realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (not shown) that includes a non-transitory storage medium such as an HDD or flash memory provided in the storage unit 150.

[0042] The storage unit 150 may be realized by the various storage devices described above, or an EEPROM, a ROM, a RAM, etc. The storage unit 150 stores, for example, a trained model 151, a violation event video folder 153, a compliance event video folder 154, and judgment result information 155. These will be described in detail later.

[0043] The control unit 100 includes, for example, a determination unit 101, a position identification unit 102, a traveling direction identification unit 103, an orientation identification unit 104, a movement direction identification unit 105, an application unit 106, a behavior identification unit 107, a judgment unit 108, a video generation unit 109, a judgment result generation unit 110, and a presentation unit 111.

[0044] The determination unit 101 determines the detection range AR and the reference direction DR based on information indicating the detection range AR corresponding to the stop position LN captured by the imaging unit 21 and information indicating the reference direction DR. The position identification unit 102 identifies the main body, excluding the forks, of the forklift TG1 in the image IMa. The travel direction identification unit 103 identifies the travel direction of the forklift TG1 shown in the video MV based on the video MV. The orientation identification unit 104 identifies the orientation of the forklift TG1 based on the video MV. The movement direction identification unit 105 identifies whether the forklift TG1, moving toward the stop position LN, is moving forward or backward based on the identified orientation and travel direction of the forklift TG1.

[0045] The application unit 106 sets a forward detection range AR', different from the predetermined detection range AR, for an image IMa that is identified as showing the forklift truck TG1 moving forward, among the images IMa constituting the video MV. The behavior identification unit 107 identifies the behavior of the forklift truck TG1 based on the video MV. The determination unit 108 determines whether the forklift truck TG1 is properly stopped within the detection range AR or the forward detection range AR' set on the movement path. The video generation unit 109 generates a violation video MV1 including scenes in which the forklift truck TG1 is not properly stopped within the detection range AR or the forward detection range AR'. The video generation unit 109 also generates a compliance video MV2 including scenes in which the forklift truck TG1 is properly stopped within the detection range AR or the forward detection range AR'. The determination result generation unit 110 generates determination result information 155 including a determination result corresponding to the violation video MV1 and a determination result corresponding to the compliance video MV2. The presentation unit 111 generates an image used to present the determination result information 155 to the user, and presents the image to the user.

[0046] [Regarding the decision unit 101] The detailed processing of each unit will be described below. The determination unit 101 identifies a moving object-free image IMa1 that does not include the forklift TG1 and includes the stopping position LN from among the images IMa constituting the moving image MV read from the server device 30. This identification process may be performed by a user who has viewed the moving image MV, or by inputting the moving object-free image IMa1 that does not include the forklift TG1 in the moving image MV to the trained model 151 that outputs the moving object-free image IMa1 that does not include the forklift TG1 in the moving image MV when the moving object MV is input. Next, the determination unit 101 generates a preset image IM1 to be used in step ST3. The preset image IM1 is an image used as a screen that prompts the user of the management device 20 to determine the detection range AR, the reference direction DR, and the predetermined distance d. The determination unit 101 presents the generated preset image IM1 to the user of the management device 20. Specifically, the determination unit 101 makes the generated preset image IM1 available for reference by the management device 20 via the network NW.

[0047] 3, the preset image IM1 includes an area T1 indicating a moving object-free image IMa1. The user of the management device 20 uses the preset image IM1 displayed on the management device 20 to specify a detection range AR and a reference direction DR.

[0048] Specifically, the user of the management device 20 inputs a plurality of vertices P representing a detection range AR corresponding to the stop position LN shown in the moving object-free image IMa1 in the region T1. In this example, the detection range AR is represented by a rectangle having four vertices P, vertices P1 to P4. The user of the management device 20 also inputs a reference direction DR indicating the travel direction to which the stop position LN shown in the moving object-free image IMa1 applies. Specifically, the user of the management device 20 inputs a start point SP and an end point FP of the reference direction DR. In this example, the travel direction from the rear side to the front side of the stop position LN is represented as the reference direction DR. An operation region is provided in the region T1, extending to the periphery of the moving object-free image IMa1, and the vertices P, start point SP, and end point FP can be selected up to the edge of the moving object-free image IMa1. Specifically, a frame image IMw that is slightly larger than the moving object-free image IMa1 is displayed in the region T1, and the inside of the frame image IMw within the region T1 is set as the operation region.

[0049] Furthermore, the user of the management device 20 specifies a moving object TG to be judged by the information processing device 10. Option C1 for selecting a moving object TG indicates "all" which means that any moving object TG is to be judged, option C2 indicates "forklift", option C3 indicates "electric car", option C4 indicates "person", option C5 indicates "truck", option C6 indicates "bus", and option C7 indicates "car".

[0050] The options for the moving object TG are merely an example and are not limited to this. Furthermore, the options for the moving object TG may show some or all of the moving objects TG moving on the travel route. The process of specifying the moving object TG to be determined may be performed by using an image other than the preset image IM1. For example, a pop-up image displayed together with the preset image IM1 may have check boxes for selecting seven options C1 to C7 as options for the moving object TG to be determined. In this case, the user of the management device 20 specifies the moving object TG to be determined by operating the check boxes on the pop-up image. Hereinafter, it is assumed that the moving object TG to be determined is a forklift TG1.

[0051] In step ST4, the determination unit 101 acquires information indicating the coordinates of each vertex P of the detection range AR in the moving object-free image IMa1. The coordinates are, for example, in units of pixels, and are coordinates where the coordinates of the bottom left corner of the moving object-free image IMa1 are (0,0). The determination unit 101 determines, as the detection range AR, a pixel region within the range specified by the vertices P in the image IMa constituting the moving image MV. The determination unit 101 also acquires information indicating the coordinates of the start point SP and the end point FP of the reference direction DR in the moving object-free image IMa1. The determination unit 101 determines, as the reference direction DR, the direction specified by the start point SP and the end point FP in the image IMa constituting the moving image MV. The determination unit 101 determines, as the reference direction DR, the angle of the line segment connecting the start point SP and the end point FP in the screen coordinate system of the image IMa, which is the screen coordinate system of the imaging unit 21.

[0052] The determination unit 101 performs the same process as described above for each of the multiple imaging units 21 included in the management device 20, and determines a detection range AR and a reference direction DR corresponding to the stop position LN captured by the imaging unit 21. The determination unit 101 associates information indicating the pixel area of ​​the detection range AR and information indicating the angle of the reference direction DR with identification information of the imaging unit 21 corresponding to this information, and stores the information in the storage unit 150. Furthermore, in procedure ST4, the determination unit 101 acquires information indicating a moving object TG to be determined by the information processing device 10. Furthermore, in procedure ST4, the determination unit 101 acquires information indicating a predetermined distance d.

[0053] [Regarding the position identification unit 102] The position identification unit 102 identifies the position of the forklift TG1 shown in the image IMa constituting the video MV. The position identification unit 102 inputs the image IMa to the trained model 151. The trained model 151 is a trained model that, for example, when the image IMa is input, outputs information indicating the position of the forklift TG1 on the image IMa. In the following description, the position of the forklift TG1, the main body part of the forklift TG1 excluding the forks, will also be referred to as the main body position of the forklift TG1. The position identification unit 102 inputs each of the images IMa constituting the video MV to the trained model 151, and acquires, for each image IMa, information indicating the main body position of the forklift TG1 shown in the image IMa using coordinates. The position identification unit 102 also inputs each of the images IMa constituting the video MV to the trained model 151, and acquires, for each image IMa, information indicating the position of the balance weight of the forklift TG1 and information indicating the position of the mast of the forklift TG1 using coordinates.

[0054] FIG. 4 shows image IMa2, one of images IMa constituting the video MV, depicting a scene in which the forklift TG1 is moving from the rear to the front of the stopping position LN. For ease of explanation, image IMa2 shows a bounding box BXa indicating the main body position of the forklift TG1 identified by the position identification unit 102. As shown in FIG. 4, when image IMa is input, for example, the trained model 151 identifies the area on image IMa in which the forklift TG1 exists by the bounding box BXa. Therefore, the trained model 151 outputs the coordinates of the bounding box BXa as the partial area identifying the forklift TG1 as information indicating the main body position of the forklift TG1. The coordinates of the bounding box BXa are indicated, for example, by the coordinates of two diagonal points.

[0055] As described above, the bounding box BXa indicates the position of the main body of the forklift TG1 shown in the image IMa, excluding the forks of the forklift TG1, as the position of the forklift TG1.

[0056] As shown in FIG. 4, when image IMa is input, the trained model 151 identifies the area on image IMa where the balance weight of the forklift TG1 is located by using a bounding box BXc. Furthermore, when image IMa is input, the trained model 151 identifies the area on image IMa where the mast of the forklift TG1 is located by using a bounding box BXd. Therefore, the trained model 151 outputs the coordinates of the bounding box BXc as a partial area that identifies the balance weight of the forklift TG1 as information indicating the first position. Furthermore, the trained model 151 outputs the coordinates of the bounding box BXd as a partial area that identifies the mast of the forklift TG1 as information indicating the second position. The coordinates of the bounding boxes BXc and BXd are indicated, for example, by the coordinates of two diagonal points. Therefore, the bounding boxes BX identified by the processing of the position identifying unit 102 are the bounding box BXa, the bounding box BXc of the balance weight at the first position, and the bounding box BXd of the mast at the second position.

[0057] [Regarding the traveling direction identification unit 103] The traveling direction identification unit 103 identifies the traveling direction of the forklift TG1 on the image IMa based on changes over time in the coordinates of the bounding box BXa indicating the forklift TG1 identified by the trained model 151. Note that the traveling direction of the forklift TG1 may also be identified based on changes over time in the coordinates of the bounding box BXc, which is information indicating the first position identified by the trained model 151, or the coordinates of the bounding box BXd, which is information indicating the second position.

[0058] [Regarding the orientation identification unit 104] The orientation direction identification unit 104 identifies the midpoint at the bottom end of the bounding box BXc in the area on the image IMa identified as the balance weight, based on the coordinates of the bounding box BXc output from the trained model 151. Furthermore, the orientation identification unit 104 identifies the midpoint at the bottom end of the bounding box BXd in the area on the image IMa identified as the mast, based on the coordinates of the bounding box BXd output from the trained model 151.

[0059] Here, the screen coordinate system, which is the coordinate system when image IMa is displayed on the screen, is a coordinate system based on the upper left corner of the screen, with the left-right direction of the screen defined as the X-axis direction and the up-down direction of the screen defined as the Y-axis direction. The orientation identification unit 104 identifies the orientation of the forklift TG1, which indicates the direction from the midpoint of the bottom end of the bounding box BXc to the midpoint of the bottom end of the bounding box BXd, within the X-axis direction in the screen coordinate system of image IMa. Furthermore, the direction from the midpoint of the bottom end of the bounding box BXc to the midpoint of the bottom end of the bounding box BXd is the direction from the balance weight to the mast.

[0060] [Regarding the moving direction identification unit 105] The movement direction identification unit 105 identifies whether the forklift TG1 is moving forward or backward based on the identified traveling direction and orientation of the forklift TG1. The movement direction identification unit 105 determines whether the identified traveling direction and orientation match. Specifically, the movement direction identification unit 105 determines whether the angle between the identified traveling direction and orientation of the forklift TG1 identified by the orientation identification unit 104 is within a predetermined angle. If the angle between the identified traveling direction and orientation of the forklift TG1 is within the predetermined angle, it is determined that the identified traveling direction and orientation of the forklift TG1 match, that is, the forklift TG1 is moving forward. If the angle between the identified traveling direction and orientation of the forklift TG1 is greater than the predetermined angle, it is determined that the identified traveling direction and orientation of the forklift TG1 do not match, that is, the forklift TG1 is moving backward. The predetermined angle is, for example, 50 degrees.

[0061] For example, if the direction from the midpoint at the bottom end of the balance weight bounding box BXc to the midpoint at the bottom end of the mast bounding box BXd is rightward, the direction of the forklift TG1 identified by the direction identification unit 104 is determined to be rightward. If the traveling direction of the forklift TG1 identified by the movement direction identification unit 105 is rightward, the forklift TG1 is determined to be moving forward. Also, if the direction from the midpoint at the bottom end of the balance weight bounding box BXc to the midpoint at the bottom end of the mast bounding box BXd is leftward, the direction of the forklift TG1 identified by the direction identification unit 104 is determined to be leftward. If the traveling direction of the forklift TG1 identified by the movement direction identification unit 105 is rightward, the forklift TG1 is determined to be moving backward.

[0062] As described above, Fig. 4 shows image IMa2 of a scene in which the forklift truck TG1 is moving from the rear side to the front side of the stopping position LN. In the example shown in Fig. 4, the movement direction identification unit 105 identifies that the forklift truck TG1 is moving forward.

[0063] [Regarding application part 106] The application unit 106 sets a forward detection range AR', which is different from the predetermined detection range AR, for an image IMa that is identified as the forklift truck TG1 moving forward among the images IMa constituting the moving image MV. As described above, in the scene of image IMa2 shown in FIG. 4, the movement direction identification unit 105 identifies the forklift truck TG1 as moving forward. Therefore, in the example shown in FIG. 4, the application unit 106 sets a forward detection range AR' instead of the detection range AR. As shown in FIG. 4, the forward detection range AR' is a range obtained by moving the detection range AR a predetermined distance d in the direction opposite to the reference direction DR. The predetermined distance d is, for example, a distance of several tens of centimeters to several meters, which corresponds to the length of the forks of the forklift truck TG1. On the other hand, when it is identified that the forklift truck TG1 is moving backward, the application unit 106 applies the detection range AR as is.

[0064] [About the behavior identification unit 107] The behavior identification unit 107 identifies the behavior of the forklift TG1 shown in the video MV based on the video MV. The behavior identification unit 107 identifies the behavior of the forklift TG1 based on, for example, changes over time in the position of the forklift TG1 identified by the position identification unit 102. Hereinafter, the behavior of the forklift TG1 is assumed to include the speed, maximum speed, minimum speed, acceleration, and the duration of time during which the speed of the forklift TG1 remains equal to or less than the speed threshold value of the forklift TG1. Note that the behavior of the forklift TG1 is merely an example and is not limited to this, and may include some of these indicators or other indicators.

[0065] The behavior identification unit 107 identifies the speed of the forklift truck TG1 based on the video MV. First, the behavior identification unit 107 identifies the time required for at least a portion of the main body of the forklift truck TG1 to enter and exit the detection range AR or the forward detection range AR' based on the position of the forklift truck TG1 identified by the position identification unit 102 and the detection range AR or the forward detection range AR' applied by the application unit 106. The following describes a case where the forklift truck TG1 is moving forward facing right. In this case, "at least a portion of the main body of the forklift truck TG1" refers to the position of the front end lower portion JP in the traveling direction of the bounding box BXa representing the forklift truck TG1 on the image IMa. In this case, the application unit 106 sets the forward detection range AR' instead of the detection range AR.

[0066] If the front lower end JP is present within the forward detection range AR' in the image IMa, the behavior identification unit 107 determines that the forklift truck TG1 has entered the forward detection range AR'. Thereafter, if the front lower end JP is no longer present within the forward detection range AR' in the image IMa, the behavior identification unit 107 determines that the forklift truck TG1 has exited the forward detection range AR'. More specifically, the behavior identification unit 107, for example, determines the period during which the forklift truck TG1 was present within the forward detection range AR' from when the forklift truck TG1 entered the forward detection range AR' to when the forklift truck TG1 was present within the forward detection range AR', based on the time that has elapsed since the image IMa was captured, in which it was determined that the front lower end JP of the bounding box BXa had entered the forward detection range AR' as the forklift truck TG1 approached the stop position LN, until the image IMa was captured, in which it was determined that the front lower end JP of the bounding box BXa had exited the forward detection range AR'.

[0067] The behavior identification unit 107 also performs general image processing on the moving image MV for the identified period to identify the speed, maximum speed, minimum speed, acceleration, etc. of the forklift truck TG1 within the forward detection range AR'.

[0068] Furthermore, the behavior identification unit 107 determines, based on the video MV, the duration during which the speed of the forklift TG1 within the forward detection range AR' remains equal to or less than the speed threshold. The speed threshold is a speed that corresponds to the regulation at the stop position LN. As described above, in this embodiment, the forklift TG1 is required to stop at the stop position LN. Therefore, in this embodiment, the speed threshold is assumed to be 0 km / h. In other words, the behavior identification unit 107 determines, based on the video MV, the duration during which the forklift TG1 remains stopped within the forward detection range AR'. The behavior identification unit 107 determines the duration during which the forklift TG1 remains within the forward detection range AR', for example, by performing a process similar to the process described above. The behavior identification unit 107 performs general image processing on the video MV for the determined duration to identify the speed of the forklift TG1 and determine the duration during which the speed remains equal to or less than the speed threshold.

[0069] When the forklift truck TG1 is moving backward, the application unit 106 applies the detection range AR. Therefore, the behavior identification unit 107 identifies the speed, maximum speed, minimum speed, acceleration, etc. of the forklift truck TG1 within the detection range AR. The process by which the behavior identification unit 107 identifies various behaviors is the same as in the case of the forward detection range AR' described above, and therefore a description thereof will be omitted.

[0070] [Regarding the determination unit 108] The determination unit 108 determines whether the forklift TG1 is performing a predetermined behavior within the detection range AR or the forward detection range AR'. As described above, in this embodiment, the forklift TG1 is required to stop at the stop position LN. Therefore, the determination unit 108 determines whether the forklift TG1 is appropriately temporarily stopping within the detection range AR or the forward detection range AR', as a predetermined behavior. Specifically, when the traveling direction of the forklift TG1 identified by the traveling direction identification unit 103 when the forklift TG1 enters the detection range AR or the forward detection range AR' matches the reference direction DR associated with the detection range AR or the forward detection range AR', the determination unit 108 determines that the forklift TG1 is the target forklift TG1.

[0071] The reference direction DR is indicated by a line segment connecting the start point SP and the end point FP (i.e., the reference direction DR), and has a predetermined angular range. For example, the reference direction DR is a bisector of an angle within the predetermined angular range. Therefore, if the traveling direction of the forklift TG1 identified by the traveling direction identification unit 103 when the forklift TG1 enters the detection range AR or the forward detection range AR' is within a predetermined angular range centered on the reference direction DR, the determination unit 108 determines that the traveling direction of the forklift TG1 identified by the traveling direction identification unit 103 matches the reference direction DR, and determines that the forklift TG1 is the forklift TG1 being determined.

[0072] Then, based on the identification result of the behavior identification unit 107, the determination unit 108 determines that the forklift TG1 has performed a predetermined behavior, that is, has not made a temporary stop, and that a violation event EV1 has occurred, if the time during which the speed of the forklift TG1 to be determined within the detection range AR or the forward detection range AR' has remained equal to or less than the speed threshold is less than a predetermined time.Furthermore, based on the identification result of the behavior identification unit 107, the determination unit 108 determines that the forklift TG1 has performed a predetermined behavior, that is, has made a temporary stop, and that a compliance event EV2 has occurred, if the time during which the speed of the forklift TG1 to be determined within the detection range AR or the forward detection range AR' has remained equal to or less than the speed threshold is greater than a predetermined time.

[0073] Note that the determination unit 108 may determine that the forklift TG1 has not performed a temporary stop, which is a predetermined behavior, and that a violation event EV1 has occurred, if the speed of the forklift TG1 to be determined does not become equal to or less than the speed threshold within the detection range AR or the forward detection range AR', based on the identification result of the behavior identification unit 107. Similarly, the determination unit 108 may determine that the forklift TG1 has performed a temporary stop, which is a predetermined behavior, and that a compliance event EV2 has occurred, if the speed of the forklift TG1 to be determined becomes equal to or less than the speed threshold within the detection range AR or the forward detection range AR', based on the identification result of the behavior identification unit 107.

[0074] [About the video generation unit 109] When the determination unit 108 determines that a violation event EV1 has occurred, the video generation unit 109 generates a violation video MV1 by extracting a portion of the video MV that includes an image IMa of the scene where the violation event EV1 has occurred from the video MV. Also, when the determination unit 108 determines that a compliance event EV2 has occurred, the video generation unit 109 generates a compliance video MV2 by extracting a portion of the video MV that includes an image IMa of the scene where the compliance event EV2 has occurred from the video MV.

[0075] Here, the image IMa of the scene where the violation event EV1 occurred is, for example, the image IMa of the image MV at the timing when the determination unit 108 determines that the violation event EV1 has occurred. That is, in the case of reverse travel, the image IMa is the image IMa at the timing when the forklift TG1 does not appropriately stop within the detection range AR and exits the detection range AR; in the case of forward travel, the image IMa is the image IMa at the timing when the forklift TG1 does not appropriately stop within the forward travel detection range AR' and exits the forward travel detection range AR'. Note that the timing when the forklift TG1 does not appropriately stop within the detection range AR or the forward travel detection range AR' and exits is merely an example and is not limited to this. The image IMa of the scene where the violation event EV1 occurred may be, for example, the timing when the forklift TG1 does not appropriately stop within the detection range AR and remains within the detection range AR, or the timing when the forklift TG1 does not appropriately stop within the forward travel detection range AR' and remains within the forward travel detection range AR'.

[0076] Furthermore, image IMa of the screen on which compliance event EV2 has occurred is, for example, image IMa from among images IMa constituting the moving image MV at the timing at which it is determined by the determination unit 108 that compliance event EV2 has occurred. That is, in the case of reverse travel, image IMa is image IMa at the timing when the forklift TG1 is within the detection range AR as a result of appropriately stopping within the detection range AR, and in the case of forward travel, image IMa is image IMa at the timing when the forklift TG1 is within the forward detection range AR' as a result of appropriately stopping within the forward detection range AR'.

[0077] Furthermore, the video generation unit 109 identifies the position of the forklift TG1 shown in the violation video MV1 based on the coordinates of the bounding boxes BXa, BXc, and BXd acquired by the position identification unit 102 for each image IMa constituting the violation video MV1. The video generation unit 109 then generates, as the violation video MV1, a video in which a line indicating the detection range AR or the forward detection range AR' and a bounding box BX indicating the identified position of the forklift TG1 are superimposed on each image IMa constituting the violation video MV1. That is, the violation video MV1 shows the forklift TG1 and the bounding box BX indicating the position of the forklift TG1 when the violation event EV1 occurred. The video generation unit 109 also performs similar processing on the compliance video MV2. That is, the compliance video MV2 shows the forklift TG1 and the bounding box BX indicating the position of the forklift TG1 when the compliance event EV2 occurred.

[0078] Hereinafter, the video generation unit 109 will generate as violation video MV1 a portion of the video MV from when the backward forklift TG1, for which it is determined that the violation event EV1 has occurred, enters the detection range AR until it exits the detection range AR. Also, the video generation unit 109 will generate as violation video MV1 a portion of the video MV from when the forward forklift TG1, for which it is determined that the violation event EV1 has occurred, enters the forward detection range AR' until it exits the forward detection range AR'.

[0079] The video generation unit 109 generates, as the compliance video MV2, a portion of the video MV from when the backward forklift TG1, for which it is determined that the compliance event EV2 has occurred, enters the detection range AR to when it exits the detection range AR. The video generation unit 109 generates, as the compliance video MV2, a portion of the video MV from when the forward forklift TG1, for which it is determined that the compliance event EV2 has occurred, enters the forward detection range AR' to when it exits the forward detection range AR'.

[0080] The video generation unit 109 stores the generated violation video MV1 in a violation event video folder 153 of the storage unit 150, while storing the generated compliance video MV2 in a compliance event video folder 154 of the storage unit 150, within the storage area of ​​the storage unit 150. The violation event video folder 153 is an example of a first storage location, and the compliance event video folder 154 is an example of a second storage location.

[0081] [Regarding the judgment result generation unit 110 and the judgment result information 155] The judgment result generating unit 110 generates / updates judgment result information 155 indicating the judgment result corresponding to the violation video MV1 and the judgment result corresponding to the compliance video MV2.

[0082] 5, the determination result information 155 includes one or more records in which the following information is associated with each other: the date and time when the violation event EV1 or the compliance event EV2 occurred, the playback time of the violation video MV1 or the compliance video MV2 corresponding to that time, the event number, image capture unit identification information that can identify the image capture unit 21, the name of the forklift TG1, moving object identification information that can identify the forklift TG1, the determination type, the determination result, the determination reason, the entry time of the forklift TG1 into the detection range AR or the entry time of the forklift TG1 into the forward detection range AR', the exit time of the forklift TG1 into the detection range AR or the exit time of the forklift TG1 into the forward detection range AR', the average speed of the forklift TG1, the maximum speed of the forklift TG1, the minimum speed of the forklift TG1, the duration for which the forklift TG1 continued to be at or below the speed threshold, and the speed threshold. The determination result information 155 is realized, for example, by a data file in CSV (Comma Separated Values) format.

[0083] Each time the judgment unit 108 judges that a violation event EV1 has occurred or that a compliance event EV2 has occurred, a record is generated and the judgment result information 155 is generated / updated by adding the generated record to the judgment result information 155. In other words, each time the video generation unit 109 generates a violation video MV1 or a compliance video MV2, a record is generated and the judgment result information 155 is generated / updated.

[0084] The determination result generation unit 110 identifies the date and time when the violation event EV1 or the compliance event EV2 occurred for the moving image MV that is the subject of determination by the determination unit 108, based on information indicating the date and time included in the moving image MV. Specifically, the determination result generation unit 110 identifies the date and time a predetermined time before the forklift TG1, for which the determination unit 108 has determined that the violation event EV1 or the compliance event EV2 has occurred, entered the detection range AR or the forward detection range AR'. The predetermined time is, for example, several seconds (specifically, approximately 3 seconds). The determination result generation unit 110 also identifies the playback duration of the moving image MV on the date and time when the violation event EV1 or the compliance event EV2 occurred.

[0085] Furthermore, the judgment result generating unit 110 identifies the event number assigned to the violation event EV1 or the compliance event EV2. The event number is, for example, a serial number that is common to the violation event EV1 and the compliance event EV2. Therefore, the judgment result generating unit 110 identifies the number next to the event number of the violation event EV1 or compliance event EV2 that occurred immediately before as the event number of the violation event EV1 or compliance event EV2 that occurred next. Note that the event number may be a serial number for each violation event EV1 and each compliance event EV2.

[0086] The determination result generating unit 110 identifies the imaging unit identification information included in the moving image MV for the moving image MV that is the subject of determination by the determining unit 108. In this example, the determination result generating unit 110 identifies all moving object names as a forklift TG1.

[0087] The judgment result generating unit 110 identifies the moving object identification information based on the corresponding violation video MV1 or compliance video MV2. Here, the forklift TG1 is each assigned with the moving object identification information. Specifically, a vehicle such as the forklift TG1 is assigned with the moving object identification information indicated by a barcode or the like in a position such as the top surface that can be imaged by the imaging unit 21. The judgment result generating unit 110 identifies the moving object identification information of the forklift TG1 being judged by performing general image processing on the violation video MV1 or compliance video MV2.

[0088] The judgment result generation unit 110 identifies the judgment type based on the processing performed by the judgment unit 108. In this example, the judgment unit 108 makes a judgment regarding a temporary stop, and therefore the judgment type is all "temporary stop." The judgment result generation unit 110 identifies the judgment result based on the processing performed by the judgment unit 108. In this example, if the judgment unit 108 determines that a violation event EV1 has occurred, the judgment result is "NG," and if the judgment unit 108 determines that a compliance event EV2 has occurred, the judgment result is "OK." The judgment result generation unit 110 identifies the judgment reason based on the processing performed by the judgment unit 108. In this example, if the judgment unit 108 determines that a violation event EV1 has occurred, the judgment reason is "no stop," and if the judgment unit 108 determines that a compliance event EV2 has occurred, the judgment result is "stop."

[0089] The determination result generating unit 110 identifies the entry time based on processing performed by the behavior identifying unit 107. As the forklift TG1 approaches the stop position LN in reverse, the behavior identifying unit 107 identifies the time that has elapsed since the image IMa was captured, in which it was determined that the front end lower JP of the bounding box BX has entered the detection range AR, to the image IMa in which it was determined that the front end lower JP of the bounding box BX has left the detection range AR, as the time that the front end lower JP has been present within the detection range AR since it entered the detection range AR. In addition, the behavior identification unit 107 identifies the time that has elapsed from when an image IMa is captured in which it is determined that the front lower end JP of the bounding box BX has entered the forward detection range AR' as the forklift TG1 approaches the stop position LN in a forward movement to when an image IMa is captured in which it is determined that the front lower end JP of the bounding box BX has left the forward detection range AR' as the time that the front lower end JP has been within the forward detection range AR' since it entered the forward detection range AR'.

[0090] Therefore, the determination result generating unit 110 specifies the time when the front lower end JP of the bounding box BX of the forklift truck TG1 being determined, moving backward, enters the detection range AR as the entry time, and specifies the time when the front lower end JP of the bounding box BX of the forklift truck TG1 being determined, leaving the detection range AR, as the exit time. Also, the determination result generating unit 110 specifies the time when the front lower end JP of the bounding box BX of the forklift truck TG1 being determined, moving forward, enters the forward detection range AR', and specifies the time when the front lower end JP of the bounding box BX of the forklift truck TG1 being determined, leaving the forward detection range AR', as the entry time.

[0091] The determination result generating unit 110 may specify, as the entry time, the playback time of the moving image MV at the entry time or the frame number of an image IMa that constitutes the moving image MV and indicates the timing of the entry time. The determination result generating unit 110 may specify, as the exit time, the playback time of the moving image MV at the exit time or the frame number of an image IMa that constitutes the moving image MV and indicates the timing of the exit time.

[0092] The determination result generating unit 110 identifies the average speed of the forklift TG1 based on the processing performed by the behavior identifying unit 107. The determination result generating unit 110 also identifies the maximum speed of the forklift TG1 based on the processing performed by the behavior identifying unit 107. The determination result generating unit 110 also identifies the minimum speed of the forklift TG1 based on the processing performed by the behavior identifying unit 107. The determination result generating unit 110 also identifies the time during which the speed of the forklift TG1 in the detection range AR or the forward detection range AR' continues to be equal to or less than the speed threshold value based on the processing performed by the behavior identifying unit 107.

[0093] The judgment result generating unit 110 generates / updates the judgment result information 155 by generating a record that associates the various pieces of information identified by the above-mentioned processing with each other for each corresponding violation video MV1 and compliance video MV2.

[0094] Furthermore, the video generation unit 109 may determine the title of the violation video MV1 corresponding to the violation event EV1 and the title of the compliance video MV2 corresponding to the compliance event EV2 based on the judgment result information 155 generated / updated by the judgment result generation unit 110. The video generation unit 109 generates the violation video MV1 and the compliance video MV2 by giving them a title such as "(event number)_(judgment name)_(moving object name)_(judgment result).mp4", for example.

[0095] [About presentation unit 111] The presentation unit 111 generates a confirmation screen IMb so as to present the determination result information 155 to the user in an easy-to-understand manner. The confirmation screen IMb is a screen that shows the occurrence status of the violation event EV1 for each imaging unit 21. The presentation unit 111 presents the generated confirmation screen IMb to the user of the management device 20. Specifically, the presentation unit 111 makes the generated confirmation screen IMb available for reference by the management device 20 via the network NW.

[0096] As shown in FIG. 6, the confirmation screen IMb includes a designation image IMc, a map image IMd, and a slider bar image IMe. The designation image IMc includes, for example, pull-down lists PD1 to PD4. The pull-down list PD1 is a pull-down list used to designate the start of a period of time desired to be confirmed among the records included in the determination result information 155. The pull-down list PD2 is a pull-down list used to designate the end of a period of time desired to be confirmed among the records included in the determination result information 155. The pull-down list PD3 is a pull-down list used to select a determination type of time desired to be confirmed among the records included in the determination result information 155. The pull-down list PD4 is a pull-down list used to select an imaging unit 21 of time desired to be confirmed among the records included in the determination result information 155.

[0097] The map image IMd is, for example, an image that schematically shows a premises or the like in which the imaging unit 21 is installed. Furthermore, icons ICN are shown at positions on the map image IMd that correspond to positions where the imaging unit 21 is installed on the travel route of the schematically shown premises or the like. In this example, seven imaging units 21, namely imaging units 21a to 21g, are installed in the premises or the like. Therefore, seven icons ICNa to ICNg are shown at corresponding positions on the map image IMd. The map image IMd is generated in advance for each premises to be determined and stored in the storage unit 150. Furthermore, of the icons ICN that represent the imaging units 21, a symbol image SY is shown superimposed on a specific icon ICN. The symbol image SY is, for example, an image that represents the occurrence status of a violation event EV1. Details of the specific icon ICN on which the symbol image SY is superimposed will be described later.

[0098] The slider bar image IMe includes, for example, a slider bar SB, a start point slider CS1, and an end point slider CS2. The slider bar SB represents the entire period specified by the pull-down lists PD1 and PD2. In this example, since a single day is specified by the pull-down lists PD1 and PD2, the slider bar SB represents the period from midnight to midnight. On the other hand, if multiple days are specified by the pull-down lists PD1 and PD2, the slider bar SB represents the multiple days. If several hours are specified by the pull-down lists PD1 and PD2, the slider bar SB represents the several hours. The start point slider CS1 and the end point slider CS2 are used to specify a specific period to be checked within the period indicated by the slider bar SB. In the initial state, the start point slider CS1 and the end point slider CS2 are located at both ends of the slider bar SB so as to specify the entire period of the pull-down lists PD1 and PD2.

[0099] The user of the management device 20 operates the pull-down lists PD1 to PD4 and the start point slider CS1 and end point slider CS2 of the slider bar SB based on the confirmation screen IMb presented by the presentation unit 111. The presentation unit 111 acquires information indicating the operations by the user of the management device 20.

[0100] The presentation unit 111 identifies a record to be displayed as a map image IMd from records included in the determination result information 155, based on information specified using the pull-down list PD and the slider bar SB on the confirmation screen IMb. First, the presentation unit 111 searches the determination result information 155 from among the records included in the determination result information 155, using a date within the period specified by the pull-down lists PD1 to PD2 as a search key. The presentation unit 111 identifies and narrows down records associated with dates matching the specified date. Next, the presentation unit 111 searches for records in the identified determination result information 155, using a date and time within the period specified using the start point slider CS1 and the end point slider CS2 as a search key. The presentation unit 111 identifies and narrows down records matching the date and time. Furthermore, the presentation unit 111 searches for records in the identified determination result information 155, using a record of the determination type specified by the pull-down list PD3 and the target imaging unit 21 specified by the pull-down list PD4 as a search key. The presentation unit 111 identifies and narrows down the records that match the specified determination type and match the imaging unit identification information of the specified imaging unit 21.

[0101] The presentation unit 111, while referring to the identified record, identifies information indicating the image capture unit identification information and the judgment result from among the information included in the record. When the identified information indicating the judgment result indicates "NG," the presentation unit 111 generates a map image IMd and updates the confirmation screen IMb so as to display a symbol image SY superimposed on the icon ICN corresponding to the image capture unit identification information associated with the "NG" judgment result. The symbol image SY also includes a number image indicating the number of identified records for which the information indicating the judgment result in the image capture unit 21 corresponding to the superimposed icon ICN was "NG." As shown in FIG. 6, in this example, the symbol image SY superimposed on the icon ICNa indicates that seven "NG" judgments were made during a specified period at the stop position LN captured by the image capture unit 21a corresponding to the icon ICNa.

[0102] The presenting unit 111 also changes the display mode of the symbol image SY based on the number of occurrences of the violation event EV1 indicated in the judgment result information 155. For example, the presenting unit 111 may enlarge, compared to its normal size, the symbol image SY superimposed on the icon ICN of the imaging unit 21 capturing an image of a regulated location where a large number of violation events EV1 occurred during a period designated by the pull-down lists PD1 to PD2. As shown in FIG. 6, in this example, the symbol image SY superimposed on the icon ICNb indicates that one "NG" judgment was made during the designated period at the stop position LN captured by the imaging unit 21b corresponding to the icon ICNb. Therefore, the number of "NG" judgments made by the imaging unit 21b is smaller than the number of "NG" judgments made by the imaging unit 21a. Accordingly, the presenting unit 111 generates a map image IMd and updates the confirmation screen IMb so that the symbol image SY superimposed on the icon ICNa is displayed larger than the symbol image SY superimposed on the icon ICNb.

[0103] [Operation of information processing device 10] Hereinafter, the details of the processing of the information processing device 10 in steps ST3 and ST4 will be described with reference to Fig. 7. The processing of the flowchart shown in Fig. 7 is executed as an initial setting, for example, when the information processing system 1 is introduced, when the management device 20 is installed on the premises or the like, when the angle of view of the imaging unit 21 is changed, when a new imaging unit 21 is installed, before the determination process is performed in the information processing device 10, etc.

[0104] First, the determination unit 101 presents a preset image IM1 to the user of the management device 20 (step S100). Next, the determination unit 101 determines whether the user of the management device 20 has performed an operation to specify a detection range AR, an operation to specify a reference direction DR, and an operation to specify a predetermined distance d (step S102). The determination unit 101 waits until the user of the management device 20 has performed an operation to specify a detection range AR, an operation to specify a reference direction DR, and an operation to specify a predetermined distance d (step S102; NO). When the determination unit 101 determines that the user of the management device 20 has performed an operation to specify a detection range AR, an operation to specify a reference direction DR, and an operation to specify a predetermined distance d (step S102; YES), the determination unit 101 determines the detection range AR based on the operation to specify the detection range AR, and determines the reference direction DR based on the operation to specify the reference direction DR, and determines the predetermined distance d used to set the forward detection range AR′ (step S104). Specifically, the determination unit 101 determines a pixel region within a range specified by vertices P in the image IMa constituting the moving image MV as the detection range AR. The determination unit 101 also acquires information indicating the coordinates of the start point SP and the end point FP of the reference direction DR in the moving object-free image IMa1. The determination unit 101 determines the direction specified by the start point SP and the end point FP in the image IMa constituting the moving image MV as the reference direction DR. The determination unit 101 also acquires information indicating the distance input to the management device 20. The determination unit 101 determines the acquired information indicating the distance as the predetermined distance d.

[0105] The process of the information processing device 10 in step ST5 will be described in detail below with reference to Fig. 8. The process of the flowchart shown in Fig. 8 is executed for a moving image MV every time the management device 20 transmits the moving image MV to the server device 30, for example.

[0106] First, the position identification unit 102 reads out the video MV from the storage unit of the server device 30 via the network NW and acquires the video MV (step S200). For the video MV acquired in step S200, the position identification unit 102 inputs the image IMa into the trained model 151. For each image IMa, the trained model 151 outputs, as coordinates, information indicating the position of the main body, the position of the mast, and the position of the balance weight of the forklift TG1 shown in the image IMa. The position identification unit 102 acquires the output coordinates (step S202).

[0107] Next, the traveling direction identification unit 103 identifies the traveling direction of the forklift truck TG1 based on the change over time in the coordinates of the bounding box BX identified by the trained model 151 (step S204). Next, the orientation identification unit 104 identifies the orientation of the forklift truck TG1 based on the change over time in the coordinates of the bounding box BXc and the coordinates of the bounding box BXd identified by the trained model 151 (step S206). Next, the movement direction identification unit 105 identifies whether the moving forklift truck TG1 is moving forward or backward based on the identified orientation and traveling direction (step S208).

[0108] Next, the application unit 106 determines whether or not it is necessary to change the detection range AR (step S210). Specifically, the application unit 106 determines whether or not it has been determined by the movement direction identification unit 105 that the forklift TG1 is moving forward. If the movement direction identification unit 105 determines that the forklift TG1 is moving backward, the application unit 106 determines that it is not necessary to change the detection range AR (step S210; NO), and proceeds to step S214, assuming that the detection range AR will be used in the subsequent determination processing. If the movement direction identification unit 105 determines that the forklift TG1 is moving forward, the application unit 106 determines that it is necessary to change the detection range AR (step S210; YES), and therefore sets a forward detection range AR' that is different from the detection range AR in the subsequent determination processing (step S212).

[0109] Next, the behavior identification unit 107 identifies the behavior of the forklift TG1 shown in the video MV based on the video MV (step S214). Specifically, the behavior identification unit 107 identifies the speed, maximum speed, minimum speed, and acceleration of the forklift TG1, and the duration of time during which the speed of the forklift TG1 within the detection range AR or the forward detection range AR' remains equal to or less than the speed threshold value, based on the video MV.

[0110] Next, the determination unit 108 determines whether the traveling direction identified by the traveling direction identification unit 103 matches the reference direction DR determined by the determination unit 101 (step S216). Specifically, if the traveling direction of the forklift TG1 identified by the traveling direction identification unit 103 when the forklift TG1 enters the detection range AR is within a predetermined angle range centered on the reference direction DR, the determination unit 108 determines that the traveling direction of the forklift TG1 identified by the traveling direction identification unit 103 matches the reference direction DR of the stop position LN shown in the video MV. If the determination unit 108 determines that the traveling direction does not match the reference direction DR (step S216; NO), the determination unit 108 determines that the forklift TG1 shown in the video MV is a forklift TG1 to which stopping at the stop position LN does not apply, and ends the series of processes without executing subsequent processes.

[0111] When the determination unit 108 determines that the traveling direction of the forklift truck TG1 identified by the traveling direction identification unit 103 matches the reference direction DR (step S216; YES), it determines whether a violation event has occurred (step S218). Specifically, the determination unit 108 determines whether the forklift truck TG1 is performing a predetermined behavior within the detection range AR or the forward detection range AR'. When the forklift truck TG1 identified as moving backward in step S208 is not performing a predetermined behavior within the detection range AR, the determination unit 108 determines that a violation event EV1 has occurred. More specifically, based on the identification result of the behavior identification unit 107, when the time during which the speed of the forklift truck TG1 within the detection range AR remains equal to or less than the speed threshold is shorter than a predetermined time, the determination unit 108 determines that the forklift truck TG1 is not performing a predetermined behavior within the detection range AR and that a violation event EV1 has occurred. The determination unit 108 determines that a violation event EV1 has occurred if the forklift truck TG1 identified in step S208 as moving forward is not performing a predetermined behavior within the forward detection range AR'. More specifically, based on the identification result of the behavior identification unit 107, if the time during which the speed of the forklift truck TG1 within the forward detection range AR' remains equal to or less than the speed threshold is less than a predetermined time, the determination unit 108 determines that the forklift truck TG1 is not performing a predetermined behavior within the forward detection range AR' and that a violation event EV1 has occurred.

[0112] On the other hand, when the traveling direction of the forklift truck TG1 identified by the traveling direction identification unit 103 matches the reference direction DR of the stop position LN shown in the video MV, and based on the identification result of the behavior identification unit 107, if the time period during which the speed of the reverse forklift truck TG1 within the detection range AR is equal to or less than the speed threshold continues for a predetermined time or longer, the determination unit 108 determines that the forklift truck TG1 is behaving in a predetermined manner within the detection range AR and therefore the violation event EV1 has not occurred (i.e., the compliance event EV2 has occurred). Furthermore, when the traveling direction of the forklift truck TG1 identified by the traveling direction identification unit 103 matches the reference direction DR of the stop position LN shown in the video MV, and based on the identification result of the behavior identification unit 107, if the time period during which the speed of the forward forklift truck TG1 within the forward detection range AR' is equal to or less than the speed threshold continues for a predetermined time or longer, the determination unit 108 determines that the forklift truck TG1 is behaving in a predetermined manner within the forward detection range AR' and therefore the violation event EV1 has not occurred (i.e., the compliance event EV2 has occurred).

[0113] When the determination unit 108 determines that the violation event EV1 has occurred (step S218; YES), the video generation unit 109 generates a violation video MV1 by cutting out a portion of the video MV that includes an image IMa of the scene where the violation event EV1 has occurred (step S220). Specifically, the video generation unit 109 cuts out, from the video MV, a portion from when the reverse forklift TG1 enters the detection range AR to when it exits the detection range AR, or a portion from when the forward forklift TG1 enters the forward detection range AR' to when it exits the forward detection range AR', where the violation event EV1 has been determined to have occurred, as the violation video MV1. Next, the video generation unit 109 generates the violation video MV1 by superimposing a bounding box BX indicating the position of the forklift TG1 identified by the trained model 151 and a line indicating the detection range AR or the forward detection range AR' on each image IMa of the cut-out video. Next, the video generating unit 109 stores the generated violation video MV1 in the violation event video folder 153 of the storage unit 150 (step S222).

[0114] When the determination unit 108 determines that the compliance event EV2 has occurred (step S218; NO), the video generation unit 109 generates a compliance video MV2 by cutting out a portion of the video MV that includes an image IMa of the screen where the compliance event EV2 has occurred (step S224). Specifically, the video generation unit 109 cuts out, from the video MV, a portion from when the reverse forklift TG1 enters the detection range AR to when it exits the detection range AR, or a portion from when the forward forklift TG1 enters the forward detection range AR' to when it exits the forward detection range AR', where the compliance event EV2 has been determined to have occurred, as the compliance video MV2. Next, the video generation unit 109 generates the compliance video MV2 by superimposing a bounding box BX indicating the position of the forklift TG1 identified by the trained model 151 and a line indicating the detection range AR or the forward detection range AR' on each image IMa of the cut-out video. Next, the video generating unit 109 stores the generated compliance video MV2 in the compliance event video folder 154 of the storage unit 150 (step S226).

[0115] Next, the judgment result generating unit 110 generates / updates the judgment result information 155 indicating the judgment result corresponding to the violation video MV1 and the judgment result corresponding to the compliance video MV2 (step S228).

[0116] The process of the information processing device 10 in procedure ST6 will be described in detail below with reference to Fig. 9. The process of the flowchart shown in Fig. 9 is executed, for example, at the timing when the user of the management device 20 wishes to check the determination result information 155.

[0117] First, the presentation unit 111 generates a confirmation screen IMb and presents it to the user of the management device 20 (step S300). Next, the presentation unit 111 determines whether the user of the management device 20 has specified a period of time for confirmation using the pull-down lists PD1 to PD2, the start point slider CS1, and the end point slider CS2 on the confirmation screen IMb (step S302). If the pull-down lists PD1 to PD2 and the slider bar SB have not been operated, the presentation unit 111 determines that no period has been specified (step S302; NO), and proceeds to step S306. If the pull-down lists PD1 to PD2, the start point slider CS1, and the end point slider CS2 have been operated to specify a period (step S302; YES), the presentation unit 111 acquires information indicating the period specified by the operation and identifies a record of the determination result information 155 that matches the period (step S304). The presentation unit 111 searches the determination result information 155 using the dates and times of the period specified by the pull-down lists PD1 to PD2 and the start point slider CS1 and end point slider CS2 as search keys. The presentation unit 111 identifies and narrows down records that match the dates of the period.

[0118] Next, the presentation unit 111 determines whether the user of the management device 20 has specified a judgment type to be confirmed using the pull-down list PD3 on the confirmation screen IMb (step S306). If the pull-down list PD3 has not been operated, the presentation unit 111 determines that no judgment type has been specified (step S306; NO), and proceeds to step S310. If the pull-down list PD3 has been operated to specify a judgment type (step S306; YES), the presentation unit 111 acquires information indicating the judgment type specified by the operation and identifies a record of the judgment result information 155 (step S308). The presentation unit 111 uses the judgment type specified by the pull-down list PD3 as a search key to search for the record of the judgment result information 155 specified in step S304, or searches for the judgment result information 155 if no record has been specified yet. The presentation unit 111 identifies and narrows down the records associated with the judgment type.

[0119] Next, the presentation unit 111 determines whether a target imaging unit desired to be confirmed has been designated by the user of the management device 20 using the pull-down list PD4 on the confirmation screen IMb (step S310). If the pull-down list PD4 has not been operated, the presentation unit 111 determines that a target imaging unit has not been designated (step S310; NO), and proceeds to step S314. If the pull-down list PD4 has been operated to designate a target imaging unit (step S310; YES), the presentation unit 111 acquires information indicating the target imaging unit designated by the operation and identifies a record of the determination result information 155 (step S312). The presentation unit 111 searches for the record of the determination result information 155 identified in step S304 or step S308, using the target imaging unit designated by the pull-down list PD4 as a search key, or searches for the determination result information 155 if no record has yet been identified. The presentation unit 111 identifies and narrows down the records associated with the target imaging unit.

[0120] Next, if a period is acquired in step S304, the presentation unit 111 generates a map image IMd representing records for that period, or if a period is not acquired, generates records for the entire period (step S314). The presentation unit 111 updates the confirmation screen IMb including the map image IMd generated in step S314 and presents it to the user of the management device 20 (step S316), and then ends the series of processes.

[0121] [Effects of the embodiment] According to the above embodiment, the following effects can be obtained. (1) The orientation identification unit 104 identifies the orientation of the forklift TG1 based on the position of the balance weight and the position of the mast, which are characteristic points of the forklift TG1 shown in the video MV. The traveling direction identification unit 103 identifies the traveling direction of the forklift TG1 based on the main body position or characteristic points of the forklift TG1 shown in the video MV. The movement direction identification unit 105 identifies whether the forklift TG1, which is moving toward the stop position LN, is moving forward or backward based on the identified orientation and traveling direction of the forklift TG1. The application unit 106 sets a forward detection range AR' that is different from the predetermined detection range AR for images IMa that constitute the video MV and that are identified as images in which the forklift TG1 is moving forward. The behavior identification unit 107 identifies the behavior of the forklift TG1 shown in the video MV based on the video MV. The determination unit 108 determines that a violation event EV1 has occurred if at least a part of the main body of the forklift TG1 has not performed a predetermined behavior within the detection range AR after entering the detection range AR based on the identified main body position of the forklift TG1. The determination unit 108 also determines that a violation event EV1 has occurred if at least a part of the main body of the forklift TG1 has not performed a predetermined behavior within the forward detection range AR' after entering the forward detection range AR'. When it is determined that a violation event EV1 has occurred, the video generation unit 109 generates a violation video MV1 by extracting a part of the video MV that includes an image IMa of the scene where the violation event EV1 occurred from the video MV.

[0122] Here, it is time-consuming for a person to check the video MV and determine whether a stop sign is being applied based on signs, etc., provided along the route of travel within the premises, etc. Furthermore, it is time-consuming for a person to edit the video MV and generate a violation event EV1 in order to share with others the scene in the video MV where a stop sign violation occurred.

[0123] With this configuration, the determination unit 108 can determine that a violation event EV1 has occurred if the speed of the forklift TG1 within the detection range AR or the forward detection range AR' remains equal to or less than the speed threshold for less than a predetermined time. Furthermore, with this configuration, the determination unit 108 can prevent itself from erroneously determining that a violation event EV1 has occurred when the forklift TG1 passes through a stop position LN beyond an intersection that is not subject to regulation for the forklift TG1 without temporarily stopping. That is, in cases where regulation does not need to be applied to the forklift TG1, as described above, the traveling direction of the forklift TG1 does not match the reference direction DR, and the forklift TG1 can be excluded from the determination target. As a result, the determination unit 108 can prevent itself from erroneously determining that a violation event EV1 has occurred.

[0124] Furthermore, when the forklift TG1 is moving backward toward the stop position LN, the driver of the forklift TG1 is required to stop the forklift TG1 so that the rear of the forklift TG1 is positioned in front of the stop position LN. In this case, the front lower end JP of the main body of the forklift TG1 is within the detection range AR. On the other hand, when the forklift TG1 is moving forward toward the stop position LN, the driver of the forklift TG1 is required to stop the forklift TG1 so that the tips of the forks of the forklift TG1 are positioned in front of the stop position LN if the forklift TG1 is not loaded, and is required to stop the forklift TG1 so that the tips of the load of the forklift TG1 are positioned in front of the stop position LN if the forklift TG1 is loaded. In this case, the front lower end JP of the main body of the forklift TG1 is located in front of the detection range AR. Therefore, when the forklift TG1 is moving forward toward the stop position LN, the determination unit 108 cannot use the detection range AR to determine whether or not to temporarily stop the forklift TG1.

[0125] On the other hand, in the above embodiment, when the forklift TG1 moving in the direction of the stop position LN is moving forward, the application unit 106 sets a forward detection range AR' that is different from the detection range AR. This allows the determination unit 108 to make a determination for the forklift TG1 to which the regulation applies based on the direction of the forklift TG1 and whether the forklift TG1 is moving forward or backward, while the video generation unit 109 can generate a video showing a scene in which the regulation is violated.

[0126] (2) The determination unit 108 determines that a compliance event has occurred if, based on the video MV in which the forklift TG1 is identified as moving backward, the speed of the forklift TG1 within the detection range AR continues to be equal to or less than the speed threshold for a predetermined period of time or more. Furthermore, the determination unit 108 determines that a compliance event EV2 has occurred if, based on the video MV in which the forklift TG1 is identified as moving forward, the speed of the forklift TG1 within the forward detection range AR' continues to be equal to or less than the speed threshold for a predetermined period of time or more. When it is determined that the compliance event EV2 has occurred, the video generation unit 109 generates a compliance video MV2 by extracting a portion of the video MV that includes an image IMa of the scene in which the compliance event EV2 occurred.

[0127] According to this configuration, the information processing device 10 can determine whether the forklift TG1 to which the regulation applies complies with the regulation, and generate a compliance video MV2 showing a scene in which the regulation is complied with.

[0128] (3) The forward detection range AR′ is a range obtained by moving the detection range AR by a predetermined distance d in the direction opposite to the reference direction DR. With this configuration, the application unit 106 can easily set the forward detection range AR′ based on the detection range AR determined by the user of the management device 20.

[0129] (4) When an image IMa constituting the video MV is input, the position identification unit 102 outputs a first position, which is the position of the balance weight on the image IMa, and a second position, which is the position of the mast on the image IMa, into the trained model 151.The trained model 151 inputs the image IMa constituting the video MV, and identifies the first position and the second position based on the information output by inputting the image IMa constituting the video MV.

[0130] According to this configuration, the position identification unit 102 can identify the first position of the balance weight and the second position of the mast as feature points of the forklift TG1 by a simple method using the trained model 151.

[0131] (5) When an image IMa constituting the video MV is input, the position identification unit 102 inputs the image IMa into the trained model 151, which outputs the position of the forklift TG1 using a bounding box BXa, and identifies the position of the forklift TG1 based on the information output by inputting the image IMa. In this case, "at least a part of the main body of the forklift TG1" refers to the lower front end JP in the traveling direction of the bounding box BXa representing the forklift TG1 on the image IMa.

[0132] According to this configuration, the position specifying unit 102 can specify the position of the main body of the forklift TG1 by a simple method using the trained model 151. (6) The judgment result generating unit 110 generates judgment result information 155 indicating the judgment result corresponding to the violation video MV1. The judgment result generating unit 110 also generates judgment result information 155 indicating the judgment result corresponding to the compliance video MV2. With this configuration, by referencing the judgment result information 155, detailed information about the violation event EV1 indicated by the violation video MV1 can be ascertained for each event. With this configuration, by referencing the judgment result information 155, detailed information about the compliance event EV2 indicated by the compliance video MV2 can be ascertained for each event.

[0133] (7) The video generation unit 109 stores the violation event EV1 in the violation event video folder 153 and the compliance event EV2 in the compliance event video folder 154 of the storage unit 150 that stores the judgment result information 155. With this configuration, the violation video MV1 and the compliance video MV2 can be accessed more easily than when the violation video MV1 and the compliance video MV2 are mixed in the same storage location.

[0134] (8) The judgment result generating unit 110 identifies the forklift TG1 involved in the violation event EV1 based on the moving object identification information attached to the forklift TG1, and generates judgment result information 155 by further including the identified moving object identification information. Also, the judgment result generating unit 110 identifies the forklift TG1 involved in the compliance event EV2 based on the moving object identification information attached to the forklift TG1, and generates judgment result information 155 by further including the identified moving object identification information.

[0135] According to this configuration, the determination result generating unit 110 can refer to information about the forklift TG1 that violates the regulations by referring to the determination result information 155. Furthermore, the determination result generating unit 110 can refer to information about the forklift TG1 that complies with the regulations by referring to the determination result information 155.

[0136] (9) The determination result generating unit 110 generates determination result information 155 including imaging unit identification information that can identify the imaging unit 21. The presenting unit 111 generates a confirmation screen IMb based on a map image IMd that indicates a predetermined position where the imaging unit 21 corresponding to the imaging unit identification information is provided and a travel route, and the determination result information 155. With this configuration, the presenting unit 111 can intuitively and easily present the position on the travel route where a violation of a regulation occurred or the position where compliance with a regulation occurred.

[0137] The above-described embodiments may be modified as follows: The above-described embodiments and the following modifications may be combined with each other within the scope of technical compatibility. The information processing system 1 may include a plurality of management devices 20. In this case, the information processing device 10 executes the above-described process for each management device 20. Furthermore, the management device 20 may be connected to one imaging unit 21.

[0138] In the case where moving images MV are generated at intervals of several minutes to several tens of minutes as described above, when the moving image generating unit 109 generates a violation moving image MV1 or a compliance moving image MV2 from a part of the moving image MV, a violation event EV1 or a compliance event EV2 may occur in a gap in the moving image MV. In this case, the moving image generating unit 109 generates the violation moving image MV1 and the compliance moving image MV2 using the moving image MV used for the judgment and a moving image MV that is chronologically consecutive to the moving image MV.

[0139] The information included in the determination result information 155 is merely an example and is not limited to this. The determination result information 155 may include information other than that described above, or may include some of the information described above.

[0140] A barcode is one example of a method for realizing the moving object identification information, and is not limited to this. The moving object identification information may be realized, for example, by RFID (Radio Frequency Identification) or a beacon. In this case, a receiving device for RFID or a beacon is provided near the imaging unit 21, and the receiving device acquires the moving object identification information by receiving the RFID or beacon attached to the forklift TG1 traveling near the stopping position LN. The management device 20 transmits information to the server device 30 that associates the moving object identification information with the date and time when the moving object identification information was read. Based on this information, the determination result generation unit 110 identifies the moving object identification information of the forklift TG1 corresponding to the violation video MV1 or the forklift TG1 corresponding to the compliance video MV2.

[0141] In the above description, the determination process is performed based on the traveling direction of the forklift truck TG1 identified by the traveling direction identification unit 103 and the predetermined angle range centered on the reference direction DR. However, this is not limited to this. The determination unit 108 may determine whether the violation event EV1 or the compliance event EV2 has occurred only when the traveling direction of the forklift truck TG1 identified by the traveling direction identification unit 103 coincides with the reference direction DR. Furthermore, the user of the management device 20 may set the predetermined angle range.

[0142] Although the case where the management device 20 transmits the video MV to the server device 30 has been described, this is not limiting. The management device 20 may transmit the video MV to the information processing device 10. Furthermore, if the information processing device 10 has a storage unit that the server device 30 has, the information processing system 1 does not need to have the server device 30. The information processing device 10 stores the received video MV in the storage unit 150 or another storage unit that has the same function as the storage unit that the server device 30 has. Each unit included in the control unit 100 performs various processes based on the video MV received by the information processing device 10.

[0143] The video MV may be constantly transmitted by the management device 20 to the server device 30 and / or the information processing device 10. The processing of the flowchart in Fig. 8 may be executed at a frequency other than every time the management device 20 transmits a moving image MV to the server device 30 or the information processing device 10. The processing of the flowchart in Fig. 8 may be executed, for example, at predetermined time intervals (for example, once a day). In this case, the processing of the flowchart shown in Fig. 8 is repeatedly executed for each moving image MV newly acquired from the server device 30.

[0144] When the moving image MV includes a plurality of forklifts TG1 moving, the information processing device 10 executes the processes of steps S202 to S226 of the flowchart shown in FIG. 8 for each of the forklifts TG1.

[0145] The pull-down list PD displayed on the confirmation screen IMb is an example and is not limited to this. For example, the pull-down list PD may be provided in a number corresponding to the number of selectable pieces of information included in the determination result information 155.

[0146] The confirmation screen IMb does not have to include the pull-down lists PD1 to PD2 and the slider bar SB. In this case, the presentation unit 111 omits the processes of steps S302 and S304 in the flowchart shown in FIG.

[0147] Of the processes shown in FIG. 9, the processes of steps S302 to S304, steps S306 to S308, and steps S310 to S312 may be executed in any order. The reference direction DR may be determined by a method other than the user of the management device 20 inputting the start point SP and the end point FP. As shown in FIG. 10 , a forklift truck TG1 moving backward stops within a detection range AR, then exceeds the detection range AR, passes a stop position LN, and then proceeds into an entry range ARa. Furthermore, a forklift truck TG1 moving forward stops within a forward detection range AR′, then moves from the forward detection range AR′ beyond the detection range AR, passes a stop position LN, and then proceeds into the entry range ARa. The determination unit 101 identifies the entry range ARa by, for example, performing general image processing on the moving-object-free image IMa1. Specifically, the image capture unit 21 captures an intersection or the like on the route of travel within the premises, and the determination unit 101 identifies the shape of the intersection shown in the moving-object-free image IMa1 to identify the central area of ​​the intersection as the entry range ARa. In this case, the determining unit 101 determines the direction connecting the detection range AR and the entry range ARa as the reference direction DR, without relying on the operation of the user of the management device 20. The direction connecting the detection range AR and the entry range ARa is, for example, the direction connecting the center of the detection range AR and the center of the entry range ARa. Furthermore, when the position of the forklift TG1 determined by the position determining unit 102 indicates that it has moved from the detection range AR to the entry range ARa, the determining unit 108 determines that the traveling direction of the forklift TG1 matches the reference direction DR.

[0148] According to this configuration, the reference direction DR can be determined without relying on the operation of the user of the management device 20, and the forklift TG1 whose traveling direction matches the reference direction can be identified as a target to which the regulation applies.

[0149] Instead of using the preset image IM1, the determination unit 101 may determine the detection range AR by performing general image processing on the moving-object-free image IMa1. Specifically, the imaging unit 21 captures an image of the movement path and the stop positions LN within the premises, and the determination unit 101 determines a predetermined area as the detection range AR by identifying the shape of the movement path shown in the moving-object-free image IMa1 and the position and direction of the stop positions LN attached to the movement path. In this case, the determination unit 101 determines the detection range AR without relying on the operation of the user of the management device 20.

[0150] Similarly to the detection range AR, the approach range ARa may be determined by the determination unit 101 in response to input of a plurality of vertices P representing the approach range ARa in the preset image IM1.

[0151] Instead of moving the detection range AR by a predetermined distance d in the direction opposite to the reference direction DR, the forward detection range AR' may be specified by the user of the management device 20, similar to the detection range AR. In this case, the determination unit 101 determines the forward detection range AR' based on information indicating the forward detection range AR' obtained from the management device 20. Specifically, the determination unit 101 obtains information indicating the coordinates of each vertex P of the forward detection range AR' in the moving object-free image IMa1. The determination unit 101 determines, as the forward detection range AR', a pixel region within the range specified by the vertices P of the image IMa constituting the moving image MV.

[0152] The detection range AR may be represented in a form other than a quadrangle having four vertices. As shown in FIG. 11, the moving object-free image IMa1 may show a portion of the entire detection range AR missing. In this case, the detection range AR cannot be represented by a quadrangle. Therefore, the detection range AR may be represented by a triangle having three vertices P or a polygon having four or more vertices P. The determination unit 101 determines, as the detection range AR, a pixel region within a range specified by the vertices P of the image IMa constituting the moving image MV. Similarly, the entry range ARa may be represented in a form other than a quadrangle having four vertices. That is, the entry range ARa may be represented by a triangle having three vertices P or a polygon having four or more vertices P.

[0153] If the moving object-free image IMa1 includes multiple stop positions LN, the detection range AR and the reference direction DR corresponding to the detection range AR may be specified for the multiple stop positions LN using the preset image IM1. The determination unit 101 determines the detection range AR and the reference direction DR based on the multiple detection ranges AR specified using the preset image IM1 and the information on the reference directions DR corresponding to each of the detection ranges AR. The determination unit 108 performs a determination process for each detection range AR. With this configuration, the determination unit 108 can make determinations regarding multiple stop positions LN based on the image IMa generated by one imaging unit 21.

[0154] The position identification unit 102 may identify the main body of the forklift TG1 using a method other than the bounding box BXa. For example, when the position identification unit 102 receives the image IMa, the position identification unit 102 may identify the main body of the forklift TG1 using a trained model that segments the entire area of ​​the image IMa of the forklift TG1. In this case, the position identification unit 102 inputs each of the images IMa constituting the video MV into the trained model and acquires, for each image IMa, information indicating the area of ​​the forklift TG1 shown in the image IMa. In this case, at least a portion of the main body of the forklift TG1 refers to the position of the front end of the area indicating the forklift TG1 in the traveling direction on the image IMa.

[0155] The orientation identification unit 104 may identify a first position, which is the position of the balance weight on the image IMa, and a second position, which is the position of the mast on the image IMa, by a method other than the bounding boxes BXc and BXd. For example, when the orientation identification unit 104 receives the image IMa, the orientation identification unit 104 may identify the first position and the second position using a trained model that segments the entire area on the image IMa of the balance weight and the entire area on the image IMa of the mast. In this case, the orientation identification unit 104 inputs each of the images IMa constituting the video MV into the trained model and acquires information indicating the area on the image IMa of the balance weight and the area on the image IMa of the mast shown in each image IMa.

[0156] If the orientation of the forklift truck TG1 can be identified by defining a position behind the forklift truck TG1 as the first position and a position in front of the forklift truck TG1 as the second position, positions other than the balance weight and the mast may be used as feature points of the forklift truck TG1. In this case, the orientation identification unit 104 identifies the positions of the feature points other than the balance weight and the mast.

[0157] The forklift TG1 may be provided with feature point identification information indicated by a barcode or the like, for example, on a position on the top surface or the like that can be imaged by the imaging unit 21. The feature point identification information is, for example, information that can identify the front and rear of the forklift TG1. Furthermore, moving object identification information may be used as the feature point identification information. The orientation identification unit 104 may identify the orientation of the target forklift TG1 by performing general image processing on the image IMa to recognize the feature point identification information.

[0158] Although the above description has been given of a case where the stop restriction imposed on the travel route is a stop position LN, this is not limited to this. A stop restriction may be indicated by a mode other than a stop position LN. Examples of other modes include a crosswalk imposed on the travel route, a no-entry area established on the travel route according to unique rules, and other areas that require a stop before entering. In this case, the determination unit 101 prompts the user of the management device 20 to set a detection range AR for these restrictions using a preset image IM1. For example, in the case of a crosswalk, by setting a detection range AR and a reference direction DR on both sides of the crosswalk before and after crossing, the determination unit 108 can prevent erroneous determination that a violation event EV1 has occurred even if the vehicle does not make a temporary stop in the detection range AR at the end of the crossing.

[0159] The restriction imposed on the travel route may be a restriction other than a stop. Specifically, the restriction imposed on the travel route may be a slow-down restriction. In this case, the determination unit 101 prompts the user of the management device 20 to set a detection range AR to which a slow-down restriction is applied using the preset image IM1. When the travel direction of the forklift TG1 identified by the travel direction identification unit 103 matches the reference direction DR of the stop position LN shown in the video MV, the determination unit 108 determines, based on the identification result of the behavior identification unit 107, that the forklift TG1 is not performing the predetermined behavior and that a violation event EV1 has occurred if the speed of the forklift TG1 in the detection range AR or the forward detection range AR' remains below the speed threshold for less than a predetermined time. The speed threshold is a value indicating a number [km / h] when slow-down is required at a restriction location. Similarly, when the traveling direction of the forklift TG1 identified by the traveling direction identification unit 103 matches the reference direction DR of the stop position LN shown in the video MV, and based on the identification result of the behavior identification unit 107, the behavior identification unit 108 determines that the forklift TG1 is performing a predetermined behavior within the detection range AR or the forward detection range AR' and that a compliance event EV2 has occurred if the speed of the forklift TG1 within the detection range AR or the forward detection range AR' remains below the speed threshold for a predetermined period of time or longer. In this case, the determination result generation unit 110 identifies the determination type as "slow driving" as information to be included in the determination result information 155. With this configuration, the determination unit 108 can determine that the forklift TG1 that is not slowing down is not performing a predetermined behavior or has performed a predetermined behavior within the detection range AR or the forward detection range AR'. In this case, the location where slow driving is applied is an example of a restriction location.

[0160] The determination unit 108 may determine whether the forklift TG1 has appropriately stopped temporarily or slowed down based on the time spent within the detection range AR or the forward detection range AR'.

[0161] In this case, the behavior identification unit 107 identifies the time that at least a portion of the main body of the forklift truck TG1 remained within the detection range AR or the forward detection range AR' after entering the detection range AR, based on the position of the forklift truck TG1 identified by the position identification unit 102. When the traveling direction of the forklift truck TG1 identified by the traveling direction identification unit 103 matches the reference direction DR of the stop position LN shown in the video MV, the determination unit 108 determines, based on the identification result of the behavior identification unit 107, that a violation event EV1 has occurred if the time that at least a portion of the main body of the forklift truck TG1 remained within the detection range AR or the forward detection range AR' after entering the detection range AR or the forward detection range AR' is less than a predetermined time. The determination unit 108 determines that a violation event EV1 has not occurred if the time that at least a portion of the main body of the forklift truck TG1 remained within the detection range AR or the forward detection range AR' after entering the detection range AR or the forward detection range AR' is equal to or longer than a predetermined time.

[0162] "At least a portion of the main body of the forklift TG1" refers to the position of the front lower end JP in the traveling direction of the bounding box BXa representing the forklift TG1 on the image IMa. For example, when the forklift TG1 is required to stop at a regulating position, the predetermined time is the time required for the forklift TG1 to pass through the detection range AR or the forward detection range AR' when the forklift TG1 has properly stopped within the detection range AR or the forward detection range AR'. Also, when the forklift TG1 is required to slow down at a regulating position, the predetermined time is the time required for the forklift TG1 to pass through the detection range AR or the forward detection range AR' when the forklift TG1 has properly slowed down within the detection range AR or the forward detection range AR'. Whether the forklift TG1 is stopped or slowing down, the predetermined time is approximately several seconds to 10-odd seconds. Meanwhile, the predetermined time is longer when the forklift TG1 is stopped than when it slows down.

[0163] Based on the identification result of the behavior identification unit 107, if the time during which at least a part of the main body of the forklift TG1 remains within the detection range AR after entering the detection range AR and the time during which the speed remains below the speed threshold is less than a predetermined time, the determination unit 108 determines that a violation event EV1 has occurred, assuming that the reversing forklift TG1 did not stop or slow down appropriately.

[0164] Furthermore, based on the identification result of the behavior identification unit 107, the determination unit 108 determines that the forklift TG1 did not stop or slow down appropriately and determines that a violation event EV1 has occurred if the time during which at least a part of the main body of the forklift TG1 remained within the forward detection range AR' after entering the forward detection range AR' and the time during which the speed remained below the speed threshold is less than a predetermined time. Similarly, the determination unit 108 determines that the forklift TG1 did stop or slow down appropriately and determines that a compliance event EV2 has occurred if the time during which at least a part of the main body of the forklift TG1 remained within the detection range AR after entering the detection range AR and the time during which the speed remained below the speed threshold is greater than or equal to a predetermined time. Furthermore, if the time during which at least a part of the main body of the forward moving forklift TG1 remains within the forward moving detection range AR' after entering the forward moving detection range AR' and the time during which the speed remains below the speed threshold is equal to or longer than a predetermined time, the judgment unit 108 judges that the forklift TG1 has stopped or slowed down appropriately and that a compliance event EV2 has occurred.

[0165] Alternatively, the determination unit 108 may determine that the reverse forklift TG1 did not stop or slow down appropriately when the speed of the reverse forklift TG1 when entering the detection range AR or the average speed of the reverse forklift TG1 within the detection range AR exceeds a speed threshold, and therefore determine that a violation event EV1 has occurred. Alternatively, the determination unit 108 may determine that the reverse forklift TG1 did not stop or slow down appropriately when the speed of the forward forklift TG1 when entering the forward detection range AR' or the average speed of the forward forklift TG1 within the forward detection range AR' exceeds a speed threshold, and therefore determine that a violation event EV1 has occurred. Similarly, the determination unit 108 may determine that the forklift TG1 did stop or slow down appropriately when the speed of the reverse forklift TG1 when entering the detection range AR or the average speed of the reverse forklift TG1 within the detection range AR is equal to or less than the speed threshold, and therefore determine that a compliance event EV2 has occurred. In addition, if the speed of the forward forklift TG1 when entering the forward detection range AR' or the average speed of the forward forklift TG1 within the forward detection range AR' becomes equal to or less than the speed threshold, the judgment unit 108 determines that the forklift TG1 has stopped or slowed down appropriately and that a compliance event EV2 has occurred.

[0166] The determination unit 108 may further determine the degree of violation of the regulation in the violation event EV1 based on the manner of movement of the forklift truck TG1 within the detection range AR or the forward detection range AR'. For example, the determination unit 108 may determine the degree of violation of the regulation as one of two levels, such as "bad" or "very bad," depending on the degree of violation. In this case, "bad" indicates that the regulation is violated but the degree of violation is minor, and "very bad" indicates that the regulation is violated and the degree of violation is severe. For example, if the regulation is a temporary stop, the determination unit 108 determines the degree of violation to be "very bad" the faster the speed of the forklift truck TG1 reversing within the detection range AR or the forward detection range AR' is above a predetermined speed or the greater the acceleration of the forklift truck TG1.

[0167] The determination result generating unit 110 generates the determination result information 155 by further including information indicating the degree of violation in the determination result information 155. With this configuration, the determination result generating unit 110 can refer to the determination result information 155 to further refer to the degree of violation of the forklift TG1 that has violated the regulation.

[0168] The determination unit 108 may further determine the level of compliance with the regulation in the compliance event EV2 based on the movement of the forklift truck TG1 within the detection range AR or the forward detection range AR'. For example, the determination unit 108 may determine the level of compliance with the regulation as one of two levels, such as "good" or "excellent." In this case, "good" indicates that the regulation is complied with but to a low degree, while "excellent" indicates that the regulation is complied with and to a high degree. For example, if the regulation is a stop sign, the determination unit 108 determines the level of compliance to be "excellent" the longer the speed of the reversing forklift truck TG1 in the detection range AR or the forward detection range AR' is at 0 km / h. For example, if the regulation is a slow-down sign, the determination unit 108 determines the level of compliance to be "excellent" the slower the speed of the reversing forklift truck TG1 in the detection range AR or the forward detection range AR' is below a predetermined speed or the smaller the acceleration of the forklift truck TG1 is. The determination result generating unit 110 generates the determination result information 155 by further including information indicating the degree of compliance in the determination result information 155. With this configuration, the determination result generating unit 110 can refer to the determination result information 155 to further refer to the degree of compliance of the forklift truck TG1 that complies with the regulations.

[0169] The degree of violation may be expressed in three stages or four or more stages other than two stages. Furthermore, the determination unit 108 may determine the degree of compliance in multiple stages instead of (or in addition to) the degree of violation.

[0170] Instead of (or in addition to) a configuration in which the size of the symbol image SY is changed depending on the number of times the violation event EV1 has occurred, the presentation unit 111 may flash the symbol image SY superimposed on the icon ICN of the imaging unit 21 that has captured an image of a regulated location where a large number of violation events EV1 have occurred.

[0171] The presentation unit 111 may change the display mode of the symbol image SY based on the number of times the compliance event EV2 has occurred instead of (or in addition to) the number of times the violation event EV1 has occurred. For example, the presentation unit 111 may enlarge, compared to a normal size, the symbol image SY to be superimposed on the icon ICN of the imaging unit 21 that has captured an image of a regulated location where a large number of compliance events EV2 have occurred during a period specified by the pull-down lists PD1 to PD2. Furthermore, the presentation unit 111 may blink the symbol image SY to be superimposed on the icon ICN of the imaging unit 21 that has captured an image of a regulated location where a large number of compliance events EV2 have occurred.

[0172] According to this configuration, the presentation unit 111 can present information indicating the number of violations or the degree of violations to the user viewing the confirmation screen IMb in an intuitive and easy-to-understand manner. The presenting unit 111 may change the display mode of the symbol image SY based on the degree of violation indicated in the determination result information 155. For example, on the confirmation screen IMb representing a record of the violation event EV1, the presenting unit 111 may make the symbol image SY superimposed on the icon ICN of the image capture unit 21 corresponding to the record larger than the normal size as the information indicating the degree of violation of the record is "extremely bad" or the proportion of "extremely bad" is higher. Furthermore, on the confirmation screen IMb representing a record of the violation event EV1, the presenting unit 111 may make the symbol image SY superimposed on the icon ICN of the image capture unit 21 corresponding to the record flash as the information indicating the degree of violation of the record is "extremely bad" or the proportion of "extremely bad" is higher.

[0173] In the above description, the control unit 100 includes the video generating unit 109, but this is not limiting. The control unit 100 does not need to include the video generating unit 109. In this case, the information processing device 10 does not generate the violation video MV1 and the compliance video MV2 based on the determination result of the determination unit 108. Furthermore, steps S216 to S226 of the process shown in FIG. 8 are omitted.

[0174] Although the above description has been given of the case where the information processing device 10 generates the determination result information 155 based on the determination result of the determination unit 108, this is not limitative. The information processing device 10 does not need to generate the determination result information 155. In this case, the control unit 100 does not include the determination result generation unit 110, and the process of step S228 in the process shown in FIG. 8 may be omitted.

[0175] Although the above description has been given of the case where the control unit 100 includes the presenting unit 111, the presenting unit 111 may not be included. In this case, the information processing device 10 does not provide the user of the management device 20 with a confirmation screen IMb based on the determination result information 155. Furthermore, the control unit 100 does not execute the process shown in FIG. 9.

[0176] When the information processing device 10 determines that a violation event EV1 has occurred, it may notify the forklift truck TG1 that the violation event EV1 has occurred. In this case, a notification unit is provided near the detection range AR and the forward detection range AR'. The notification unit may be, for example, a speaker that notifies by sound or a rotating light that notifies by light. The control unit 100 also includes a notification control unit. When the determination unit 108 determines that a violation event EV1 has occurred, the notification control unit identifies the detection range AR in which the forklift truck TG1 to be determined is located. Specifically, the notification control unit identifies the image capture unit 21 that generated the video MV that was the source data for the determination process of the determination unit 108, and identifies the detection range AR captured by the identified image capture unit 21. The notification control unit notifies the forklift truck TG1 that the violation event EV1 has occurred via the notification unit corresponding to the identified detection range AR. According to this configuration, the notification control unit notifies the forklift TG1 located within the detection range AR or in the vicinity of the detection range AR or the forward detection range AR' that a violation event EV1 has occurred, thereby alerting the occupant or other people if the forklift TG1 is a vehicle.

[0177] The notification unit may be provided in the forklift TG1 instead of (or in addition to) being provided near the detection range AR or the forward detection range AR'. Specifically, the notification unit may be provided in, for example, the driver's seat of the forklift TG1. In this case, when the determination unit 108 determines that a violation event EV1 has occurred, the notification control unit identifies the moving object identification information of the forklift TG1 being determined. The process of identifying the moving object identification information is similar to the process executed by the determination result generation unit 110 described above, and therefore, description thereof will be omitted. Based on the identified moving object identification information, the notification control unit notifies the forklift TG1 associated with the identified moving object identification information that the violation event EV1 has occurred. With this configuration, the notification control unit notifies the forklift TG1 that caused the violation event EV1 that the violation event EV1 has occurred, thereby alerting the occupant or other people, if the forklift TG1 is a vehicle.

[0178] When the notification control unit performs the notification, the video MV is constantly transmitted by the management device 20 to the server device 30 or the information processing device 10. The processing of the flowchart shown in FIG. 8 is executed immediately when the server device 30 or the information processing device 10 receives the video MV. The processing by the notification control unit is executed between step S218 and step S220 or step S224 of the processing shown in FIG. 8. With this configuration, the notification control unit can perform the notification at a timing when the forklift truck TG1 that caused the violation event EV1 is closer to the detection range AR. When the video MV is constantly transmitted to the information processing device 10, the information processing device 10 includes a storage unit.

[0179] Instead of generating the violation video MV1 and the compliance video MV2, the determination result generating unit 110 may generate link information that associates the date and time when the violation event EV1 occurred with the playback time of the video MV at that date and time, and include the link information in the determination result information 155. With this configuration, the user of the management device 20 can easily check the video at the time when the violation event EV1 occurred by checking the determination result information 155.

[0180] The management device 20 may extract a portion of the video MV that shows the forklift TG1 and transmit it to the server device 30 or the information processing device 10. With this configuration, when the position identification unit 102 identifies the position of the forklift TG1 from the video MV, the number of images IMa that make up the video MV is reduced, thereby reducing the processing load.

[0181] The management device 20 may execute part of the processing executed by the information processing device 10. For example, the management device 20 may execute the processing executed by the determination unit 101 among the processing executed by the information processing device 10. In this case, the management device 20 transmits information indicating the detection range AR and information indicating the reference direction DR to the information processing device 10. The determination unit 108 performs determination processing based on the information indicating the detection range AR and the information indicating the reference direction DR received from the management device 20.

[0182] The detection range AR does not necessarily have to be associated with the reference direction DR. In this case, the information processing device 10 or the management device 20 does not need to execute the process of determining the reference direction DR corresponding to the detection range AR. Here, the travel direction of the forklift TG1 may be uniquely determined rather than bidirectionally determined along the travel route. In this case, at a location where a restriction is imposed on the travel route, only the forklift TG1 traveling in the travel direction to which the restriction applies will travel. Therefore, the determination unit 108 does not need to execute the process of determining whether the travel direction identified by the travel direction identification unit 103 matches the reference direction DR determined by the determination unit 101. In this case, the determination unit 108 omits the process of step S216 in the process of the flowchart shown in FIG. 8.

[0183] The information processing device 10 may perform at least one of a process related to a violation of a regulation and a process related to compliance with a regulation. A restriction other than a stop or slow-down may be applied to the travel route as a predetermined behavior. In this case, the determination unit 108 executes a process for determining whether the forklift truck TG1 moving backward appropriately performs a predetermined operation corresponding to the restriction other than a stop or slow-down within the detection range AR defined on the travel route. The determination unit 108 also executes a process for determining whether the forklift truck TG1 moving forward appropriately performs a predetermined operation corresponding to the restriction other than a stop or slow-down within the forward detection range AR' defined on the travel route.

[0184] The technical ideas that can be understood from the above-described embodiment and modified examples will be described. [Aspect 1] A computer presets a detection range corresponding to a restriction position of the forklift shown in an image generated by capturing an image of the path of the forklift using an imaging unit provided at a predetermined position capable of capturing the image of the path of the forklift, identifies the direction of travel and orientation of the forklift shown in the video based on a time-series video of the images, identifies the main body of the forklift excluding the forks shown in the images constituting the video, and identifies whether the forklift moving toward the restriction position is moving forward or backward based on the identified orientation and direction of travel of the forklift, and performs a detection process for the forklift identified as moving forward. a forward detection range different from the predetermined detection range for the images constituting the video, and based on the video, identifying the behavior of the forklift shown in the video; determining that a violation event has occurred if the forklift is not performing a predetermined behavior in the video from when at least a part of the main body enters the detection range to when it exits the detection range when the forklift is moving backward; and determining that a violation event has occurred if the forklift is not performing a predetermined behavior in the video from when at least a part of the main body enters the forward detection range to when it exits the forward detection range when the forklift is moving forward.

[0185] [Aspect 2] The information processing method according to aspect 1, wherein the predetermined behavior includes the forklift stopping temporarily or moving slowly. [Aspect 3] An information processing method described in [Aspect 1] or [Aspect 2], in which a reference direction corresponding to the detection range and the forward detection range is set in advance, and when the traveling direction entering the detection range matches the reference direction of the detection range, it is determined whether or not the violation event has occurred, and when the traveling direction entering the forward detection range matches the reference direction, it is determined whether or not the violation event has occurred.

[0186] [Aspect 4] An information processing method described in [Aspect 3], which determines multiple detection ranges on the movement path and reference directions corresponding to each of the multiple detection ranges, and determines whether or not a violation event has occurred for each of the multiple detection ranges.

[0187] [Aspect 5] An information processing method described in [Aspect 3] or [Aspect 4], in which, when the direction of travel and the reference direction match, if the time during which the speed of the forklift in the detection range or the forward detection range remains below a speed threshold is less than a predetermined time, it is determined that the violation event has occurred.

[0188] [Aspect 6] An information processing method as described in [Aspect 3] or [Aspect 4], in which if the direction of travel and the reference direction match and the speed of the forklift does not reach a speed threshold in the detection range or the forward detection range, it is determined that the forklift is not stopping or slowing down.

[0189] [Aspect 7] An information processing method described in any one of [Aspect 3] to [Aspect 6], in which the reference direction is the direction connecting the detection range and the entry range into which the forklift enters after passing through the detection range, and when the identified position of the forklift indicates that it has moved from the detection range to the entry range, it is determined that the direction of travel and the reference direction match.

[0190] [Aspect 8] An information processing method described in any one of [Aspect 1] to [Aspect 7], wherein the forward detection range is a range obtained by moving the detection range a predetermined distance in the direction opposite to the reference direction.

[0191] [Aspect 9] An information processing method described in any one of [Aspect 1] to [Aspect 8], which identifies, for each image constituting the video, a first position, which is the position of the balance weight of the forklift on the image, and a second position, which is the position of the mast of the forklift on the image, and identifies the orientation of the forklift based on the positional relationship between the first position and the second position.

[0192] [Aspect 10] An information processing method described in [Aspect 9], in which the first position and the second position are identified based on information output by inputting each of the images that make up the video into a trained model that outputs the first position and the second position when the images that make up the video are input.

[0193] [Aspect 11] An information processing method described in any one of [Aspect 1] to [Aspect 10], which, when it is determined that the violation event has occurred, generates a violation video by extracting a portion of the video that includes the image of the scene where the violation event occurred.

[0194] [Aspect 12] An information processing method described in any one of [Aspect 1] to [Aspect 11], which, when it is determined that the violation event has occurred, generates judgment result information indicating the judgment result corresponding to the violation event.

[0195] [Aspect 13] An information processing method as described in [Aspect 12], which identifies the forklift related to the violation event based on identification information corresponding to the forklift, and generates the judgment result information by further including moving object identification information that can identify the identified forklift.

[0196] [Aspect 14] An information processing method described in [Aspect 12] or [Aspect 13], wherein the judgment result information includes imaging unit identification information that can identify the imaging unit, and a confirmation screen is generated that displays the occurrence status of the violation event for each imaging unit on the map image based on a map image showing the specified position where the imaging unit corresponding to the imaging unit identification information is installed and the movement route, and the judgment result information.

[0197] [Aspect 15] An information processing method described in any one of [Aspect 1] to [Aspect 14], which notifies the forklift that the violation event has occurred if it is determined that the violation event has occurred.

[0198] [Aspect 16] An information processing method described in any one of [Aspect 1] to [Aspect 15], in which, based on the identified position of the forklift, if at least a portion of the forklift enters the detection range and then behaves in a predetermined manner within the detection range, it is determined that a compliance event has occurred, and if at least a portion of the forklift enters the forward detection range and then behaves in a predetermined manner within the forward detection range, it is determined that the compliance event has occurred.

[0199] [Aspect 17] An information processing method described in [Aspect 16], in which, when it is determined that the compliance event has occurred, a compliance video is generated by extracting a portion of the video that includes the image of the scene where the compliance event occurred.

[0200] [Aspect 18] An information processing method according to [Aspect 17], which generates judgment result information indicating a judgment result corresponding to the compliance event. [Aspect 19] An information processing method described in any one of [Aspect 1] to [Aspect 18], wherein at least a portion of the forklift is the front end of the forklift in the direction of travel on the image.

[0201] [Aspect 20] An information processing method described in any one of [Aspect 1] to [Aspect 19], in which the position of the forklift is identified based on the information output by inputting each of the images that make up the video into a trained model that outputs the position of the forklift using a bounding box when the images that make up the video are input, and at least a portion of the forklift is the lower front end in the direction of travel of the bounding box that represents the forklift on the image.

[0202] [Aspect 21] An information processing method described in any one of [Aspect 1] to [Aspect 20], further determining the severity of the violation event based on the manner in which the forklift moves within the detection range or the forward detection range.

[0203] [Aspect 22] An information processing system including an imaging unit provided at a predetermined position capable of imaging a path of travel of a forklift, one or more management devices that store a video in a cloud server device in which images of the path of travel are captured by the imaging unit and generated as a chronological sequence of consecutive images, and an information processing device that executes various processes based on the video stored on the server device, wherein the information processing device includes a traveling direction specifying unit that specifies the traveling direction of the forklift shown in the video based on the video, a position specifying unit that specifies the main body, excluding the fork, of the forklift shown in the images constituting the video, and an orientation specifying unit that specifies the orientation of the forklift based on the video, and a function that specifies whether the forklift moving toward the restriction position is moving forward or backward based on the specified orientation of the forklift and the traveling direction of the forklift. an application unit that sets a forward detection range that is different from the predetermined detection range for the images that constitute the video in which the forklift is identified as moving forward; a behavior identification unit that identifies the behavior of the forklift shown in the video based on the video; and a determination unit that, when the forklift is moving backward, determines that a violation event has occurred if the forklift does not perform a predetermined behavior in the video from when at least a part of the main body enters the forward detection range to when it exits the forward detection range, and, when the forklift is moving forward, determines that the violation event has occurred if the forklift does not perform a predetermined behavior in the video from when at least a part of the main body enters the forward detection range to when it exits the forward detection range.

[0204] [Aspect 23] An information processing system including an imaging unit provided at a predetermined position capable of imaging a path of travel of a forklift, one or more management devices that store in an information processing device a video of a chronological sequence of images generated by the imaging unit capturing the path of travel, and an information processing device that executes various processes based on the video, wherein the information processing device includes a traveling direction specifying unit that specifies the traveling direction of the forklift shown in the video based on the video, a position specifying unit that specifies the main body, excluding the fork, of the forklift shown in the images that constitute the video, a direction specifying unit that specifies the orientation of the forklift based on the video, and a direction specifying unit that specifies whether the forklift moving toward the restriction position is moving forward or backward based on the specified orientation of the forklift and the traveling direction of the forklift. an application unit that sets a forward detection range that is different from the predetermined detection range for the images that constitute the video in which the forklift is identified as moving forward; a behavior identification unit that identifies the behavior of the forklift shown in the video based on the video; and a determination unit that, when the forklift is moving backward, determines that a violation event has occurred if the forklift does not perform a predetermined behavior in the video from when at least a part of the main body enters the detection range to when it exits the detection range, and, when the forklift is moving forward, determines that the violation event has occurred if the forklift does not perform a predetermined behavior in the video from when at least a part of the main body enters the forward detection range to when it exits the forward detection range.

[0205] [Aspect 24] A trained model that, when an imaging unit capable of capturing an image of a forklift's travel path inputs an image generated by capturing the travel path, outputs a first position, which is the position of the forklift's balance weight on the image, and a second position, which is the position of the forklift's mast on the image. [Explanation of symbols]

[0206] 1...information processing system, 10...information processing device, 20...management device, 21, 21a, 21b, 21c, 21d, 21e, 21f, 21g...imaging unit, 30...server device, 102...position identification unit, 103...traveling direction identification unit, 104...orientation identification unit, 105...movement direction identification unit, 106...application unit, 107...behavior identification unit, 108...judgment unit, 151...trained model, 155...judgment result information, AR...detection range, AR'...forward detection range, ARa...entry range, BX, BXa, BXc, BXd...bounding box, d...predetermined distance, DR...reference direction, EV1...violation event, EV2...compliance event, IMa...image, IMb...confirmation screen, IMd...map image, JP...lower front end, MV...video, MV1...violation video, MV2...compliance video, TG...moving object, TG1...forklift.

Claims

1. The computer a detection range corresponding to a restriction position of the forklift truck shown in an image generated by capturing an image of the travel path of the forklift truck using an imaging unit provided at a predetermined position capable of capturing an image of the travel path of the forklift truck; Identifying the direction of travel and orientation of the forklift shown in the video based on the video of consecutive images in time series; Identifying a main body portion excluding a fork in the forklift shown in the image constituting the video; Identifying whether the forklift moving toward the restriction position is moving forward or backward based on the identified orientation of the forklift and the traveling direction of the forklift; a forward detection range that is different from the predetermined detection range for the images that constitute the video in which the forklift is identified as moving forward; Identifying a behavior of the forklift shown in the video based on the video; When the forklift is moving backward, if the forklift does not exhibit a predetermined behavior in the video from when at least a part of the main body enters the detection range to when it leaves the detection range, it is determined that a violation event has occurred; When the forklift is moving forward, if the forklift does not exhibit a predetermined behavior in the video from when at least a part of the main body enters the forward detection range to when it leaves the forward detection range, it is determined that the violation event has occurred. Information processing methods.

2. The predetermined behavior includes stopping or slowing down the forklift. The information processing method according to claim 1 .

3. a reference direction associated with the detection range and the forward detection range is set in advance; determining whether or not the violation event has occurred when the traveling direction entering the detection range matches the reference direction of the detection range, and determining whether or not the violation event has occurred when the traveling direction entering the forward detection range matches the reference direction; The information processing method according to claim 1 .

4. determining a plurality of detection ranges and the reference directions respectively associated with the plurality of detection ranges on the movement path; determining whether or not the violation event has occurred for each of the plurality of detection ranges; The information processing method according to claim 3 .

5. When the traveling direction and the reference direction match, if the time during which the speed of the forklift in the detection range or the forward detection range continues to be equal to or less than a speed threshold is shorter than a predetermined time, it is determined that the violation event has occurred. The information processing method according to claim 3 .

6. When the traveling direction and the reference direction match, and the speed of the forklift does not reach a speed threshold in the detection range or the forward detection range, it is determined that the forklift is not stopping or slowing down. The information processing method according to claim 3 .

7. the reference direction is a direction connecting the detection range and an entry range into which the vehicle enters after passing through the detection range, If the identified position of the forklift indicates that it has moved from the detection range to the entry range, it is determined that the traveling direction matches the reference direction. The information processing method according to claim 3 .

8. the forward detection range is a range obtained by moving the detection range by a predetermined distance in a direction opposite to the reference direction, The information processing method according to claim 3 .

9. for each of the images constituting the video, a first position which is a position of a balance weight of the forklift on the image and a second position which is a position of a mast of the forklift on the image are identified, and the orientation of the forklift is identified based on a positional relationship between the first position and the second position. The information processing method according to claim 1 .

10. Identifying the first position and the second position based on information output by inputting the images constituting the video into a trained model that outputs the first position and the second position when the images constituting the video are input. The information processing method according to claim 9.

11. generating a violation video by extracting a part of the video that includes the image of the scene where the violation event occurred, when it is determined that the violation event has occurred; The information processing method according to claim 1 .

12. generating determination result information indicating a determination result corresponding to the violation event when it is determined that the violation event has occurred; The information processing method according to claim 1 .

13. Identifying the forklift involved in the violation event based on identification information corresponding to the forklift; generating the determination result information by further including moving object identification information that can identify the specified forklift; The information processing method according to claim 12.

14. the determination result information includes imaging unit identification information that can identify the imaging unit, generating a confirmation screen that displays the occurrence status of the violation event for each imaging unit on the map image based on a map image that indicates the predetermined position where the imaging unit corresponding to the imaging unit identification information is provided and the movement route, and the determination result information; The information processing method according to claim 12.

15. If it is determined that the violation event has occurred, notify the forklift that the violation event has occurred. The information processing method according to claim 1 .

16. Based on the identified positions of the forklifts, if at least a part of the forklifts enters the detection range and then behaves in a predetermined manner within the detection range, it is determined that a compliance event has occurred, and if at least a part of the forklifts enters the forward detection range and then behaves in a predetermined manner within the forward detection range, it is determined that the compliance event has occurred. The information processing method according to claim 1 .

17. generating a compliance video by extracting a part of the video including the image of the scene where the compliance event occurred, when it is determined that the compliance event has occurred; 17. The information processing method according to claim 16.

18. generating judgment result information indicating a judgment result corresponding to the compliance event; 18. The information processing method according to claim 17.

19. At least a part of the forklift is a front end portion of the forklift in the traveling direction on the image. The information processing method according to claim 1 .

20. Identifying the position of the forklift based on information output by inputting each of the images constituting the video into a trained model that outputs the position of the forklift using a bounding box when the images constituting the video are input; The at least part of the forklift is a lower front end portion in the traveling direction of the bounding box representing the forklift on the image. The information processing method according to claim 1 .

21. further determining the severity of the violation event based on a movement pattern of the forklift in the detection range or the forward detection range; The information processing method according to claim 1 .

22. An information processing system comprising: an imaging unit provided at a predetermined position capable of capturing an image of a travel path of a forklift; one or more management devices that store, in a server device on a cloud, a video in which images generated by the imaging unit capturing the travel path are successively displayed in time series; and an information processing device that executes various processes based on the video stored in the server device, The information processing device includes: a traveling direction identification unit that identifies a traveling direction of the forklift shown in the video based on the video; a position specifying unit for specifying a main body portion excluding a fork in the forklift shown in the image constituting the video; an orientation identification unit that identifies an orientation of the forklift based on the video; a movement direction identification unit that identifies whether the forklift moving toward the restriction position is moving forward or backward based on the identified orientation of the forklift and the traveling direction of the forklift; an application unit that sets a forward detection range corresponding to the restriction position for the image that constitutes the video in which the forklift is identified as moving forward, the forward detection range being different from the predetermined detection range; a behavior identification unit that identifies a behavior of the forklift shown in the video based on the video; a determination unit that determines that a violation event has occurred if the forklift does not behave in a predetermined manner in the video from when at least a part of the main body enters the forward detection range until when it leaves the forward detection range when the forklift is moving backward, and that determines that a violation event has occurred if the forklift does not behave in a predetermined manner in the video from when at least a part of the main body enters the forward detection range until it leaves the forward detection range when the forklift is moving forward, Information processing system.

23. An information processing system comprising: an imaging unit provided at a predetermined position capable of capturing an image of a travel path of a forklift; one or more management devices that store, in an information processing device, a video in which images generated by the imaging unit capturing the travel path are successively displayed in time series; and an information processing device that executes various processes based on the video, The information processing device includes: a traveling direction identification unit that identifies a traveling direction of the forklift shown in the video based on the video; a position specifying unit for specifying a main body portion excluding a fork in the forklift shown in the image constituting the video; an orientation identification unit that identifies an orientation of the forklift based on the video; a movement direction identification unit that identifies whether the forklift moving toward the restriction position is moving forward or backward based on the identified orientation of the forklift and the traveling direction of the forklift; an application unit that sets a forward detection range corresponding to the restriction position for the image that constitutes the video in which the forklift is identified as moving forward, the forward detection range being different from the predetermined detection range; a behavior identification unit that identifies a behavior of the forklift shown in the video based on the video; a determination unit that determines that a violation event has occurred if the forklift does not behave in a predetermined manner in the video from when at least a part of the main body enters the forward detection range until when it leaves the forward detection range when the forklift is moving backward, and that determines that a violation event has occurred if the forklift does not behave in a predetermined manner in the video from when at least a part of the main body enters the forward detection range until it leaves the forward detection range when the forklift is moving forward, Information processing system.

24. When an image capturing unit capable of capturing an image of a movement path of a forklift truck receives an image generated by capturing the movement path, the unit outputs a first position, which is a position of a balance weight of the forklift truck on the image, and a second position, which is a position of a mast of the forklift truck on the image. Trained model.

Images