Work monitoring device, work monitoring method, and work monitoring program

Abstract

This allows monitoring whether or not the locking mechanism on the trailer is being switched. [Solution] A work monitoring device installed at a container terminal for loading and unloading containers for maritime transport, comprising: a camera that photographs within a field of view set in an area where work related to locking the container loaded on a trailer is performed; an object detection unit configured to detect at least the container loaded on the trailer and the driver in the image obtained by the camera, using a detection model built in advance by machine learning; and a work execution determination unit configured to determine whether or not work related to locking the container has been performed based on the detection result of the container and the detection result of the driver in the image.

Description

Technical Field

[0001] The present disclosure relates to a work monitoring device, a work monitoring method, and a work monitoring program.

Background Art

[0002] Patent Document 1 discloses a container trailer capable of carrying a container. This container trailer is provided with twist locks for fixing the container.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a trailer loaded with a container travels on land, it is necessary to fix the container with a locking mechanism such as a twist lock. On the other hand, at a container terminal, when unloading the container from the trailer, it is necessary to release the fixing of the container by the locking mechanism. The switching between the fixed state and the released state by the above locking mechanism is manually performed by the driver of the trailer, and if such a switching operation is forgotten, it can cause a major problem.

[0005] The present disclosure provides a work monitoring device, a work monitoring method, and a work monitoring program capable of monitoring whether or not to execute the switching operation of a locking mechanism provided on a trailer.

Means for Solving the Problems

[0006] [1] A work monitoring device installed at a container terminal for loading and unloading containers for maritime transport, comprising: a camera that photographs within a field of view set in an area where work related to locking the containers loaded on a trailer is performed; an object detection unit configured to detect at least the containers loaded on the trailer and the driver in the images obtained by the camera, using a detection model built in advance by machine learning; and a work execution determination unit configured to determine whether or not work related to locking the containers has been performed based on the detection results of the containers and the driver in the images.

[0007] [2] The trailer is provided with a twist lock as a mechanism for securing the container, and the work related to locking the container loaded on the trailer is the work of securing the container with the twist lock, or the work of releasing the container from being secured with the twist lock, as described in [1] above.

[0008] [3] The work monitoring device according to [1] or [2] above, wherein the target detection unit is configured to further detect the trailer, and the target detection unit is configured to identify the color of the trailer when the trailer is detected, and to identify the color of the container when the container is detected.

[0009] [4] The work monitoring device according to any one of [1] to [3] above, wherein the work execution determination unit is configured to further determine whether or not the driver is wearing a helmet when the driver is detected.

[0010] [5] A work monitoring method performed at a container terminal for loading and unloading containers for maritime transport, comprising: taking photographs with a camera within a field of view set in an area where work related to locking the container loaded on a trailer is performed; using a detection model built in advance by machine learning to detect at least the container loaded on the trailer and the driver in the image obtained by the camera; and determining whether or not work related to locking the container has been performed based on the detection result of the container and the detection result of the driver in the image.

[0011] [6] A work monitoring program that causes a computer to perform the following steps: acquire an image obtained by taking a picture with a camera within a field of view set in an area of ​​a container terminal for loading and unloading containers for maritime transport where work related to locking the containers loaded on trailers is performed; use a detection model built in advance by machine learning to detect at least the containers loaded on the trailers and the driver in the image; and determine whether or not work related to locking the containers has been performed based on the detection results of the containers and the driver in the image. [Effects of the Invention]

[0012] According to this disclosure, a work monitoring device, a work monitoring method, and a work monitoring program are provided that can monitor whether or not a switching operation of a locking mechanism provided on a trailer is being performed. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a schematic diagram illustrating a work monitoring device and trailer. [Figure 2] Figure 2 is a schematic plan view illustrating a container terminal where work monitoring equipment is installed. [Figure 3] Figure 3 is a schematic diagram illustrating the camera arrangement and the mechanical configuration of the computing unit. [Figure 4] FIG. 4 is a schematic diagram illustrating an image obtained by photographing with a camera. [Figure 5] FIG. 5 is a schematic diagram illustrating an image obtained by photographing with a camera. [Figure 6] FIG. 6 is a schematic diagram illustrating the hardware configuration of an arithmetic unit. [Figure 7] FIG. 7 is a flowchart illustrating a series of processes executed by an arithmetic unit. [Figure 8] FIG. 8 is a flowchart illustrating a determination flow of whether work is being executed. [Figure 9] FIG. 9(a) is a schematic diagram illustrating the discrimination result of the driver's posture. FIG. 9(b) is a schematic diagram illustrating the detection result of the helmet worn by the driver.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an embodiment will be described with reference to the drawings. In the description, the same reference numerals are given to the same elements or elements having the same function, and duplicate descriptions are omitted.

[0015] [Operation Monitoring Device] In FIG. 1, an operation monitoring device according to an embodiment and a trailer on which work of a monitoring target by the operation monitoring device is performed are schematically shown. In FIG. 2, a plan view of a container terminal where the operation monitoring device is installed is schematically shown. The operation monitoring device 1 shown in FIG. 1 is a device (system) that monitors whether the work related to the lock of the container 110 loaded on the trailer 100 has been executed. The container 110 is used for maritime transport of goods and is formed in a rectangular parallelepiped shape.

[0016] The trailer 100 on which the work of the monitoring target by the operation monitoring device 1 is performed is also referred to as a container trailer and is a vehicle that transports the container 110 containing goods by land transportation. The trailer 100 is, for example, a towed vehicle and includes a truck head 102, a chassis 104, and a plurality of locking mechanisms 106.

[0017] The truck head 102, which is also referred to as a trailer head or a tractor head, is provided with an engine, a driver's seat, and other components. The chassis 104, which is also referred to as a trailer chassis, is detachably connected to the truck head 102. The chassis 104 has, for example, a frame structure for mounting (supporting) the container 110. The plurality of locking mechanisms 106 are mechanisms (members) for fixing the container 110 to the trailer 100 and are provided on the chassis 104. The plurality of locking mechanisms 106 are arranged to correspond to the four corners of the bottom of the container 110 placed on the chassis 104. The locking mechanism 106 is, for example, a twist lock. Corner castings into which twist locks are inserted may be provided at each corner of the bottom of the container 110.

[0018] The container terminal 150 shown in FIG. 2 is a place (facility) for loading and unloading the sea - transportation container 110. In FIG. 2, "S" represents the sea or ocean, and "G" represents land or ground. The container terminal 150, which is also referred to as a container yard, functions as a place connecting sea transportation and land transportation. The container terminal 150 also functions as a place for temporarily storing the container 110. The container handling operations at the container terminal 150 include the operation of unloading the container 110 from the ship 180 and the operation of loading the container 110 onto the ship 180.

[0019] The container terminal 150 includes, for example, a storage area 152, an exit gate 162, an entrance gate 164, a fixing work area 154, and a releasing work area 156. The storage area 152 is an area where a large number of containers 110 temporarily stored in the container terminal 150 are arranged side by side. In the storage area 152, the operation of loading the container 110 onto the trailer 100 is possible, and the operation of loading and unloading the container 110 from the trailer 100 is possible.

[0020] Exit gate 162 is a gate that a trailer 100 loaded with containers 110 in storage area 152 must pass through before leaving container terminal 150. Entrance gate 164 is a gate that a trailer 100 loaded with containers 110 must pass through before entering storage area 152 within container terminal 150. At each of the exit gate 162 and entrance gate 164, for example, individual information of the trailer 100 (vehicle number, etc.), information about the driver of the trailer 100, information about the containers 110 loaded on the trailer 100 (for example, container identification information, cargo information, etc.), and information about the date and time of passage are recorded. This information may be managed by a software system called "TOS" (Terminal Operation System). Hereafter, unless otherwise specified, "container 110" means the container as it is loaded on the trailer 100.

[0021] The securing work area 154 is the area where work (hereinafter referred to as "securing work") is performed to secure the container 110 to the chassis 104 using locking mechanisms 106 such as twist locks before departing from the container terminal 150. It is necessary that the securing work is performed securely to prevent the container 110 from falling when traveling on public roads. The securing work (locking work) is performed by the trailer driver manually operating multiple locking mechanisms 106. For example, the driver rotates the pin of a twist lock by 90° by operating a handle or lever to switch it from the unlocked state to the locked state. The driver operates each of the multiple (e.g., four) twist locks to switch them to the locked state.

[0022] The fixed work area 154 is located near the exit gate 162. The fixed work area 154 may include an area where the trailer 100 waits before passing through the exit gate 162. The fixed work area 154 may also include an area where the trailer 100 can stop immediately after passing through the exit gate 162. In this disclosure, the fixed work area 154 includes not only areas explicitly designated as fixed work areas by the container terminal 150 administrator, but also areas where fixed work can be performed by the driver. In Figure 2, the arrows labeled "Ro" indicate the route and direction of travel from the storage area 152, through the exit gate 162 and the fixed work area 154, to the exit of the container terminal 150.

[0023] The unlocking work area 156 is an area where work (hereinafter referred to as "unlocking work") is performed to release the locking mechanism 106, such as a twist lock, from the container 110 before it enters the storage area 152 of the container terminal 150. When the container 110 is unloaded from the trailer 100, the container 110 is lifted by a crane. Therefore, from the standpoint of safely unloading the container 110, it is necessary that the unlocking work is carried out reliably. The unlocking work (unlocking work) is performed by the driver manually operating multiple locking mechanisms 106, similar to the locking work.

[0024] The unloading area 156 is located near the entrance gate 164. The unloading area 156 may include an area where the container 110 can stop before passing through the entrance gate 164. In this disclosure, the unloading area 156 includes not only the area explicitly designated as an area for unloading by the administrator of the container terminal 150, but also the area where unloading can be performed by the driver. In Figure 2, the arrow labeled "Ri" indicates the route and direction of travel from outside the container terminal 150, through the unloading area 156 and the entrance gate 164, into the storage area 152.

[0025] Returning to Figure 1, the work monitoring device 1 is installed in the container terminal 150 as illustrated above. The work monitoring device 1 monitors whether at least one of the fixing work and the release work is performed. That is, the work monitoring device 1 monitors whether either the fixing work or the release work, or both the fixing work and the release work, are performed. The work monitoring device 1 comprises a calculation device 10 and a camera 30.

[0026] The arithmetic unit 10 is a device that performs calculations to monitor whether or not at least one of the fixing operation and the release operation is performed, based on information obtained from the imaging device 30. The arithmetic unit 10 is composed of one or more computers. The computers that make up the arithmetic unit 10 may be personal computers, tablet computers (tablet terminals), smartphones, wearable devices, workstations, server computers, or general-purpose computers. If the arithmetic unit 10 is composed of two or more computers, these two or more computers may be connected to each other in a way that allows them to communicate with one another. The arithmetic unit 10 may be connected to other devices in a way that allows them to communicate with one another via wireless, wired, or communication network.

[0027] The imaging device 30 is equipped with one or more cameras. The cameras in the imaging device 30 are capable of imaging within a field of view set in the area where the work related to locking the container 110 is performed. The cameras in the imaging device 30 may be fixed in a predetermined position and imaging within the field of view. Alternatively, the cameras in the imaging device 30 may be mounted on a mobile device such as a drone and imaging within the field of view. In the following, the details of the imaging device 30 and the computing device 10 will be described using as an example a case in which the work monitoring device 1 monitors whether the locking work has been performed and the cameras included in the imaging device 30 have a fixed field of view.

[0028] As illustrated in Figure 3, the imaging device 30 may include a first camera 32 and a second camera 34. Each of the first camera 32 and the second camera 34 is a camera that captures within a field of view (shooting range) set in the fixed work area 154. The field of view of the first camera 32 is set to a part of the area of ​​the fixed work area 154 before passing through the exit gate 162. The first camera 32 may be installed at the exit gate 162.

[0029] The field of view of the second camera 34 is set to cover a portion of the area after passing through the exit gate 162 within the fixed work area 154. The second camera 34 may also be installed at the exit gate 162. Both the first camera 32 and the second camera 34 may be digital cameras that focus visible light to generate image data. Both the first camera 32 and the second camera 34 may generate video data or a series of still images as image data.

[0030] The arithmetic unit 10 has the following functional components (referred to as "functional blocks" in this disclosure): a data acquisition unit 12, a target detection unit 14, a task execution determination unit 16, and a data recording unit 18. The processing performed by each of the data acquisition unit 12, the target detection unit 14, the task execution determination unit 16, and the data recording unit 18 corresponds to the processing performed by the arithmetic unit 10.

[0031] The data acquisition unit 12 is configured to acquire image data obtained by the cameras included in the imaging device 30. The data acquisition unit 12 acquires image data generated by the first camera 32 from the first camera 32, and acquires image data generated by the second camera 34 from the second camera 34. The data acquisition unit 12 may repeatedly acquire image data from the first camera 32 and the second camera 32 at predetermined acquisition timings.

[0032] The object detection unit 14 is configured to detect at least the container 110 and the driver (hereinafter referred to as "driver D") in the image obtained by the camera included in the imaging device 30, using a detection model built in advance by machine learning. The object detection unit 14 may be configured to detect the container 110, the truck head 102, and driver D in the image. The object detection unit 14 may repeat the process of detecting the object at predetermined detection timings. The detection timing may be set in accordance with the timing of acquiring images from the camera.

[0033] Figure 4 schematically illustrates image P1 obtained by the first camera 32, and Figure 5 schematically illustrates image P2 obtained by the second camera 34. The target detection unit 14 uses a trained detection model to detect the container 110 and the driver D contained in image P1. In addition to detection in image P1, the target detection unit 14 also uses a trained detection model to detect the container 110 and the driver D contained in image P2. The objects to be detected from image P1 or image P2 may include the truck head 102 in addition to the container 110 and the driver D. The detection of the truck head 102 is substantially equivalent to the detection of the trailer 100.

[0034] As shown in Figure 4, the object detection unit 14 detects a rectangular bounding box surrounding each type of object to be detected in image P1. In Figure 4, "Bt" represents the bounding box detected as truck head 102. "Bc" represents the bounding box detected as container 110. "Bd" represents the bounding box detected as driver D. Depending on the timing at which the image data related to image P1 is obtained (shooting timing), at least a part of container 110, truck head 102, and driver D may not be detected. In this case, the object detection unit 14 may determine that the object to be detected is not detected in image P1.

[0035] As shown in Figure 5, the object detection unit 14 detects a rectangular bounding box surrounding each type of object in image P2, similar to the detection of objects in image P1. The object detection unit 14 not only detects the presence of objects, but also identifies the type of detected object and determines the position of that object (bounding box). The object detection unit 14 may use the same detection model or different detection models for detection from image P1 and detection from image P2.

[0036] A pre-trained detection model is built in advance and stored in the computing unit 10. The detection model used by the target detection unit 14 is constructed to output the results of detecting objects within an image, categorized by type, in response to the image input. The specific machine learning algorithm used to construct the detection model is not particularly limited, but the detection model is constructed using machine learning, for example, a convolutional neural network (CNN).

[0037] The object detection unit 14 may identify the color of the truck head 102 when it is detected in image P1 or image P2. The object detection unit 14 may also identify the color of the container 110 when it is detected in image P1 or image P2. In one example, the object detection unit 14 identifies the color of the truck head 102 from the pixel values ​​of one or more representative points within the bounding box Bt representing the detection result of the truck head 102. Similarly, the object detection unit 14 identifies the color of the container 110 from the pixel values ​​of one or more representative points within the bounding box Bc. The object detection unit 14 may identify (classify) the color of the detected object from the pixel values ​​of the representative points into one of several dozen color names, including basic colors such as red, blue, green, and yellow. Identifying the color of the truck head 102 is substantially equivalent to identifying the color of the trailer 100.

[0038] The work execution determination unit 16 is configured to determine whether or not a locking operation (fixing operation) of the container 110 has been performed, based on the detection result of the container 110 and the detection result of the driver D in the image obtained by the camera included in the imaging device 30. For example, if the container 110 is detected in image P1, the work execution determination unit 16 determines whether or not a fixing operation has been performed on the container 110, based on the detection result of the container 110 in image P1 and the detection result of the driver D. For example, if the container 110 is detected in image P2, the work execution determination unit 16 determines whether or not a fixing operation has been performed on the container 110, based on the detection result of the container 110 in image P2 and the detection result of the driver D.

[0039] As described above, in order to secure the container 110 to the locking mechanism 106, the driver D must get out of the driver's seat and operate the locking mechanism 106 in the vicinity of the container 110. For example, if the driver D is not detected in the vicinity of the container 110 during the entire period in which the container 110 is detected, it is highly likely that the securing work has not been performed. On the other hand, if the driver D is detected in the vicinity of the container 110 for at least a portion of the period in which the container 110 is detected, it can be presumed that the securing work has been performed. A specific example of the determination method by the work execution determination unit 16 will be described later.

[0040] The data recording unit 18 records the determination result made by the work execution determination unit 16. When a container 110 is detected in image P1, the data recording unit 18 records the determination result of whether or not a fixing operation was performed on the container 110. At this time, the data recording unit 18 may also record the color identification result of the truck head 102 (trailer 100) in association with the above determination result. When a container 110 is detected in image P2, the data recording unit 18 records the determination result of whether or not a fixing operation was performed on the container 110. At this time, the data recording unit 18 may also record the color identification result of the truck head 102 (trailer 100) in association with the above determination result. The information recorded in the data recording unit 18 may be transferred to an external device of the arithmetic unit 10.

[0041] Figure 6 schematically shows the hardware configuration of the arithmetic unit 10. As shown in Figure 6, the arithmetic unit 10 has a circuit 50. The circuit 50 has a processor 51, a memory 52, a storage 53, a timer 54, an input / output port 55, and a communication port 56. The storage 53 is composed of one or more non-volatile memory devices such as flash memory or a hard disk. The storage 53 stores a work monitoring program that causes the computer to execute a work monitoring method described later. The memory 52 is composed of one or more volatile memory devices such as random access memory. The memory 52 temporarily stores programs loaded from the storage 53.

[0042] The processor 51 is composed of one or more computing devices such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The processor 51 constitutes the above-mentioned functional block by executing the work monitoring program loaded into the memory 52. ​​The calculation results by the processor 51 are temporarily stored in the memory 52. ​​The timer 54 measures the elapsed time by counting clock pulses. The input / output port 55 performs input and output of electrical signals to and from the imaging device 30, etc., in response to requests from the processor 51. The communication port 135 communicates with external devices of the work monitoring device 1 via wireless, wired, or network lines.

[0043] The arithmetic unit 10 is not necessarily limited to having each function configured by a program. For example, the arithmetic unit 10 may have at least some functions configured by a dedicated logic circuit or an ASIC (Application Specific Integrated Circuit) that integrates such circuits. The work monitoring program may be provided by being permanently recorded on a tangible recording medium such as a CD-ROM, DVD-ROM, or semiconductor memory. Alternatively, the work monitoring program may be provided via a communication network as a data signal superimposed on a carrier wave.

[0044] [Work Monitoring Method] Next, a work monitoring method performed using the work monitoring device 1 will be described. This work monitoring method is performed at a container terminal 150 that handles containers 110 for maritime transport. The work monitoring method includes, for example, a shooting step, an acquisition step, a detection step, a determination step, and a recording step.

[0045] The imaging process involves using the cameras of the imaging device 30 to image the area within the field of view set in the fixed work area 154 where the work related to locking the container 110 is performed. In the imaging process, for example, imaging is performed by the first camera 32 and the second camera 34. The acquisition process involves the computing device 10 (data acquisition unit 12) acquiring the image data generated by the execution of imaging in the imaging process.

[0046] The detection step is a step of detecting at least the container 110 and the driver D in the image obtained by the camera of the camera 30, using a detection model built in advance by machine learning. In the detection step, it may be determined using the detection model whether or not the image P1 contains the truck head 102, the container 110, and the driver D. In the detection step, it may be determined using the detection model whether or not the image P2 contains the truck head 102, the container 110, and the driver D. In the detection step, if the truck head 102 is detected, the color of the truck head 102 may be identified. In the detection step, if the container 110 is detected, the color of the container 110 may be identified. The detection step may be performed by the target detection unit 14 of the computing unit 10.

[0047] The determination step is a step of determining whether or not the securing work for the container 110 has been performed, based on the detection result of driver D. In the determination step, if the container 110 is detected in image P1, it is determined whether or not the securing work for the container 110 has been performed, based on the detection result of driver D in image P1. In the determination step, if the container 110 is detected in image P2, it is determined whether or not the securing work for the container 110 has been performed, based on the detection result of driver D in image P2. The determination step may also be performed by the work execution determination unit 16 of the arithmetic unit 10.

[0048] The recording step may record the result of the determination in the judgment step, specifically whether or not fixing work was performed on the detected container 110. In this case, the determination result may be recorded in association with the color identification result of the truck head 102 and the color identification result of the container 110. The recording step may be performed by the data recording unit 18 of the arithmetic unit 10.

[0049] Figure 7 is a flowchart illustrating the processing flow executed by the arithmetic unit 10 in the work monitoring method. The arithmetic unit 10 may perform the same processing flow in parallel for monitoring based on image data captured by the first camera 32 and monitoring based on image data captured by the second camera 34. The processing flow for monitoring based on image data captured by the first camera 32 will be described below, but the processing flow for monitoring based on image data captured by the second camera 34 will be executed similarly. In one example, the arithmetic unit 10 starts the processing flow based on instructions from the user of the work monitoring device 1. Alternatively, the arithmetic unit 10 may start the processing flow at an arbitrarily set time during the day.

[0050] First, the arithmetic unit 10 executes step S01. In step S01, for example, the data acquisition unit 12 outputs a signal to the first camera 32 to start taking pictures. This starts the acquisition of image P1 by the first camera 32. From there, the first camera 32 continues to take pictures until the processing flow is completed, and as the taking continues, the data acquisition unit 12 repeatedly acquires image P1 at predetermined acquisition timings.

[0051] Next, the arithmetic unit 10 executes step S02. In step S02, for example, the object detection unit 14 starts the process of detecting objects from image P1. The object detection unit 14 starts the process of individually detecting the truck head 102, the container 110, and the driver D as objects to be detected included in image P1, for example, using a detection model that has been built in advance by machine learning.

[0052] In one example, the object detection unit 14 detects bounding boxes for each type of object to be detected. The coordinates of the detected bounding boxes allow the position of the object to be detected (detected object) within image P1 to be determined. From there, the process of detecting objects to be detected within image P1 is repeatedly executed at predetermined detection timings until the processing flow is completed.

[0053] Next, the arithmetic unit 10 executes step S03. In step S03, for example, the work execution determination unit 16 waits until a container 110 is detected in the image P1. Until the container 110 is detected, the process of individually detecting various objects continues.

[0054] If container 110 is detected (step S03: YES), the processing performed by the arithmetic unit 10 proceeds to step S04. In step S04, for example, the work execution determination unit 16 waits until it moves out of the field of view (out of the field of view) related to image P1. In one example, the work execution determination unit 16 determines that container 110 has moved out of the field of view related to image P1 when container 110 is no longer detected in the process of detecting objects to be detected. The process of individually detecting various objects to be detected continues until the movement of container 110 out of the field of view is detected.

[0055] If it is determined that container 110 has moved out of the field of view, the processing performed by the computing unit 10 proceeds to step S05. In step S05, for example, the work execution determination unit 16 determines whether or not a fixing operation related to container 110 has been performed, based on the detection result of container 110 in image P1 and the detection result of driver D. That is, the work execution determination unit 16 determines whether or not a fixing operation of the locking mechanism 106 has been performed for container 110 that was determined to have moved out of the field of view in step S04 (hereinafter referred to as "container 110 subject to determination"). A specific example of step S05 will be described later.

[0056] Next, the computing unit 10 executes step S06. In step S06, for example, the computing unit 10 determines whether or not there is an instruction to terminate monitoring based on image data captured by the first camera 32. In one example, the computing unit 10 determines that there is an instruction to terminate monitoring based on an instruction from the user of the work monitoring device 1. Alternatively, the computing unit 10 may determine that there is an instruction to terminate monitoring at a time arbitrarily set during the day.

[0057] If it is determined in step S06 that there is no instruction to terminate monitoring (step S06: NO), the processing executed by the arithmetic unit 10 returns to step S03, and thereafter the arithmetic unit 10 repeats the series of processes from steps S03 to S06. On the other hand, if it is determined in step S06 that there is an instruction to terminate monitoring (step S06: YES), the arithmetic unit 10 terminates the processing flow.

[0058] Figure 8 is a flowchart illustrating a method for determining whether or not a fixing operation is performed in step S05. In step S05, the arithmetic unit 10 first executes step S51. In step S51, for example, the operation execution determination unit 16 determines whether or not the driver D was detected during a period that overlaps with at least a portion of the period during which the container 110 to be determined was detected. The period during which the container 110 to be determined was detected is the period from the time when the container 110 to be determined was first detected in image P1 to the time when the container 110 to be determined was no longer detected in image P1.

[0059] If driver D is detected in step S51 (step S51: YES), the processing performed by the arithmetic unit 10 proceeds to step S52. In step S52, for example, the work execution determination unit 16 determines whether the container 110 and driver D (bounding boxes corresponding to the container 110 and driver D, respectively) detected in image P1 satisfy predetermined determination conditions. The predetermined determination conditions are predetermined conditions and are set based on the results of detecting the container 110 and driver D in image P1, which are presumed to indicate that the locking operation of the locking mechanism 106 has been performed. In other words, determining that the above determination conditions are satisfied means that the results of detecting the container 110 and driver D in image P1 are presumed to indicate that the locking operation of the locking mechanism 106 has been performed.

[0060] In one example, when the target detection unit 14 detects driver D, it also determines the driver D's posture within the bounding box Bd (see Figure 9(a)). In the schematic diagram illustrated in Figure 9(a), the result of determining the driver D's posture within the bounding box Bd is shown as "D1". For example, the relationship between the determination result of the driver D's posture relative to the detection position of the container 110 and the result of whether or not the fixing operation was actually performed is accumulated, and the above determination condition is determined from this accumulated relationship. The result of whether or not the fixing operation was actually performed may be determined by visual inspection from the image, or data recorded in the entrance gate 164 may be used.

[0061] In the case where four twist locks are provided on the trailer 100, the work execution determination unit 16 may determine that the above determination condition is met if the condition that fixing work has been performed on one or more of the four twist locks, or on two or more of the locks, is met (i.e., it may be estimated that fixing work related to the container 110 has been performed). Multiple first cameras 32 or multiple second cameras 34 may be provided so that the trailer 100 to be determined can be photographed from different angles in the area where the driver D performs the fixing work on the locking mechanism. In this case, the work execution determination unit 16 may determine whether fixing work has been performed on different twist locks for each image obtained from different angles, and then estimate whether fixing work related to the container 110 has been performed. The number of twist locks to be determined and the number of cameras that generate the monitoring images are arbitrary when estimating whether fixing work related to the container 110 has been performed.

[0062] In step S52, if it is determined that the detected container 110 and driver D satisfy the above determination conditions (step S52: YES), the processing performed by the arithmetic unit 10 proceeds to step S53. In step S53, the work execution determination unit 16 determines that the locking operation of the locking mechanism 106 has been performed on the container 110 that is the target of the determination.

[0063] On the other hand, if in step S52 it is determined that the detected container 110 and driver D do not meet the above determination conditions (step S52: NO), or if in step S51 it is determined that driver D has not been detected (step S51: NO), the processing executed by the arithmetic unit 10 proceeds to step S54. In step S54, the work execution determination unit 16 determines that the locking operation of the locking mechanism 106 has not been performed on the container 110 that is the target of the determination.

[0064] After step S53 or step S54 is executed, the arithmetic unit 10 executes step S55. In step S55, for example, the data recording unit 18 records the determination result from step S53 or step S54. The data recording unit 18 may also record the color identification result of the container 110 to be determined in association with the determination result. If a truck head 102 that is presumed to be carrying the container 110 to be determined is detected, the data recording unit 18 may also record the color identification result of that truck head 102 in association with the determination result. The data recording unit 18 may transmit information indicating the determination result, etc., to an external device of the work monitoring device 1 (for example, a management system such as TOS).

[0065] [Differentiation] The series of processes shown in Figures 7 and 8 are examples and can be modified as appropriate. In the above series of processes, one step and the next step may be executed in parallel, and some steps may be executed in a different order than the example above. In place of at least some of the steps in the above series of processes, or in addition to the above series of processes, steps with content different from the example above may be executed.

[0066] The work execution determination unit 16 may be configured to further determine whether or not driver D is wearing a helmet h when driver D is detected in image P1 or image P2 (see Figure 9(b)). As illustrated in Figure 9(b), the target detection unit 14 may be configured to further detect the helmet h worn by driver D in image P1 or image P2. The detection target objects used by the detection model of the target detection unit 14 may include the helmet h. The detection model used by the target detection unit 14 may detect driver D wearing a helmet h and driver D not wearing a helmet h as separate detection targets.

[0067] The target detection unit 14 may be configured to detect the trailer 100 on which the container 110 is loaded using a detection model. That is, the object to be detected by the detection model may be the trailer 100 on which the container 110 is loaded, instead of the container 110 and the truck head 102. In this case, detecting the trailer 100 on which the container 110 is loaded is substantially equivalent to detecting the container 110.

[0068] Depending on the settings of the field of view captured by the first camera 32 and the second camera 34, multiple trailers 100 may be captured in image P1 or image P2. If multiple trailers 100 are detected for at least partially overlapping periods, the computing unit 10 may assign an identification code to each trailer 100 and then determine whether or not to perform the operation related to locking the locking mechanism 106 for each trailer 100.

[0069] In addition to the first camera 32, the imaging device 30 may include one or more additional cameras that capture a different field of view set in the area before passing through the exit gate 162. In addition to the second camera 34, the imaging device 30 may include one or more additional cameras that capture a different field of view set in the area immediately after passing through the exit gate 162. In these cases, the computing device 10 may perform, for each camera, a determination of whether or not to perform the operation related to detecting an object to be detected and locking based on the image data.

[0070] The predetermined determination conditions used in step S52 are not limited to the examples described above. The work execution determination unit 16 may determine whether or not the locking operation has been performed based on whether or not the time change in the relative positional relationship between the bounding box Bc corresponding to the container 110 and the bounding box Bd corresponding to the driver D satisfies the conditions. The work execution determination unit 16 may also determine whether or not the locking operation has been performed using a trained model separate from the detection model.

[0071] [Verification Example] As described above, the inventors conceived the idea that it is possible to monitor whether or not the operation related to locking container 110 has been performed based on the detection results of container 110 and driver D in the image. To verify this idea, they checked the relationship between image data taken in the area before passing through the exit gate of an actual container terminal (specifically, "Island City Container Terminal") and the records at the exit gate. An example of the results of the check is shown in Table 1 below.

[0072] [Table 1]

[0073] In Table 1, "Truck Color" refers to the color of truck head 102, and "Container Color" refers to the color of container 110. In the "Front Lock" column, "Lock" indicates that the front locking mechanism 106 (twist lock) was used to secure the container, and "Unlock" indicates that the front locking mechanism 106 (twist lock) was not used to secure the container. In the "Rear Lock" column, "Lock" indicates that the rear locking mechanism 106 (twist lock) was used to secure the container, and "Unlock" indicates that the rear locking mechanism 106 (twist lock) was not used to secure the container. In "Example 4," the container color was not recorded, and it was recorded that the front and rear locks were not performed when passing through the gate, but were performed in the area after passing through the gate.

[0074] To determine whether a driver was detected in the image, the image containing the trailer 100 (the image reflecting the detection results of the object) was visually inspected, and it was evaluated whether a driver D was detected around the container 110 loaded on the trailer 100. As shown in Examples 1 to 3, if a driver D was detected around the container 110 in the image, the locking mechanism 106 had been secured. As shown in Example 4, if the securing operation was performed after passing through the gate, a driver D was not detected around the container 110 in the image. In addition, in each of Examples 1 to 4, the color identification results in the image matched the recorded colors. Thus, it was confirmed that it is possible to estimate whether or not the securing operation has been performed based on the detection result of a driver D around (near) the container 110 in the image.

[0075] [Summary of this disclosure] The work monitoring device (1) described above is a device installed in a container terminal (150) that handles containers (110) for maritime transport. This work monitoring device (1) includes cameras (32, 34) that capture images within a field of view set in an area where work related to locking containers (110) loaded on a trailer (100) is performed, an object detection unit (14) configured to detect at least the containers (110) loaded on the trailer (100) and the driver (D) in images (P1, P2) obtained by the cameras (32, 34) using a detection model built in advance by machine learning, and an operation execution determination unit (16) configured to determine whether or not work related to locking the container (110) has been performed based on the detection results of the containers (110) and the driver (D) in the images (P1, P2).

[0076] To date, no attempts have been made to monitor container (110) locking operations from image data at a container terminal (150). Under these circumstances, the inventors conceived the idea that a driver (D) manually performs locking operations around a container (110), and that it is possible to determine whether or not the container (110) locking operations have been performed based on the detection results of the container (110) and the driver (D) in the image. In the above-mentioned work monitoring device (1), it is determined whether or not the container (110) locking operations have been performed based on the detection results of the container (110) and the driver (D) in the images (P1, P2). Therefore, it is possible to monitor whether or not the switching operation of the locking mechanism (106) provided on the trailer (100) has been performed.

[0077] In the work monitoring device (1) described above, the trailer (100) may be provided with a twist lock as a mechanism (106) for securing the container (110). The work related to locking the container (110) loaded on the trailer (100) may be the work of securing the container (110) with the twist lock, or the work of releasing the container (110) with the twist lock. When securing the container (110) with the twist lock, or releasing the container (110), the driver (D) operates the twist lock's operating part around the container (110). Therefore, the characteristics of when the driver (D) performs switching operations such as securing or releasing with the twist lock are reflected in the image data. In the above configuration, the container (110) and the driver (D) are detected in the image, and based on the detection results, it is determined whether or not a securing or releasing operation with the twist lock has been performed. Therefore, it is possible to monitor whether or not the fixing or releasing operation is being performed using the twist lock provided on the trailer (100).

[0078] In the work monitoring device (1) described above, the target detection unit (14) may be configured to further detect the trailer (100). The target detection unit (14) may be configured to identify the color of the trailer (100) when the trailer (100) is detected, and to identify the color of the container (110) when the container (110) is detected. In this case, it is possible to record the colors of the trailer (100) and the container (110) in conjunction with the determination result of whether or not the work related to locking is performed.

[0079] In the work monitoring device (1) described above, the work execution determination unit (16) may be configured to further determine whether or not the driver (D) is wearing a helmet (h) when the driver (D) is detected. In this case, it is possible to monitor whether or not the driver (D) is wearing a helmet (h), and based on the monitoring result or the fact that monitoring is taking place, it is possible to prompt the driver (D) to wear a helmet (h).

[0080] The work monitoring method described above is a method that is performed at a container terminal (150) that handles containers (110) for maritime transport. This work monitoring method includes the steps of: using cameras (32, 34) to photograph the area (154, 156) set up in the area where work related to locking the container (110) loaded on the trailer (100) is performed; using a detection model built in advance by machine learning to detect at least the container (110) loaded on the trailer (100) and the driver (D) in the images (P1, P2) obtained by the cameras (32, 34); and determining whether or not work related to locking the container (110) has been performed based on the detection results of the container (110) and the driver (D) in the images (P1, P2). Similar to the work monitoring device (1) described above, this work monitoring method can monitor whether or not the switching operation of the locking mechanism (106) provided on the trailer (100) has been performed.

[0081] The work monitoring program described above is a program that causes a computer to perform the following steps: acquire images (P1, P2) obtained by taking pictures with cameras (32, 34) within a field of view set in an area (154, 156) of a container terminal (150) where containers (110) for maritime transport are loaded onto a trailer (100) and where work related to locking containers (110) is performed; use a detection model built in advance by machine learning to detect at least the containers (110) loaded onto the trailer (100) and the driver (D) in the images (P1, P2); and determine whether or not work related to locking the containers (110) has been performed based on the detection results of the containers (110) and the driver (D) in the images (P1, P2). Similar to the work monitoring device (1) described above, this work monitoring program is capable of monitoring whether or not the switching operation of the locking mechanism (106) provided on the trailer (100) has been performed. [Explanation of Symbols]

[0082] 1...Work monitoring device, 10...Calculation unit, 14...Target detection unit, 16...Work execution determination unit, 30...Photography device, 32...First camera, P1...Image, 34...Second camera, P2...Image, 100...Trailer, 102...Truck head, 106...Locking mechanism, 110...Container, D...Driver, h...Helmet, 150...Container terminal, 154...Fixed work area, 156...Release work area.

Claims

1. A work monitoring device installed at a container terminal for handling containers for maritime transport, A camera that captures the field of view set in the area where the work related to locking the container loaded on the trailer is performed, A target detection unit is configured to use a detection model pre-built by machine learning to detect at least the container loaded on the trailer and the driver in the image captured by the camera. A work execution determination unit is configured to determine whether or not the operation related to locking the container has been performed, based on the detection result of the container in the aforementioned image and the detection result of the driver. A work monitoring device equipped with the following features.

2. The trailer is equipped with a twist lock as a mechanism for securing the container. The work related to locking the container loaded on the trailer is the work of securing the container with the twist lock, or the work of releasing the container from being secured with the twist lock. The work monitoring device according to claim 1.

3. The aforementioned target detection unit is configured to further detect the trailer, The target detection unit is configured to identify the color of the trailer when the trailer is detected, and to identify the color of the container when the container is detected. The work monitoring device according to claim 1 or 2.

4. The aforementioned work execution determination unit is configured to further determine whether or not the driver is wearing a helmet when the driver is detected. The work monitoring device according to claim 1 or 2.

5. A method for monitoring operations performed at a container terminal where containers for maritime transport are loaded and unloaded, A process of using a camera to photograph the area within the field of view set in the area where the work related to locking the container loaded on the trailer is performed, Using a detection model pre-built by machine learning, the process includes detecting at least the container loaded on the trailer and the driver in the image captured by the camera, A step of determining whether or not the operation related to locking the container has been performed, based on the detection result of the container in the aforementioned image and the detection result of the driver, A work monitoring method, including the following.

6. A process of acquiring images obtained by taking pictures with a camera within a field of view set in an area of ​​a container terminal where containers for maritime transport are loaded onto trailers and the work related to locking the containers is performed, Using a detection model pre-built by machine learning, the process includes detecting, in the image, at least, the container loaded on the trailer and the driver. A step of determining whether or not the operation related to locking the container has been performed, based on the detection result of the container in the aforementioned image and the detection result of the driver, A task monitoring program that causes a computer to perform a task.

Images