Information processing device, information processing system, information processing method, and computer program

Abstract

[Problem] To provide an information processing device, an information processing system, an information processing method, and a computer program for outputting information about cargo handling on the basis of images. [Solution] An information processing device according to the present invention comprises an acquisition unit that acquires a time series of images that include the hull of a cargo ship and the surrounding environment, a detection unit that detects an object related to cargo handling from each of the images, a tracking unit that tracks the movement of the object through the images, and a determination unit that determines whether the movement of the object as based on the results of the tracking meets a prescribed operation condition for evaluating achievement of cargo handling work.

Description

Information processing device, information processing system, information processing method, and computer program

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, and a computer program that output information related to cargo handling based on an image.

[0002] Systems that identify objects captured in images taken by a camera and analyze the identified objects are becoming widespread. Patent Document 1 discloses a control device that controls a crane using a state determination model that has been trained to output the state of the crane when an image of the movement of a crane transporting garbage is input.

[0003] International Publication No. 2021 / 187185

[0004] Conventionally, in cargo handling on board a cargo ship or in coastal cargo handling, understanding of the cargo handling situation has often relied on workers' visual inspection.

[0005] The present disclosure aims to provide an information processing device, an information processing system, an information processing method, and a computer program that output information related to cargo handling based on images.

[0006] An information processing device according to one aspect of the present disclosure includes an acquisition unit that acquires a plurality of images in chronological order, the images including the hull of a cargo ship and the surrounding environment; a detection unit that detects an object related to loading and unloading from each of the plurality of images; a tracking unit that tracks the movement of the object across the plurality of images; and a determination unit that determines whether the movement of the object based on the tracking results satisfies predetermined operating conditions for assessing whether loading and unloading work is occurring.

[0007] In one aspect of the present disclosure, whether or not cargo handling is taking place is automatically determined by tracking the movement of objects related to cargo handling from an image that includes the cargo ship's hull and the wharf within its capture range, thereby enabling an operator to recognize the progress of cargo handling without visual inspection.

[0008] An information processing device according to one aspect of the present disclosure includes a measurement unit that measures the progress of loading and unloading based on the judgment result of the judgment unit, and an output unit that outputs data regarding the progress of loading and unloading of the cargo ship measured by the measurement unit.

[0009] In one aspect of the present disclosure, the progress of cargo handling is measured based on the result of the determination of whether cargo handling work is being performed, and the automatically measured progress of cargo handling is output so that it can be confirmed visually using graphics such as graphs or audibly by reading out numbers, etc. This makes it possible for an operator to quantitatively grasp the progress of cargo handling rather than directly checking the cargo handling with his or her own eyes.

[0010] In an information processing device of one aspect of the present disclosure, when an image is input, the detection unit determines whether or not an object related to loading and unloading is captured in the image, and detects the object using a learning model that has been trained to output the position of the object within the image if the object is captured.

[0011] In one aspect of the present disclosure, a method for detecting objects related to cargo handling from an image uses image recognition based on a learning model using a neural network, a support vector machine, etc. This makes it possible to grasp the progress of cargo handling with a good balance between accuracy and processing speed.

[0012] In the information processing device according to one aspect of the present disclosure, the object related to loading and unloading is a part of a crane or cargo.

[0013] In one aspect of the present disclosure, an object detected as being captured in an image is a part of a crane related to loading and unloading or cargo. Detecting the movement of the crane makes it possible to measure the progress of loading and unloading. Detecting the cargo allows an operator to accurately recognize the progress of loading and unloading while determining whether the loading and unloading equipment, such as a crane, is actually carrying cargo.

[0014] In an information processing device of one aspect of the present disclosure, the determination unit determines whether or not a condition is satisfied as the specified operating condition, which condition is that the object has crossed a specified boundary between the hull and the outside of the ship, which is set for the image.

[0015] In one aspect of the present disclosure, whether or not cargo handling work is being performed is determined based on whether or not a specific object related to cargo handling that is captured in an image has crossed the boundary between the hull and the outside of the ship. During unloading or loading, an easily recognizable object such as a crane hook moves back and forth between the hull and the shore side, and the presence or absence of cargo handling work can be determined by counting the number of times this object has crossed over.

[0016] In the information processing device according to one aspect of the present disclosure, the determination unit determines whether or not a condition that the moving speed of the object is equal to or greater than a predetermined value is satisfied as the predetermined operating condition.

[0017] In one aspect of the present disclosure, whether or not cargo handling is being performed is determined based on whether or not a specific object related to cargo handling that is captured in an image is moving at a predetermined speed or faster, and this determination can be made based on whether or not a predetermined boundary is repeated in a similar pattern, without the need to set a predetermined boundary for each ship.

[0018] In the information processing device according to one aspect of the present disclosure, the determination unit determines whether or not a condition that the amount of rotation of the crane is equal to or greater than a predetermined amount is satisfied as the predetermined operating condition.

[0019] In one aspect of the present disclosure, whether or not loading and unloading work is being performed is determined based on whether or not the amount of rotation of the crane related to loading and unloading shown in the image is equal to or greater than a predetermined amount. When unloading or loading, the crane rotates by more than a predetermined amount, and by recognizing this, it is possible to determine whether or not loading and unloading work is being performed. Even if it is difficult to recognize the crane hook, other detection methods can be appropriately selected as long as they are possible.

[0020] In the information processing device according to one aspect of the present disclosure, the determination unit determines whether or not a condition that a pattern of change in the skeleton of the crane is a predetermined pattern is satisfied as the predetermined operating condition.

[0021] In one aspect of the present disclosure, whether or not loading and unloading work is being performed is determined based on whether or not the pattern of changes in the crane's skeleton related to loading and unloading shown in the image matches a predetermined pattern. When unloading or loading, the crane's joints and other parts move in a specific pattern, and by recognizing this, it is possible to determine whether or not loading and unloading work is being performed. Even if it is difficult to recognize the crane hook, other detection methods can be selected as appropriate as long as they are possible.

[0022] In the information processing device according to one aspect of the present disclosure, the determination unit determines whether or not a condition that the predetermined operating condition is further satisfied is that it is recognized that cargo is being transported by a crane.

[0023] In one aspect of the present disclosure, a determination as to whether or not it is recognized that cargo is actually being transported by a crane is added to the pattern for determining whether or not cargo handling work is being performed as described above. This allows counting when cargo handling work is being performed reliably.

[0024] In the information processing device of one aspect of the present disclosure, the measurement unit measures the number of times cargo is loaded onto or unloaded from the cargo ship as the progress of loading and unloading.

[0025] In one aspect of the present disclosure, the number of loadings or unloadings is measured as the progress of cargo handling. One movement of the crane hook or one rotation of the crane corresponds to one loading or unloading (of a predetermined unit of cargo), so by tracking this, progress can be measured with high accuracy.

[0026] In the information processing device according to one aspect of the present disclosure, the measurement unit measures, as the progress of cargo handling, the ratio of the number of times the cargo has been loaded or unloaded to the planned number of times cargo handling operations have been performed on the cargo ship.

[0027] In one aspect of the present disclosure, if the number of times of unloading or loading is given in advance, the progress can be measured as a percentage using the measured number of times, thereby making it possible to grasp the progress relative to the overall loading and unloading process.

[0028] In the information processing device according to one aspect of the present disclosure, the measurement unit estimates the end time of cargo handling based on the rate of increase in the number of times cargo is loaded or unloaded onto the cargo ship as the progress of cargo handling.

[0029] In one aspect of the present disclosure, when the number of loading or unloading operations is given in advance, it is possible to estimate the time when the loading or unloading operation will be completed from the measured number of operations and the elapsed time. This makes it possible to grasp the progress based on the navigation schedule of the target ship and the usage schedule of coastal facilities, etc.

[0030] In an information processing device according to one aspect of the present disclosure, the output unit creates screen data including a graph showing the progress of the loading and unloading measured by the measuring unit, and outputs the created screen data to a display device.

[0031] In one aspect of the present disclosure, a screen is output so that the automatically measured progress of cargo handling can be visually confirmed in a graph, which allows an operator to intuitively and accurately grasp the progress of cargo handling compared to directly visually confirming the cargo handling.

[0032] In an information processing device according to one aspect of the present disclosure, the detection unit extracts different regions from each of the multiple images and detects the object in each of the different regions, and the tracking unit tracks the movement of the object in each of the different regions.

[0033] According to one aspect of the present disclosure, even if there are multiple specific objects related to cargo handling captured in an image, it is possible to measure the progress of cargo handling by detecting the movement of each of them, making it possible to measure the progress of a variety of cargo handling equipment.

[0034] In the information processing device according to one aspect of the present disclosure, the acquisition unit acquires an image from a camera installed on a cargo ship so as to include the upper surface of the deck of the hull in its angle of view.

[0035] In one aspect of the present disclosure, a camera is used that is installed on a cargo ship so that the upper surface of the deck of the hull is included in the angle of view. A camera that has other uses, such as observing ocean conditions, may also be used, and it is possible to recognize the progress of cargo handling by using a camera that has multiple uses without installing a special sensor for monitoring cargo handling.

[0036] An information processing system of one aspect of the present disclosure includes a camera installed on a cargo ship so that the upper surface of the hull's deck is included in its field of view, and an information processing device that acquires images captured by the camera, wherein the information processing device acquires multiple images in chronological order that include the hull of the cargo ship and the surrounding environment, detects objects related to loading and unloading from each of the multiple images, tracks the movement of the objects across the multiple images, and determines whether the movement of the objects based on the results of the tracking satisfies predetermined operating conditions for evaluating whether loading and unloading work is being performed.

[0037] An information processing method according to one aspect of the present disclosure includes a computer acquiring multiple images in chronological order that include the hull of a cargo ship and the surrounding environment, detecting objects related to loading and unloading from each of the multiple images, tracking the movement of the objects across the multiple images, and determining whether the movement of the objects based on the tracking results satisfies predetermined operating conditions for assessing whether loading and unloading operations are occurring.

[0038] A computer program according to one aspect of the present disclosure causes a computer to execute a process of acquiring multiple images in chronological order, including the hull of a cargo ship and the surrounding environment, detecting objects related to loading and unloading from each of the multiple images, tracking the movement of the objects across the multiple images, and determining whether the movement of the objects based on the tracking results satisfies predetermined operating conditions for assessing whether loading and unloading operations are occurring.

[0039] 1 is a schematic diagram of an information processing system according to a first embodiment; FIG. 2 is a block diagram showing the configuration of an information processing device; FIG. 3 is a block diagram showing the configuration of an output device; FIG. 4 is an explanatory diagram of functions based on an information processing program; FIG. 5 is an explanatory diagram showing an example of an image obtained from a camera; FIG. 6 is an explanatory diagram showing an example of an image obtained from a camera; FIG. 7 is a schematic diagram of a learning model used in an information processing device; FIG. 8 is a flowchart showing an example of a processing procedure by an information processing device; FIG. 9 is an explanatory diagram of a determination of whether unloading of cargo is progressing; FIG. 10 is a flowchart showing an example of an output process of information relating to cargo handling from an information processing device; FIG. 11 is an explanatory diagram of an example of a display screen; FIG. 12 is an explanatory diagram of another example of a display screen; FIG. 13 is a flowchart showing an example of a processing procedure by an information processing device according to a third embodiment; FIG. 14 is a flowchart showing an example of a processing procedure by an information processing device according to the third embodiment; FIG. 15 is an explanatory diagram of processing contents of the information processing device according to the third embodiment; FIG. 16 is a schematic diagram of an information processing system according to a fourth embodiment; FIG. 17 is a block diagram showing the configuration of a server device; FIG. 18 is a flowchart showing an example of a processing procedure by an information processing device according to the fourth embodiment;

[0040] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present disclosure will be specifically described with reference to the accompanying drawings illustrating embodiments thereof. In the following embodiments, an information processing system including an information processing device according to the present disclosure will be described.

[0041] 1 is a schematic diagram of an information processing system 100 according to a first embodiment. The information processing system 100 includes a camera 2 provided on the deck of a ship S, an information processing device 1 that processes images captured by the camera 2, and an output device 3 that outputs the results of processing by the information processing device 1. The ship S is a cargo ship such as a container ship.

[0042] Camera 2 uses a visible light imaging element. Camera 2 captures images in time series at a predetermined rate, such as 10 frames per second, and continues to output image data. Camera 2 is installed high on the deck of the hull, facing the bow so that the top surface of the upper deck of the hull and the direction of travel are included in the angle of view (see Figures 5 and 6). Camera 2 is also used to capture images of sea conditions while ship S is sailing. Camera 2 may use not only visible light but also infrared light imaging elements.

[0043] A communication device 21 is connected to the camera 2 via a network SN installed on the ship S. The communication device 21 is a device that connects to the onboard network SN and realizes communication with the outside of the ship via a network N that includes communication media such as satellite communication and a carrier network. The communication device 21 may include a communication device that communicates with land-based devices or other ships using an AIS-dedicated frequency, or may be a wireless communication device that connects to a carrier network, or may be a wireless communication device for Wi-Fi.

[0044] The information processing device 1 is installed on land. The information processing device 1 acquires images captured by a camera 2 via a communication device 21, and performs sea condition assessment and other processing on the acquired images, as described below. The information processing device 1 can be connected for communication with an output device 3. The information processing device 1 outputs the results of processing on the images to the output device 3, and provides sea condition observation and cargo handling monitoring services.

[0045] The output device 3 is used on land or on board the ship. The output device 3 is an information terminal that can be connected to the information processing device 1. The output device 3 is used by the owner of the ship S, the manager of the cargo loaded on the ship S, etc., to check the operating status of the ship S and monitor cargo handling.

[0046] In the information processing system 100, the information processing device 1 can output sea conditions based on analysis of images from the camera 2 described above, and information obtained from each device installed on the ship S. The information processing device 1 derives the progress of cargo handling using images acquired from the camera 2, which captures within its field of view the ship S moored at a wharf, the work of unloading cargo from the deck to land, or the work of loading cargo onto the ship, and makes it possible to check the progress of cargo handling on the output device 3. Below, a detailed description will be given of how the information processing device 1 derives and checks the progress of cargo handling using images from the camera 2.

[0047] 2 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 uses a server computer and includes a processing unit 10, a storage unit 11, a first communication unit 12, and a second communication unit 13.

[0048] The processing unit 10 includes one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), etc. The processing unit 10 also includes a temporary storage medium such as a static random access memory (SRAM) or a dynamic random access memory (DRAM). The processing unit 10 reads an information processing program P1 stored in the storage unit 11 into the temporary storage medium and executes it, thereby causing a general-purpose computer to perform various processes described below and function as the information processing device 1 of the present disclosure.

[0049] The memory unit 11 is a relatively large-capacity non-volatile memory medium such as an SSD (Solid State Drive) or a hard disk. The memory unit 11 stores a program (program product) required for the processing unit 10 to execute processing, and reference setting data. The setting data may include the type of ship S on which the information processing device 1 is installed and identification information. The setting data includes data used to determine image processing, which will be described later. The program product includes the information processing program P1, a server program for displaying a screen on the output device 3, images to be included on the screen, and their design (arrangement). The program product includes a learning model M1.

[0050] The information processing program (program product) P1 and the learning model M1 stored in the storage unit 11 may be the information processing program P9 and the learning model M9 stored in a computer-readable non-transitory storage medium 9 that are read by the processing unit 10 and stored in the storage unit 11. The information processing program P1 and the learning model M1 may be the information processing program P9 and the learning model M9 that the processing unit 10 downloads from a download server via the second communication unit 13 and stores in the storage unit 11.

[0051] The storage unit 11 stores image data acquired from the camera 2 in association with time information so that the image data can be referenced in chronological order. The storage unit 11 may also store data relating to the progress of loading and unloading operations obtained as a result of processing the image data, which will be described later.

[0052] The first communication unit 12 is a communication device that realizes a communication connection with the camera 2. The first communication unit 12 enables communication with the onboard communication device 21 to which the camera 2 is connected. The first communication unit 12 may be a communication device that communicates via satellite communication, or may be a communication device that communicates using an AIS-dedicated frequency. The first communication unit 12 may be a wireless communication device for a carrier network, or may be a wireless communication device for Wi-Fi.

[0053] The second communication unit 13 is a communication device that realizes a communication connection with the output device 3. The second communication unit 13 enables a communication connection from the output device 3 via a network N including a public communication network and a carrier network. The second communication unit 13 may be a network card for a wired LAN, a communication device that realizes carrier communication via a carrier network, or a communication device compatible with a wireless network such as Wi-Fi or Bluetooth (registered trademark).

[0054] 3 is a block diagram showing the configuration of the output device 3. The output device 3 is a personal computer, a smartphone, or a tablet terminal. The output device 3 may be used by a user of the cargo handling monitoring service or by an operator of the information processing device 1 of the cargo handling monitoring service.

[0055] The output device 3 includes a processing unit 30, a storage unit 31, a communication unit 32, a display unit 33, and an operation unit 34. The processing unit 30 includes one or more processors such as a CPU, an MPU, a GPU, etc. The processing unit 30 also includes a memory that is a temporary storage medium such as an SRAM or a DRAM.

[0056] The storage unit 31 is a relatively large-capacity non-volatile storage medium such as an SSD or a hard disk. The storage unit 31 stores a client program corresponding to a service that provides information related to sea conditions and cargo handling monitoring from the information processing device 1. The client program is, for example, a web browser program. The client program is not limited to a web browser program, but may be a special program that causes the processing unit 30 to execute a process of displaying data transmitted from the information processing device 1 on a screen.

[0057] The communication unit 32 is a communication device that realizes a communication connection with the information processing device 1 via the network N. The communication unit 32 may be a communication device that realizes a communication connection with the information processing device 1 via a dedicated line.

[0058] The display unit 33 uses a display such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 33 corresponds to a "display device." The display unit 33 displays a web page including text and images through processing based on the client program of the processing unit 30. The display unit 33 may use a display with a built-in touch panel.

[0059] The operation unit 34 is a user interface such as a keyboard or a pointing device that accepts operations from a user or an operator. The operation unit 34 may be a touch panel built into the display of the display unit 33, or may be physical buttons. The operation unit 34 may be a voice input unit that accepts operations by voice using a voice recognition function. The operation unit 34 can notify the processing unit 30 of operation information by the user or operator.

[0060] In the information processing system 100 configured in this manner, the information processing device 1 can execute a process of combining the image obtained from the camera 2 with information obtained from each sensor of the ship S and outputting the combined image so that it can be displayed on the output device 3. In addition to these functions, when the target ship S anchors at a port and loading / unloading operations begin, the information processing device 1 recognizes, from the image obtained from the camera 2, the loading / unloading of cargo that was loaded on board the ship to the shore and vice versa, and outputs the progress of the loading / unloading operations.

[0061] The following describes the function of outputting the progress of cargo handling based on images obtained from the camera 2. Figure 4 is an explanatory diagram of the function based on the information processing program P1. The processing unit 10 of the information processing device 1 performs the various functions shown in Figure 4 based on the information processing program P1.

[0062] The processing unit 10 functions as an acquisition unit 101 that acquires, in time series, a plurality of images including the hull and cargo handling locations of the ship S. As the acquisition unit 101, the processing unit 10 acquires images (see FIGS. 5 and 6 ) from a camera 2 that is installed on the upper deck of the ship S facing the bow.

[0063] The processing unit 10 functions as a detection unit 102 that detects objects related to cargo handling from the images acquired by the acquisition unit 101. As the detection unit 102, the processing unit 10 detects specific objects related to cargo handling from the images using a learning model M1. The specific objects are parts that repeatedly move in a predetermined pattern during cargo handling and are objects that are easy to capture with the camera 2. Examples of objects related to cargo handling include the hook of a crane, the cargo to be handled, the boom of a crane, the skeleton of a crane, or the rotation mechanism of a crane. The detection method used by the detection unit 102 is not limited to object detection using the learning model M1, and may also be a detection method using pattern recognition.

[0064] The processing unit 10 functions as a tracking unit 103 that tracks the movement of an object across multiple images that can be acquired in time series by the acquisition unit 101. As the tracking unit 103, the processing unit 10 calculates a movement trajectory that chronologically connects ranges in which an object related to cargo handling is detected within images acquired in time series from the camera 2. As the tracking unit 103, the processing unit 10 calculates a movement trajectory, for example, by chronologically connecting the centers (or center of gravity) of the object detection ranges.

[0065] The processing unit 10 functions as a determination unit 104 that determines whether the movement of a specific object related to cargo handling tracked by the tracking unit 103 satisfies a predetermined operating condition for evaluating whether or not cargo handling is occurring. The predetermined operating condition for evaluating whether or not cargo handling is occurring is, for example, that cargo is suspended from the hook of a crane detected by the detection unit 102 as an object related to cargo handling, and that the crane hook is moving between the ship and the surrounding environment. The surrounding environment may be, for example, wharf-side facilities outside the ship (coastal facilities). It may also be other ships or marine structures present around the ship S at sea. Details of the predetermined operating condition will be described later. However, the processing unit 10, using the function of the determination unit 104, determines whether the position in the image of the hook of the crane with cargo suspended at its end crosses a predetermined boundary set within the range of the image (see FIG. 9 ). When the hook moves from the ship to outside (the wharf side) with cargo suspended from the end of the hook and crosses a predetermined boundary, the processing unit 10 can determine that one cargo handling operation (unloading) has been performed and that the predetermined operating condition is satisfied. Conversely, when the hook moves from outside (the wharf side) to onboard with cargo suspended from the end of the hook and crosses a predetermined boundary, the processing unit 10 can determine that one cargo handling operation (loading) has been performed. The predetermined operating condition may be another condition. Further details of the function of the determination unit 104 will be described later.

[0066] The above-mentioned specified boundary is a line corresponding to the boundary of the hull in the image defined for the camera 2, and is stored in the memory unit 11 in association with the identification data of the ship S on which the camera 2 is installed, and can be referenced.

[0067] The processing unit 10 functions as a measurement unit 105 that measures the progress of cargo handling based on the determination result by the determination unit 104. When a predetermined operating condition that allows the determination unit 104 to evaluate that cargo handling work is being performed is satisfied, the processing unit 10, as the measurement unit 105, measures the number of times as the progress of cargo handling, or measures the ratio of the progress to the planned number of times. The processing unit 10, as the measurement unit 105, may measure the elapsed time as the progress of cargo handling and estimate the end time of cargo handling.

[0068] The processing unit 10 functions as an output unit 106 that outputs data relating to the progress of loading and unloading of the ship S measured by the measurement unit 105. The processing unit 10 uses the function of the output unit 106 to output data to an external device having a display, such as the output device 3. Using the function of the output unit 106, the processing unit 10 may output text or images indicating the progress, or may output audio indicating the progress.

[0069] 5 and 6 are explanatory diagrams showing example images obtained from the camera 2. FIGS. 5 and 6 are line diagrams showing images obtained by the processing unit 10 from the camera 2 using the acquisition unit 101. FIG. 5 shows an image taken while the ship S is sailing, and FIG. 6 shows an image taken while the ship S is moored at a wharf during cargo handling. Both of the images in FIGS. 5 and 6 show the upper deck on the bow side of the ship. The image in FIG. 5 shows the ocean ahead of the bow, including the horizon, as the ship S sails. The image in FIG. 6 shows the ship S, as well as the wharf where the ship S is moored. The image in FIG. 6 shows the upper deck raised, revealing the cargo hold inside the ship and the cargo inside the hold. The image in FIG. 6 shows cargo (tires in the image in FIG. 6) being lifted by a crane installed at the wharf.

[0070] FIG. 7 is a schematic diagram of the learning model M1 used in the information processing device 1. When the image shown in FIG. 5 or 6 is input, the learning model M1 is trained to detect objects related to cargo handling from the image. The learning model M1 is a deep learning model for image recognition using a neural network. The learning model M1 is a model trained using, for example, an SSD (Single Shot MultiBox Detector). The learning model M1 may employ a model without a convolutional layer, such as a Vision Transformer, or may employ other architectures. The learning model M1 may also employ a model using a support vector machine, etc.

[0071] In the example shown in FIG. 7 , the learning model M1 employs a CNN (Convolution Neural Network)-based image recognition model. The learning model M1 includes an input layer to which an image is input and an intermediate layer including a convolution layer that performs filtering on the image. The learning model M1 includes an output layer that outputs coordinate data within the image of an area in which an object related to cargo handling is captured and a score indicating the likelihood that the object captured in that area is the target object. The learning model M1 detects, for example, an area in which a crane hook is captured and cargo as specific objects related to cargo handling. The learning model M1 may be generated by learning separate models for detecting a crane hook and a cargo. Different detection models may be generated depending on the type of cargo.

[0072] The learning model M1 may be trained to detect an area in which the skeleton of a crane is captured as a specific object related to cargo handling. The learning model M1 may be trained to detect an area in which a special hand attached to the tip of a crane is captured. The learning model M1 may be trained to detect cargo being carried by a crane as a specific object related to cargo handling.

[0073] The learning model M1 is trained using, as training data, image data obtained from the camera 2 and coordinate data indicating a known range in which a specific part is captured on the image, as shown in Figure 5 or 6. The learning model M1 may be retrained after the start of operation.

[0074] The processing in the information processing system 100 configured as described above will be described. FIG. 8 is a flowchart showing an example of a processing procedure by the information processing device 1. When receiving an instruction to start processing, the processing unit 10 of the information processing device 1 executes the processing procedure shown below. The information processing device 1 may receive the instruction to start processing by an operator operating the operation unit 34 of the output device 3, or may receive the instruction to start processing by acquiring information related to the navigation of the ship S (e.g., berthing, start of cargo handling) from another device. For example, when the processing unit 10 can recognize that the doors of the upper deck or cargo hold have been opened from the switch states of those devices, it may determine that an instruction to start processing has been received and start the following processing.

[0075] The processing unit 10 acquires an image from the camera 2 via the first communication unit 12 using the function of the acquisition unit 101 (step S101). The processing unit 10 provides the acquired image to the learning model M1 (step S102). The processing unit 10 acquires a detection result from the learning model M1 using the function of the detection unit 102 (step S103). In steps S102 and S103, the processing unit 10 detects the crane hook as described with reference to FIGS. 5 to 7.

[0076] The processing unit 10 stores the image acquired in step S101 and the detection result acquired in step S103 in the storage unit 11 in association with time information (step S104). In step S104, the processing unit 10 stores the identification data of the ship S on which the target camera 2 is mounted in association with the image acquired in step S101 and the detection result acquired in step S103.

[0077] The processing unit 10 compares the detection result acquired in step S103 with past detection results, and tracks the movement of the object related to the loading and unloading process using the function of the tracking unit 103 (step S105). The processing unit 10 stores (appends) the tracking result of step S105 in the storage unit 11 (step S106).

[0078] The processing unit 10 determines whether or not the movement of the object related to cargo handling satisfies a predetermined operation condition for evaluating whether or not cargo handling work is being performed (step S107) by using the function of the determination unit 104. The processing unit 10 performs the process of step S107 to determine whether or not cargo handling work, such as unloading or loading of cargo, is progressing.

[0079] If the processing unit 10 determines in step S107 that the predetermined operating conditions are satisfied (S107: YES), the measuring unit 105 uses its function to advance the progress of the loading and unloading process in predetermined increments in association with the identification data and time information of the ship S, and stores the progress (step S108). In step S108, the processing unit 10 adds up the number of times cargo is loaded and unloaded each time as the progress of the loading and unloading process. The processing unit 10 may increase the required number of times set at the start of loading and unloading, i.e., the ratio of the number of times of unloading to the number of cargo pieces to be unloaded.

[0080] The processing unit 10 determines whether or not the loading and unloading process has been completed (step S109). If the processing unit 10 determines that the loading and unloading process has not been completed (S109: NO), the processing unit 10 returns the process to step S101.

[0081] In step S109, the processing unit 10 determines whether or not the loading and unloading process has been completed based on whether or not an instruction to end the process has been received. The processing unit 10 may receive the instruction to end the process by an operator operating the operation unit 34 of the output device 3, or may receive the instruction to end the process by acquiring information related to the navigation of the ship S (completion of loading and unloading, departure from port) from another device. For example, the processing unit 10 may determine that an instruction to end the process has been received when it can recognize that the doors to the upper deck or cargo hold have been closed from the state of their switches. The processing unit 10 may automatically determine that an instruction to end the process has been received when a predetermined time (e.g., a long period such as 12 hours) has elapsed since the start of the process.

[0082] In step S109, if the processing unit 10 determines that the loading and unloading has been completed (S109: YES), the processing unit 10 ends the processing.

[0083] If the processing unit 10 determines in step S107 that the predetermined operating conditions are not satisfied (S107: NO), it determines whether an abnormality is suspected based on the tracking results of step S105 (step S110). In step S110, the processing unit 10 determines that an abnormality is suspected if a predetermined time or more has passed since the predetermined operating conditions were most recently satisfied, based on the tracking results stored in the memory unit 11. If progress is stalled, such as when the crane hook is no longer detected or when the crane hook is detected but stopped, the processing unit 10 can determine that an abnormality is suspected.

[0084] If the processing unit 10 determines that no abnormality is suspected (S110: NO), the processing unit 10 proceeds to step S109.

[0085] If the processing unit 10 determines that an abnormality is suspected (S110: YES), it stores in the memory unit 11 the fact that an abnormality is suspected, in association with the identification data and time information of the ship S (step S111), and proceeds to step S109.

[0086] FIG. 9 is an explanatory diagram of determining whether cargo unloading is progressing. FIG. 9 shows a specific example of the processing unit 10 determining whether a predetermined operating condition is satisfied in step S107. FIGS. 9A to 9C are arranged in chronological order. Each of FIGS. 9A to 9C shows a predetermined boundary L and a hook trajectory T superimposed on a line diagram of an image corresponding to FIG. 6. The predetermined boundary L corresponding to the boundary of the hull is indicated by a dashed line in FIGS. 9A and 9B, and the hook trajectory T is indicated by a thick solid line. The predetermined boundary L is set toward the top of the image, outside the boundary between the area in which the hull is captured and the rest of the image. In the example images shown in FIGS. 9A and 9B, the area in which the hull is captured is a trapezoidal area extending from the center to the bottom edge of the image, and the predetermined boundary L is defined by two straight lines that are close to each other at the top along the outsides of both legs of this trapezoid. The predetermined boundary L is stored in the memory unit 11 in association with the identification data of the ship S on which the camera 2 is installed, when the camera 2 is set up. The predetermined boundary L may be stored in the memory unit 11 as setting data for each type of ship S.

[0087] The example image shown in Figure 9A, similar to Figure 6, shows a ship's hull with its upper deck open and the cargo hold's interior in the captured range, as well as a pier. In the example image of Figure 9A, the area F in which the crane hook detected using learning model M1 is captured is indicated by hatching. The example image of Figure 9A shows cargo suspended from the hook being lifted by the crane's operation. In the example image shown in Figure 9A, the area F in which the hook is captured overlaps the area in the image in which the ship's hull is captured, and is located between the two straight lines that define the predetermined boundary L.

[0088] The example image shown in Figure 9B, like Figure 9A, shows a ship with an open upper deck and a wharf, as well as a crane hook and cargo hanging from the hook. In the example image shown in Figure 9B, compared to the image shown in Figure 9A, the area F in which the hook is visible, indicated by hatching, is higher in the image. In the example image shown in Figure 9B, the area F in which the hook is visible is outside the area in which the ship is visible and is located on the wharf side of the two straight lines that define the predetermined boundary L.

[0089] The example image shown in Figure 9C, like Figure 9A, shows a ship with an open upper deck and a wharf, as well as a crane hook and cargo hanging from the hook. In the example image shown in Figure 9C, compared to the images shown in Figures 9A and 9B, the area F in which the hook is visible, indicated by hatching, has risen to the top of the image and moved further toward the wharf. In the example image shown in Figure 9C, the area F in which the hook is visible is outside the area in which the ship is visible and is located further toward the wharf outside the two straight lines that define the predetermined boundary L.

[0090] The processing unit 10, which acquires the images shown in Figures 9A, 9B, and 9C in chronological order, determines in step S107 that the area F in which the hook is visible has crossed a predetermined boundary L within the image at the time the image shown in Figure 9B is acquired. The processing unit 10 further recognizes cargo below the area F in which the hook is visible in the images from Figures 9A to 9C. Therefore, the processing unit 10 determines that the movement satisfies the processing condition for determining that unloading of cargo is progressing, and in step S108, increments the progress (number of times) of unloading by one. The processing unit 10 may determine that the movement satisfies the processing condition for determining that unloading of cargo is progressing by confirming that the area F in which the hook is visible has moved further toward the wharf at the time the image changes from that shown in Figure 9B to that shown in Figure 9C.

[0091] In the case of loading, the images are taken in the reverse order of the chronological order from Figure 9A to Figure 9C. In other words, when it is determined that cargo is suspended below and the range F of the hook moves from the wharf side to the ship side and crosses the predetermined boundary L, the processing unit 10 determines that the movement of the object related to loading and unloading satisfies the predetermined operation condition (S107), and advances the loading progress (number of times) by one.

[0092] 8, the information processing device 1 executes a process of outputting the progress of cargo handling of the ship S to the output device 3 in response to a request from the output device 3. When the output device 3 accesses a portal screen of the cargo handling monitoring service provided by the information processing device 1 based on a client program (web browser program), the following process is started. Figure 10 is a flowchart showing an example of a process of outputting information related to cargo handling from the information processing device 1.

[0093] The processing unit 30 of the output device 3 receives a selection of identification data of a ship S to be monitored for cargo work from the portal screen via the operation unit 34 (step S301). The processing unit 30 transmits a request including the selected identification data to the information processing device 1 (step S302). The processing unit 30 may display on the display unit 33 a list of ships S that are permitted as monitoring targets for the account of the operator of the output device 3 and are currently loading and unloading, and receive a selection of a cargo work monitoring target from the list.

[0094] The processing unit 10 of the information processing device 1 receives a request from the output device 3 (step S121). The processing unit 10 identifies the account of the operator of the output device 3 that made the request and the identification data of the target ship S (step S122). The processing unit 10 reads data on the progress of the most recent cargo handling that is stored in the memory unit 11 in association with the identification data of the identified ship S (step S123). In step S123, the processing unit 10 may perform approval processing, such as determining whether the identification data of the ship S is appropriate as a target for cargo handling monitoring based on navigation information (such as the position of the ship S) that can be sequentially obtained from other services, and whether the identification data can be output to the operator's account.

[0095] The processing unit 10 creates a screen including the read data on the progress of the cargo handling (step S124) and transmits the created screen data to the output device 3 (step S125). In step S124, the processing unit 10 creates a screen including, for example, the number of times unloading or loading has occurred, i.e., the number of times it has been determined that a predetermined operating condition has been met (the number of times the range F of the hook suspending the cargo has crossed the predetermined boundary L), as data on the progress of the cargo handling. The processing unit 10 may also create this screen by including a graph showing the change in the number of unloading or loading events over time. The processing unit 10 may also calculate the progress rate of the cargo handling from the rate of increase in the number of unloading or loading events and create a screen including a number indicating this progress rate (rate of increase). In step S124, the processing unit 10 may also calculate the ratio of the number of unloading or loading events to the planned number of unloading or loading events and create a screen showing this as the progress rate. When the processing unit 10 can acquire data on the planned number of times from the loading and unloading schedule data stored in association with the identification data of the ship S, it can create a screen that includes this ratio as the progress rate. When the processing unit 10 can acquire data on the planned number of times, it may estimate the completion time of unloading or loading (loading end time) based on the current progress rate and the time required to reach the current progress rate, and create a screen that includes the estimated loading and unloading end time together with the progress rate data. The processing unit 10 may also create a screen that includes images obtained from the camera 2.

[0096] The processing unit 30 of the output device 3 receives the screen data (step S303) and, based on the received data, displays a screen showing data on the progress of loading and unloading on the monitored vessel S on the display unit 33 (step S304). The screen displayed in step S304 is preferably updated by the client program every time a predetermined time elapses, so that the latest progress data is displayed.

[0097] In addition to step S124, the processing unit 10 of the information processing device 1 may create a sound notifying the progress and transmit the sound to the output device 3.

[0098] FIG. 11 is an explanatory diagram of an example of a display screen. FIG. 11 shows an example of a display screen 330 displayed on the display unit 33 based on a client program. The display screen 330 shown in FIG. 11 includes a graph 331 showing the change in the number of unloading or loading operations over time as data on the progress of loading and unloading. The horizontal axis of the graph 331 represents elapsed time, and the vertical axis represents the number of operations. The graph 331 shown in FIG. 11 indicates that the number of unloading or loading operations does not increase at a specific time, i.e., loading and unloading operations are stopped. The graph 331 indicates that the angles when the number of unloading or loading operations increases are the same, indicating that the loading and unloading progress speed is constant. An operator viewing the display screen 330 can understand the loading and unloading stop period and the loading and unloading progress speed from the graph 331. The processing unit 10 of the information processing device 1 may calculate the progress speed and create a screen that displays the progress speed together with the graph 331. In this way, the information processing system 100 of the present disclosure can output the display screen 330 on the output device 3, allowing the loading and unloading status to be grasped remotely.

[0099] FIG. 12 is an explanatory diagram of another example of the display screen. Similar to FIG. 11, FIG. 12 shows an example of a display screen 330 displayed on the display unit 33. The display screen 330 shown in FIG. 12 is similar to the display screen 330 shown in FIG. 11, but displays a notification indicating a suspected abnormality along with a graph 331. The graph 331 in the display example of FIG. 12 shows a flat period in which the number of loading or unloading operations does not increase for more than a predetermined time. The area corresponding to the flat period is highlighted in the graph 331 as indicated by hatching. In addition, text indicating the situation, such as "Loading and unloading operations are delayed," is displayed next to the graph 331. In this way, the loading and unloading operations can be monitored remotely along with the progress of the loading and unloading operations.

[0100] [Second Embodiment] In the second embodiment, instead of using the learning model M1 to detect the hook in the image, the movement of the crane boom and the pattern of the crane's skeleton are detected to determine whether the cargo handling is progressing. The configuration of the information processing system 100 of the second embodiment is the same as the configuration of the information processing system 100 of the first embodiment, except for some of the functions of the information processing device 1 described later. Therefore, the same reference numerals are used for the common configuration, and detailed description thereof will be omitted.

[0101] In the second embodiment, the processing unit 10 also executes the processing procedure shown in FIG. 8 . However, in steps S102 and S103, instead of detecting the crane hook as an object related to cargo handling as shown in the first embodiment, the processing unit 10 detects the crane boom and specific parts that can recognize the rotation of the crane. Alternatively, in steps S102 and S103, the processing unit 10 detects the skeletal parts of the crane, which make specific movements when lifting and lowering cargo, as objects related to cargo handling. For this reason, in the second embodiment, the learning model M1 is trained using training data to detect specific parts, such as the crane's rotation axis and joints, rather than the crane hook.

[0102] In step S105, the processing unit 10 tracks the movement of the parts detected in steps S102 and S103. Specifically, the processing unit 10 tracks the rotation of a rotating part such as a boom, or tracks the opening or closing of a skeleton part.

[0103] In step S107, the processing unit 10 determines whether it can be confirmed based on tracking that the amount of rotation of the crane is equal to or greater than a predetermined amount and that cargo is hanging from the crane hook at that time. The processing unit 10 may also determine whether the condition that the moving speed of the hook is equal to or greater than a predetermined speed is satisfied. The learning model M1 may be one that has been trained to output the amount of rotation when multiple consecutive images (videos) are input, and the processes from step S102 to step S107 may be executed using this model.

[0104] In step S107, the processing unit 10 also determines whether or not it can be confirmed that the pattern of change in the crane's skeleton satisfies a predetermined pattern as a result of tracking, and that cargo is hanging from the crane hook at that time. The learning model M1 may be one that has been trained to determine movement when a plurality of consecutive images (videos) are input, and the processes from step S102 to S107 may be executed using this model.

[0105] The information processing device 1 executes the other steps S108 to S111 in the same manner as in the first embodiment. As a result, the movement of objects related to loading and unloading is detected using images obtained from the camera 2, and the progress of loading and unloading can be automatically quantified from the results of tracking that movement. In the second embodiment, the information processing device 1 also outputs data on the progress of loading and unloading from the display unit 33 of the output device 3. As a result, it becomes possible to grasp the progress remotely without having to visually check the progress at the loading and unloading site.

[0106] The progress of cargo handling can be measured not only by the hook but also by the movement of the crane boom, as shown in the second embodiment, and the pattern of how the frame opens, etc. By being able to appropriately select the detection and tracking of objects related to cargo handling depending on various circumstances such as the type of crane and the pattern of the hook, it becomes possible to allow the operator to more accurately recognize the progress of cargo handling.

[0107] [Third Embodiment] The third embodiment corresponds to a case where two or more cranes are used for cargo handling on one ship S. The configuration of the information processing system 100 of the third embodiment is similar to the configuration of the information processing system 100 of the first embodiment, except for some of the functions of the information processing device 1 described later, and therefore the same reference numerals are used for the common configuration, and detailed description thereof will be omitted.

[0108] 13 and 14 are flowcharts showing an example of a processing procedure performed by the information processing apparatus 1 according to the third embodiment.

[0109] In the third embodiment, the processing unit 10 of the information processing device 1 acquires an image from the camera 2 via the first communication unit 12 using the function of the acquisition unit 101 (step S131).

[0110] The processing unit 10 extracts multiple regions from the image acquired in step S131 (step S132). In step S132, the processing unit 10 extracts the regions based on coordinate data specifying the regions that are set in advance. The coordinate data specifying the regions may be stored in the memory unit 11 in association with identification data of the ship S on which the camera 2 is installed, or may be stored in association with identification data of the pier (port) at which the ship S is moored. The coordinate data specifying the regions may be specified by the operator while checking the displayed image via the output device 3 operated by the operator before the start of processing. The coordinate data specifying the regions may be set in advance according to the type and number of cranes used for cargo handling. In this case, the processing unit 10 recognizes the type and number of cranes, or extracts the regions based on set data corresponding to the type and number of cranes selected by the operator.

[0111] The processing unit 10 provides the image of each extracted region to the learning model M1 (step S133). The processing unit 10 obtains the detection result for each extracted region from the learning model M1 (step S134).

[0112] The processing unit 10 stores the image acquired in step S131 and the detection results in association with time information for each region in the storage unit 11 (step S135). In step S135, the processing unit 10 stores the images acquired in step S131 and the detection results in association with the identification data of the vessels S so that the detection results can be read out in chronological order for each vessel S.

[0113] The processing unit 10 compares the detection result with past detection results for each different extracted region to track the movement of the object related to loading and unloading (step S136). The processing of step S135 is the same as step S105 of the first embodiment, and the processing unit 10 stores (appends) the tracking result of step S136 for each region in the storage unit 11 (step S137).

[0114] The processing unit 10 determines, for each region, based on the tracking result, whether the movement of the object related to the loading and unloading satisfies a predetermined operation condition (step S138). The processing for each region in step S138 is the same as step S107 in Fig. 8 of the first embodiment, and therefore a detailed description thereof will be omitted.

[0115] If the processing unit 10 determines in step S138 that the predetermined operation conditions are satisfied for each area (S138: YES), it advances the loading / unloading progress by a predetermined unit in association with the identification data and time information of the ship S and stores the progress (step S139). If the processing unit 10 determines in step S139 that the predetermined operation conditions are satisfied in one area, it advances the loading / unloading progress of the crane corresponding to the target area by one. If the processing unit 10 determines that the predetermined operation conditions are satisfied in two areas, it advances the loading / unloading progress of the two cranes corresponding to each area by one. The same applies to three or more areas. The processing unit 10 may advance the overall loading / unloading progress of the target ship S based on the determination of the operation conditions for the crane movements for each extracted area. If the processing unit 10 determines that the predetermined operation conditions are satisfied in each of two areas, it may advance the loading / unloading progress by two.

[0116] The processing unit 10 determines whether or not the loading and unloading process has been completed (step S140). If the processing unit 10 determines that the loading and unloading process has not been completed (S140: NO), the processing returns to step S131. The process of step S140 is the same as step S109 in FIG. 8 of the first embodiment, and therefore a detailed description thereof will be omitted.

[0117] In step S140, if the processing section 10 determines that the loading and unloading has been completed (S140: YES), the processing section 10 ends the processing.

[0118] If the processor 10 determines in step S138 that the predetermined operating conditions are not satisfied (S138: NO), that is, if it determines that the predetermined operating conditions are not satisfied in any of the areas, it determines whether an abnormality is suspected (step S141). The process of step S141 is similar to step S110 in Fig. 8 of the first embodiment, and therefore a detailed description thereof will be omitted. If the processor 10 determines that the loading and unloading progress is progressing in any of the areas, it may skip the determination of step S141.

[0119] If the processing unit 10 determines that no abnormality is suspected (S141: NO), the processing unit 10 proceeds to step S140.

[0120] If the processing unit 10 determines that an abnormality is suspected (S141: YES), it stores in the memory unit 11 the fact that an abnormality is suspected, in association with the identification data and time information of the ship S (step S142), and proceeds to step S140.

[0121] Fig. 15 is an explanatory diagram of the processing content of the information processing device 1 of the third embodiment. Similar to Fig. 6, Fig. 15 shows, in a line diagram, an image captured during loading and unloading on a ship S moored at a wharf, with regions R1 and R2 indicating different extracted regions superimposed on the line diagram. Regions R1 and R2 are indicated by thick dashed dotted lines. In Fig. 15, a predetermined boundary L is also superimposed on the line diagram.

[0122] Region R1 is set for the cargo hold that is shown at the back of the hull and wharf in the image. Region R2 is set for the cargo hold that is moved to the front of the hull and wharf in the image. The processing unit 10 detects, for each of regions R1 and R2, an area that shows, for example, a crane hook, and determines the movement of that area and whether cargo is hanging from the end of the hook.

[0123] In the example image shown in FIG. 15 , regions R1 and R2 overlap. When regions R1 and R2 are extracted while distinguishing between the rear and front sides as shown in FIG. 15 , the crane hook corresponding to the other region is captured in both regions R1 and R2. Therefore, in the example image shown in FIG. 15 , the processing unit 10 adopts the detection result based on the size of the range F in which the crane hook is detected. For example, for region R1, the processing unit 10 adopts the detection result for range F below a predetermined size, and does not adopt the detection result for range F above the predetermined size, since the object is located in region R2. Similarly, for region R2, the processing unit 10 adopts the detection result for range F above the predetermined size, and does not adopt the detection result for range F below the predetermined size, since the object is located in region R2. The processing unit 10 may also determine whether to adopt or not adopt the detection result based on the size of the suspended cargo, etc.

[0124] The settings of the regions R1 and R2 shown in Fig. 15 differ for each camera 2 and depending on the coastal facilities, such as the number of corresponding cranes. Therefore, the setting data for the different regions may be set one by one before the start of loading and unloading, or may be set in advance according to the combination of the installation environment of the camera 2 (ship S) and the wharf facilities. Note that the regions R1 and R2 may be set so that they do not overlap.

[0125] In this way, even when multiple cranes are used to simultaneously load and unload cargo on a single cargo ship, progress can be calculated by detecting the load using images with distinct areas.

[0126] [Fourth Embodiment] In the fourth embodiment, an information processing device 1 is installed on a cargo ship, and transmits processing results to an output device 3 via a server device 4 installed on land via a network N. FIG. 16 is a schematic diagram of an information processing system 200 in the fourth embodiment. In the fourth embodiment, the information processing device 1 is communicatively connected to the camera 2 via a network SN installed on the ship, without going through a communication device 21. The information processing device 1 realizes communication with a server device 4 outside the ship via a network N including communication media such as satellite communication and a carrier network. The information processing device 1 may also be communicatively connected to the network N via the communication device 21.

[0127] In the fourth embodiment, a server device 4 is provided on land. The server device 4 is a device that stores data transmitted from the information processing device 1 and outputs data relating to the progress of cargo handling in response to a request from the output device 3.

[0128] In the fourth embodiment, the information processing device 1 does not need to transmit images outside the ship and is installed on board, so an edge computer with relatively limited computing resources is used. The information processing device 1 may be configured to execute processing distributed across multiple edge computers to simultaneously perform other functions. In the fourth embodiment, the first communication unit 12 of the information processing device 1 may be a communication device that realizes communication with the camera 2, such as a wired LAN, a wireless communication device for WiFi, a coaxial cable, a USB, or a short-range wireless communication device such as Bluetooth (registered trademark). The second communication unit 13 is a communication device that realizes communication with the output device 3 via the server device 4. The second communication unit 13 may be a communication device that communicates via satellite communication or a communication device that communicates using an AIS-dedicated frequency. The first communication unit 12 may be a wireless communication device for a carrier network or a wireless communication device for WiFi. In the fourth embodiment, the memory unit 11 does not need to store images and their designs (arrangements) for displaying on the screen of the output device 3. Other functions and processing of the memory unit 11 are basically the same as those of the first embodiment, so detailed description thereof will be omitted.

[0129] 17 is a block diagram showing the configuration of the server device 4. The server device 4 includes a processing unit 40, a storage unit 41, a first communication unit 42, and a second communication unit 43. The processing unit 40 includes one or more arithmetic processing units such as a CPU, an MPU, or a GPU. The processing unit 40 also includes a temporary storage medium such as an SRAM or a DRAM.

[0130] The storage unit 41 is a relatively large-capacity non-volatile storage medium such as an SSD, a hard disk, etc. The storage unit 41 stores a server program, and also stores images to be displayed on the screen of the output device 3 and their designs (layouts).

[0131] The storage unit 41 stores the data transmitted from the information processing device 1 in association with the identification data of the information processing device 1 or the identification data of the ship on which the information processing device 1 is installed.

[0132] The first communication unit 42 is a communication device that realizes a communication connection with the information processing device 1. The first communication unit 42 may be a communication device that communicates via satellite communication, or may be a communication device that communicates using an AIS-dedicated frequency. The first communication unit 42 may be a wireless communication device for a carrier network, or may be a wireless communication device for Wi-Fi.

[0133] The second communication unit 43 is a communication device that realizes a communication connection with the output device 3. The second communication unit 43 enables a communication connection from the output device 3 via a network N including a public communication network and a carrier network. The second communication unit 43 may be a network card for a wired LAN, a communication device that realizes carrier communication via a carrier network, or a communication device compatible with a wireless network such as Wi-Fi or Bluetooth (registered trademark).

[0134] Fig. 18 is a flowchart showing an example of a processing procedure by the information processing device 1 of the fourth embodiment. Of the processing procedures shown in Fig. 18, steps common to the processing procedures shown in Fig. 8 of the first embodiment are assigned the same step numbers, and detailed descriptions thereof will be omitted.

[0135] In the fourth embodiment, the processing unit 10 similarly executes the processes of steps S101 to S108 shown in Fig. 8. After the processing unit 10 advances and stores the degree of progress of loading and unloading in predetermined increments in association with the identification data and time information of the vessel S (S108), the processing unit 10 transmits the degree of progress of loading and unloading in association with the identification data and time information of the vessel S from the second communication unit 13 to the server device 4 (step S151).

[0136] The processing unit 10 similarly executes the processes of steps S109-S111. If the processing unit 10 determines in step S110 that an abnormality is suspected (S110: YES), it stores the fact that an abnormality is suspected in the storage unit 11 (S111) and notifies the server device 4 (step S152). In step S152, the processing unit 10 transmits an abnormality notification to the server device 4 in association with the identification data of the vessel S. When the server device 4 receives the abnormality notification from the information processing device 1, it may output a push notification to the output device 3 of the account associated with the associated identification data of the vessel S.

[0137] The embodiments disclosed above are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the claims, and includes all modifications within the meaning and scope of the claims.

[0138] Furthermore, independent claims and dependent claims described in the claims can be combined with each other in any and all combinations, regardless of the reference format. Furthermore, although the claims use a format in which a claim references two or more other claims (multiple claim format), this is not limited to this format. Multiple claims that reference at least one other multiple claim (multi-multi claim format) may also be used.

[0139] The following additional notes are provided regarding the above-described embodiment.

[0140] (Supplementary Note 1) An information processing device comprising: an acquisition unit that acquires multiple images including the hull of a cargo ship and the surrounding environment in chronological order; a detection unit that detects objects related to loading and unloading from each of the multiple images; a tracking unit that tracks the movement of the objects across the multiple images; and a determination unit that determines whether the movement of the objects based on the tracking results satisfies predetermined operating conditions for evaluating whether loading and unloading work is being performed.

[0141] (Supplementary Note 2) The information processing device according to Supplementary Note 1, comprising: a measuring unit that measures the progress of loading and unloading based on the determination result of the determining unit; and an output unit that outputs data relating to the progress of loading and unloading of the cargo ship measured by the measuring unit.

[0142] (Supplementary Note 3) The information processing device according to Supplementary Note 1 or 2, wherein the detection unit, when an image is input, determines whether an object related to loading and unloading is captured in the image, and detects the object using a learning model that has been trained to output the position of the object in the image if the object is captured.

[0143] (Supplementary Note 4) The information processing device according to any one of Supplementary Notes 1 to 3, wherein the object related to loading and unloading is a part of a crane or cargo.

[0144] (Supplementary Note 5) The information processing device according to any one of Supplementary Notes 1 to 4, wherein the determination unit determines whether a condition that the object has crossed a predetermined boundary between the hull and the outside of the ship, which is set for the image, is satisfied as the predetermined operating condition.

[0145] (Supplementary Note 6) The information processing device according to any one of Supplementary Notes 1 to 5, wherein the determination unit determines whether or not a condition that a moving speed of the object is equal to or greater than a predetermined value is satisfied as the predetermined operating condition.

[0146] (Supplementary Note 7) The information processing device according to any one of Supplementary Notes 4 to 6, wherein the determination unit determines whether or not a condition that a rotation amount of a crane is equal to or greater than a predetermined amount is satisfied as the predetermined operating condition.

[0147] (Supplementary Note 8) The information processing device according to any one of Supplementary Notes 4 to 7, wherein the determination unit determines whether or not a condition that a pattern of change in a skeleton of a crane is a predetermined pattern is satisfied as the predetermined operation condition.

[0148] (Supplementary Note 9) The information processing device according to any one of Supplementary Notes 4 to 8, wherein the determination unit determines whether or not a condition that the predetermined operating condition is further satisfied is that it is recognized that cargo is being transported by a crane.

[0149] (Supplementary Note 10) The information processing device according to any one of Supplementary Notes 1 to 9, wherein the measurement unit measures the number of times cargo is loaded onto or unloaded from the cargo ship as the progress of cargo handling.

[0150] (Supplementary Note 11) The information processing device according to any one of Supplementary Notes 2 to 10, wherein the measurement unit measures, as the progress of cargo handling, a ratio of the number of times the cargo has been loaded or unloaded to a planned number of times cargo handling operations have been performed on the cargo ship.

[0151] (Supplementary Note 12) The information processing device according to any one of Supplementary Notes 2 to 11, wherein the measurement unit estimates an end time of cargo handling based on an increasing rate of the number of times cargo is loaded or unloaded onto the cargo ship as the progress of cargo handling.

[0152] (Supplementary Note 13) The information processing device described in any one of Supplementary Notes 2 to 12, wherein the output unit creates screen data including a graph showing the progress of the loading and unloading measured by the measurement unit, and outputs the created screen data to a display device.

[0153] (Supplementary Note 14) The information processing device according to any one of Supplementary Notes 1 to 13, wherein the detection unit extracts different regions from each of the plurality of images, and detects the object for each different region, and the tracking unit tracks movement of the object for each different region.

[0154] (Supplementary Note 15) The information processing device according to any one of Supplementary Notes 1 to 14, wherein the acquisition unit acquires an image from a camera installed on the cargo ship so as to include an upper surface of a deck of the hull in an angle of view.

[0155] (Supplementary Note 16) An information processing system comprising: a camera installed on a cargo ship so as to include the upper surface of the deck of the hull in its angle of view; and an information processing device that acquires images captured by the camera, wherein the information processing device acquires a plurality of images including the hull of the cargo ship and the surrounding environment in chronological order, detects an object related to cargo handling from each of the plurality of images, tracks the movement of the object across the plurality of images, and determines whether the movement of the object based on the tracking results satisfies predetermined operating conditions for evaluating whether or not cargo handling work is being performed.

[0156] (Supplementary Note 17) An information processing method in which a computer acquires a plurality of images including the hull of a cargo ship and the surrounding environment in chronological order, detects an object related to loading and unloading from each of the plurality of images, tracks the movement of the object across the plurality of images, and determines whether the movement of the object based on the tracking results satisfies predetermined operating conditions for evaluating whether loading and unloading work is occurring.

[0157] (Supplementary Note 18) A computer program that causes a computer to execute the following processes: acquire a plurality of images including the hull of a cargo ship and the surrounding environment in chronological order; detect an object related to loading and unloading from each of the plurality of images; track the movement of the object across the plurality of images; and determine whether the movement of the object based on the tracking results satisfies predetermined operating conditions for evaluating whether loading and unloading work is occurring. term

[0158] Not necessarily all objects or advantages may be achieved in accordance with any particular embodiment described herein. Thus, for example, one skilled in the art will appreciate that a particular embodiment may be configured to operate to achieve or optimize one or more advantages as taught herein without necessarily achieving other objects or advantages as taught or suggested herein.

[0159] All processes described herein may be embodied and fully automated by software code modules executed by a computing system including one or more computers or processors. The code modules may be stored on any type of non-transitory computer-readable medium or other computer storage device. Some or all of the methods may be embodied in dedicated computer hardware.

[0160] Many other variations beyond those described herein will be apparent from this disclosure. For example, depending on the embodiment, certain operations, events, or functions of any of the algorithms described herein may be performed in a different sequence, added, merged, or omitted entirely (e.g., not all described acts or events are necessary to execute an algorithm). Furthermore, in certain embodiments, operations or events may be performed in parallel rather than sequentially, e.g., via multithreading, interrupt processing, or multiple processors or processor cores, or on other parallel architectures. Furthermore, different tasks or processes may be performed by different machines and / or computing systems that may function together.

[0161] The various illustrative logical blocks and modules described in connection with the embodiments disclosed herein may be implemented or executed by a machine such as a processor. The processor may be a microprocessor, but alternatively, the processor may be a controller, microcontroller, or state machine, or a combination thereof. The processor may include electrical circuitry configured to process computer-executable instructions. In another embodiment, the processor includes an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable device that performs logical operations without processing computer-executable instructions. A processor may also be implemented as a combination of computing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Although described herein primarily with reference to digital technology, a processor may also include primarily analog elements. For example, some or all of the signal processing algorithms described herein may be implemented by analog circuitry or mixed analog and digital circuitry. The computing environment can include any type of computer system, including, but not limited to, a microprocessor, mainframe computer, digital signal processor, portable computing device, device controller, or computer system based on a computational engine within an appliance.

[0162] Unless otherwise specified, conditional language such as "can," "could," "would," or "potential" is understood within the context in which it is generally used to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language does not generally imply that features, elements, and / or steps are required in any manner in one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included in or performed in any particular embodiment.

[0163] Disjunctive language such as "at least one of X, Y, Z," unless specifically stated otherwise, is understood in its general context to indicate that an item, term, etc. can be either X, Y, Z, or any combination thereof (e.g., X, Y, Z). Thus, such disjunctive language does not generally imply that a particular embodiment requires at least one of X, at least one of Y, or at least one of Z, respectively, to be present.

[0164] Any process descriptions, elements, or blocks in the flow diagrams described herein and / or illustrated in the accompanying drawings should be understood as potentially representing modules, segments, or portions of code, comprising one or more executable instructions for implementing a particular logical function or element in the process. Alternative embodiments are included within the scope of the embodiments described herein, in which elements or functions may be performed out of order, substantially simultaneously, or in reverse order from that shown or described, depending on the functionality involved, as will be understood by those skilled in the art.

[0165] Unless otherwise expressly stated, numeral terms such as "one" should generally be construed to include one or more described items. Thus, phrases such as "one device configured to" are intended to include one or more listed devices. Such one or more listed devices may also be collectively configured to perform the recited reference. For example, "a processor configured to perform the following A, B, and C" may include a first processor configured to perform A and a second processor configured to perform B and C. Additionally, even if a specific number of enumerations of the introduced embodiments are explicitly recited, those skilled in the art should construe such enumerations to typically mean at least the recited number (e.g., the mere enumeration of "two enumerations" without other modifiers typically means at least two enumerations, or two or more enumerations).

[0166] In general, it will be appreciated by those skilled in the art that the terms used herein generally intend "non-limiting" terms (e.g., the term "including" should be interpreted as "including but not limited to at least," the term "having" should be interpreted as "having at least," the term "including" should be interpreted as "including, but not limited to," etc.).

[0167] For purposes of description, the term "horizontal" as used herein is defined as a plane parallel to the plane or surface of the floor of the area in which the described system is used or the plane in which the described method is performed, regardless of its orientation. The term "floor" can be interchanged with the terms "ground" or "water surface." The term "vertical / plumb" refers to a direction perpendicular / vertical to a defined horizontal line. Terms such as "upper," "lower," "below," "top," "side," "higher," "lower," "above," "over," "below," etc. are defined relative to the horizontal plane.

[0168] As used herein, the terms "attach," "connect," "mate," and other related terms, unless otherwise noted, should be interpreted to include detachable, movable, fixed, adjustable, and / or removable connections or couplings. Connections / couplings include direct connections and / or connections with intermediate structures between the two components described.

[0169] Unless otherwise expressly stated, as used herein, numbers preceded by terms such as "approximately," "about," and "substantially" are inclusive of the recited number and also refer to an amount close to the recited amount that performs the desired function or achieves the desired result. For example, "approximately," "about," and "substantially" refer to values ​​less than 10% of the recited numerical value, unless otherwise expressly stated. As used herein, features of the disclosed embodiments preceded by terms such as "approximately," "about," and "substantially" refer to features that have some variability that also perform the desired function or achieve the desired result for that feature.

[0170] Many variations and modifications may be made to the above-described embodiments, and these elements should be understood to be among other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure and are protected by the following claims.

[0171] REFERENCE SIGNS LIST 1 Information processing device 10 Processing unit 11 Storage unit 12 Communication unit P1 Information processing program (computer program) M1 Learning model 2 Camera 3 Output device 30 Processing unit 32 Communication unit 33 Display unit 330 Display screen

Claims

1. An information processing device comprising: an acquisition unit that acquires multiple images including the hull of a cargo ship and the surrounding environment in chronological order; a detection unit that detects objects related to loading and unloading from each of the multiple images; a tracking unit that tracks the movement of the objects across the multiple images; and a determination unit that determines whether the movement of the objects based on the tracking results satisfies predetermined operating conditions for evaluating whether loading and unloading work is occurring.

2. An information processing device as described in claim 1, comprising: a measuring unit that measures the progress of loading and unloading based on the judgment result of said judgment unit; and an output unit that outputs data regarding the progress of loading and unloading of said cargo ship measured by said measuring unit.

3. The information processing device according to claim 1, wherein the detection unit, when an image is input, determines whether or not an object related to loading and unloading is captured in the image, and detects the object using a learning model that has been trained to output the position of the object in the image if the object is captured.

4. The information processing device according to claim 1, wherein the object related to loading and unloading is a part of a crane or cargo.

5. The information processing device according to claim 1, wherein the determination unit determines whether or not the condition that the object has crossed a predetermined boundary between the hull and the outside of the ship, which is set for the image, is satisfied as the predetermined operating condition.

6. The information processing device according to claim 1, wherein the determination unit determines whether or not the predetermined operating condition is satisfied, the condition being that the moving speed of the object is equal to or greater than a predetermined value.

7. The information processing device according to claim 4, wherein the determination unit determines whether or not the predetermined operating condition is satisfied, the condition being that the amount of rotation of the crane is equal to or greater than a predetermined amount.

8. The information processing device according to claim 4, wherein the determination unit determines whether or not the predetermined operating condition is satisfied, that is, whether or not the pattern of change in the crane's skeleton is a predetermined pattern.

9. An information processing device according to any one of claims 5 to 8, wherein the determination unit determines whether or not the predetermined operating condition further satisfies a condition that it is recognized that cargo is being transported by a crane.

10. The information processing device according to claim 2, wherein the measurement unit measures the number of times cargo is loaded onto or unloaded from the cargo ship as the progress of cargo handling.

11. The information processing device according to claim 10, wherein the measurement unit measures the ratio of the number of times the cargo has been loaded or unloaded to the planned number of times the cargo handling work has been carried out on the cargo ship as the progress of the handling work.

12. The information processing device according to claim 10, wherein the measurement unit estimates the time when cargo handling will be completed based on the rate of increase in the number of times cargo is loaded or unloaded onto the cargo ship as the progress of cargo handling.

13. The information processing device according to claim 2, wherein the output unit creates screen data including a graph showing the progress of the loading and unloading measured by the measuring unit, and outputs the created screen data to a display device.

14. The information processing device according to claim 1, wherein the detection unit extracts different regions from each of the plurality of images, and detects the object in each of the different regions, and the tracking unit tracks the movement of the object in each of the different regions.

15. The information processing device according to claim 1, wherein the acquisition unit acquires images from a camera installed on a cargo ship so as to include the upper surface of the deck of the hull in its angle of view.

16. An information processing system comprising: a camera installed on a cargo ship so that the upper surface of the hull deck is included in the field of view; and an information processing device that acquires images captured by said camera, wherein said information processing device acquires a plurality of images including the hull of the cargo ship and the surrounding environment in chronological order, detects objects related to cargo handling from each of said plurality of images, tracks the movement of said objects across said plurality of images, and determines whether the movement of said objects based on the results of said tracking satisfies predetermined operating conditions for evaluating whether or not cargo handling work is being carried out.

17. An information processing method in which a computer acquires multiple images in chronological order, including the hull of a cargo ship and the surrounding environment, detects objects related to loading and unloading from each of the multiple images, tracks the movement of the objects across the multiple images, and determines whether the movement of the objects based on the tracking results satisfies specified operating conditions for assessing whether loading and unloading operations are occurring.

18. A computer program that causes a computer to execute the following process: acquire multiple images in chronological order that include the hull of a cargo ship and the surrounding environment; detect objects related to loading and unloading from each of the multiple images; track the movement of the objects across the multiple images; and determine whether the movement of the objects based on the tracking results satisfies specified operating conditions for assessing whether loading and unloading operations are occurring.

Images