Control system, control method, and program

Abstract

To promote the automation of grab dredging operations. [Solution] A control system according to one embodiment controls a grab-type dredging device for performing grab-type dredging work, which involves using a crane installed on a ship for seabed dredging and a grab bucket suspended by the crane to grab mud from the seabed in an area enclosed by a pollution control frame and load the mud onto a soil carrier moored to the ship. The control system includes an acquisition means for acquiring dynamic information, including at least one of the position, height, displacement, speed, rotation angle, angular velocity, and vibration frequency of the crane, the grab bucket, and at least one of the objects to be targeted in the grab-type dredging work, using sensing data obtained from a remote sensing means attached to the crane and an AI that analyzes the sensing data, and a control means for controlling the crane or the grab bucket according to the dynamic information.

Description

Technical Field

[0004] , , , , , , ,

[0005]

[0001] The present invention relates to grab dredging.

Background Art

[0002] Techniques for automating dredging work are known. For example, Patent Document 1 discloses a technique for machine learning the relationship between sensor data (i.e., the state of each part) and an operation that optimizes construction efficiency by a skilled operator in a pump dredging device. Also, although not related to dredging work, Patent Document 2 discloses a technique for determining the position of a movable element by machine learning from a camera image of the movable element in a work machine having the movable element. Patent Document 3 discloses a system for measuring the distance between a work machine and an object around it based on the parallax of a plurality of stereo image data in which the same area around the work machine is photographed, and controlling an alarm signal by a warning device based on the distance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 relates to a pump dredging device and is not applicable to grab dredging.

[0005] In contrast, the present invention provides a technique for automatically performing the work of grab dredging while corresponding to changes in the surrounding situation during operation.

Means for Solving the Problems

[0006] One aspect of the present disclosure provides a control system for a grab dredging device for performing grab dredging work, which involves using a crane installed on a ship for seabed dredging and a grab bucket suspended by the crane to grab mud from the seabed in an area enclosed by a pollution control frame and load the mud onto a soil carrier moored to the ship, and the control system includes an acquisition means for acquiring dynamic information, including position, height, displacement, speed, rotation angle, angular velocity, and at least one of the objects to be targeted in the grab dredging work, using sensing data obtained from a remote sensing means attached to the crane and an AI that analyzes the sensing data, and a control means for controlling the crane or the grab bucket according to the dynamic information.

[0007] The object in question is the pollution control frame, and the mechanical information may include displacement information indicating the displacement of the pollution control frame.

[0008] The remote sensing means includes an RGB camera, the sensing data includes image data of the pollution prevention frame obtained from the RGB camera, the AI ​​has an extraction means for performing predetermined image processing on the image data and extracting feature portions of the pollution prevention frame, a position detection means for detecting the position of the pollution prevention frame using at least the feature portions obtained by the image processing, and a calculation means for calculating the displacement from the time change of the position detected by the position detection means, and the acquisition means may acquire information regarding the displacement of the pollution prevention frame calculated by the calculation means as the displacement information.

[0009] The image processing may include at least one of an edge detection process for detecting edges from the image data and a color identification process for extracting portions of a specific color from the image data.

[0010] The remote sensing means includes a LiDAR, the sensing data includes point cloud data of the area around the pollution control frame obtained from the LiDAR, the AI ​​has an extraction means for performing predetermined data processing on the point cloud data and extracting feature portions of the pollution control frame, a position detection means for detecting the position of the pollution control frame using at least the feature portions obtained by the data processing, and a calculation means for calculating the displacement from the time change of the position detected by the position detection means, and the acquisition means may acquire information regarding the displacement of the pollution control frame calculated by the calculation means as the displacement information.

[0011] The data processing may include at least one of the following processes: a filtering process for filtering point clouds from the point cloud data that satisfy a distance condition; a clustering process for extracting clusters where point clouds are densely packed from the point cloud data; a corner identification process for identifying the positions of the corners of the pollution control frame from the point cloud data; and a position estimation process for estimating the position of the pollution control frame from the point cloud data and the dimensional data of the pollution control frame.

[0012] The object in question is the soil transport vessel, and the mechanical information may include height information indicating the height of the coaming of the soil transport vessel, and position information indicating its location.

[0013] The remote sensing means includes an RGB camera, the sensing data includes image data of the comb obtained from the RGB camera, the AI ​​has an extraction means for performing predetermined image processing on the image data to extract the comb, and a calculation means for calculating the height and position of the comb extracted from the image data, and the acquisition means may acquire information regarding the height and position of the comb calculated by the calculation means as the height information and the position information.

[0014] The image processing may include at least one of the following: edge detection processing for detecting edges from the image data; color identification processing for extracting portions of a specific color from the image data; and line detection processing for detecting straight lines from the image data.

[0015] The remote sensing means includes a LiDAR, the sensing data includes point cloud data of the combing obtained from the LiDAR, the AI ​​has a calculation means for calculating the height and position of the combing from the point cloud data corresponding to the combing, and the acquisition means may acquire the information regarding the height and position of the combing calculated by the calculation means as the height information and the position information.

[0016] The object is the grab bucket, and the mechanical information may include oscillation information indicating the rotation or swing of the grab bucket.

[0017] The remote sensing means includes an RGB camera, the sensing data includes image data of the grab bucket obtained from the RGB camera, the AI ​​has a calculation means that performs predetermined image processing on the image data and calculates the aspect ratio of the grab bucket in the image data, and a rotation detection means that detects the rotation state of the grab bucket using at least the aspect ratio obtained by the image processing, and the acquisition means may acquire information regarding the rotation detected by the rotation detection means as oscillation information.

[0018] The image processing may include a color identification process that extracts a portion of a specific color from the image data.

[0019] The remote sensing means includes a LiDAR, the sensing data includes point cloud data of the area around the grab bucket obtained from the LiDAR, the AI ​​has a calculation means that performs predetermined data processing on the point cloud data and calculates the aspect ratio of the grab bucket in the point cloud data, and a rotation detection means that detects the rotation state of the grab bucket using at least the aspect ratio obtained by the data processing, and the acquisition means may acquire information regarding the rotation detected by the rotation detection means as oscillation information.

[0020] The data processing may include a filtering process that filters out point cloud data for points whose distance satisfies a certain condition.

[0021] The remote sensing means includes an RGB camera, the sensing data includes video data of the grab bucket obtained from the RGB camera, the AI ​​has a determination means that performs predetermined image processing on the frames included in the video data to determine the center position of the grab bucket in the frame, and a vibration detection means that detects the vibration of the grab bucket according to the time change of the center position, and the acquisition means may acquire information regarding the detected vibration as vibration information.

[0022] The remote sensing means may include a LiDAR, the sensing data may include point cloud data of the grab bucket obtained from the LiDAR, the AI ​​may have a calculation means for calculating the position of the grab bucket according to the time change of the point cloud data, and a vibration detection means for detecting the vibration of the grab bucket according to the time change of the position of the grab bucket, and the acquisition means may acquire information regarding the vibration detected by the vibration detection means as vibration information.

[0023] The object is the grab bucket, and the mechanical information may include grab operation information indicating at least one of the trajectory of the grab bucket, the rotation of the grab bucket, or the swing of the grab bucket.

[0024] When it is determined that the operating information of the grab bucket or the positional relationship between the grab bucket and the object in the trajectory is in a defined dangerous state, the control means may perform control to change the trajectory according to the positional relationship.

[0025] The AI further analyzes the sensing data and outputs soil position information indicating the position where the soil is loaded in the belly of the dump truck, and the control means may control the crane according to the soil position information.

[0026] The object of the control performed by the control means may be at least any one of the slewing amount or slewing speed of the crane, the payout or take-up amount of the wire suspending the grab bucket, and the opening and closing of the grab bucket.

[0027] Another aspect of the present disclosure is a control method for controlling a grab dredging device for performing grab dredging work of scooping up soil from the bottom of a water area surrounded by a pollution prevention frame using a crane installed on a ship for underwater dredging and a grab bucket suspended by the crane, and loading the soil onto a dump truck alongside the ship. The method includes obtaining mechanical information including at least one of the position, height, displacement, speed, rotation angle, angular velocity, and frequency of at least one of the crane, the grab bucket, and an object in the grab dredging work using sensing data obtained from remote sensing means attached to the crane and an AI for analyzing the sensing data, and a step of controlling the crane or the grab bucket according to the mechanical information.

[0028] A further aspect of this disclosure provides a program for a computer that controls a grab dredging device for performing grab dredging work, which involves using a crane installed on a ship for seabed dredging and a grab bucket suspended by the crane to grab mud from the seabed of an area enclosed by a pollution control frame and load the mud onto a soil carrier moored to the ship, to perform the steps of: acquiring dynamic information including position, height, displacement, velocity, rotation angle, angular velocity, and at least one of the objects in the grab dredging work, using sensing data obtained from remote sensing means attached to the crane and AI that analyzes the sensing data; and controlling the crane or the grab bucket in accordance with the dynamic information. [Effects of the Invention]

[0029] According to the present invention, grab dredging work can be automated in response to changes in the surrounding sea conditions, such as fluctuations in tidal levels. [Brief explanation of the drawing]

[0030] [Figure 1] A diagram showing an overview of a dredging control system 1 according to one embodiment. [Figure 2] A diagram illustrating the functional configuration of the dredging control system 1. [Figure 3] A diagram illustrating the structure of crane 30. [Figure 4] A diagram illustrating the structure of the grab bucket 40. [Figure 5] A diagram illustrating the hardware configuration of the control device of the dredging apparatus 10. [Figure 6] A diagram illustrating the representation of position in the dredging control system 1. [Figure 7] A diagram illustrating the representation of position in the dredging control system 1. [Figure 8] A diagram illustrating an annotated image. [Figure 9] A flowchart illustrating the position detection process for the pollution control frame 50. [Figure 10] A diagram illustrating the data at each step of the position detection process. [Figure 11] A flowchart illustrating the coaming height detection process for soil transport vessel 60. [Figure 12] A flowchart illustrating the rotation detection process for the grab bucket 40. [Figure 13] A diagram illustrating the aspect ratio of the grab bucket 40. [Figure 14] A diagram illustrating the area of ​​the grab bucket 40 that will be detected. [Figure 15] A flowchart illustrating the vibration detection process for grab bucket 40. [Figure 16] A diagram illustrating the time evolution of the central 3D coordinate. [Figure 17] A flowchart illustrating the process for detecting the trajectory of the grab bucket 40. [Figure 18] A flowchart illustrating the process for detecting the location of soil discharge. [Modes for carrying out the invention]

[0031] 1. Structure Figure 1 is a diagram illustrating an overview of a dredging control system 1 according to one embodiment. The dredging control system 1 comprises a dredging device 10, a dredging vessel 20, a control device 25, a pollution control frame 50, and a soil transport vessel 60. The dredging vessel 20 is a vessel for seabed dredging. The dredging device 10 is installed on the dredging vessel 20. The dredging device 10 comprises a control device 25, a crane 30, and a grab bucket 40. The control device 25 controls the operation of the crane 30 and the grab bucket 40. The crane 30 suspends the grab bucket 40 with a wire. The dredging device 10 uses the grab bucket 40 to grab mud from the seabed in the area enclosed by the pollution control frame 50. The pollution control frame 50 is a structure with a pollution control membrane in the water to prevent pollution from mud and pollutants generated during dredging operations using a grab bucket from spreading to the surrounding waters. The dredging device 10 loads the grabbed mud onto a soil transport vessel 60 using a crane. The soil transport vessel 60 is a ship that transports mud. These operations are called grab-type dredging operations.

[0032] The dredging control system 1 utilizes AI to control at least one of the crane 30 and the grab bucket 40.

[0033] Figure 2 illustrates the control function configuration of the control device 25 in the dredging device 10. The dredging device 10 has a control device 25 which includes a remote sensing means 19, an acquisition means 11, an AI 12, a control means 13, and a positioning means 14 to control the operation of the crane 30 and the grab bucket 40. The remote sensing means 19 senses the state of the structure of the dredging control system 1 and outputs data indicating the sensing result (hereinafter referred to as "sensing data"). The structure of the dredging control system 1 is a concept that includes the crane 30, the grab bucket 40, the dredger 20, the pollution control frame 50, and the soil transport vessel 60. The remote sensing means 19 senses at least a part of these structures. The remote sensing means 19 is attached to the crane 30. AI12 is installed in an information processing device located in the driver's cab of the crane 30 or the dredger 20, analyzes sensing data, and outputs dynamic information related to the dredging control system 1. The dynamic information related to the dredging control system 1 consists of mechanical parameters relating to the crane 30, the grab bucket 40, and at least one of the objects to be dredged using the grab method, and includes at least one of the following: position, height, displacement, velocity, rotation angle, angular velocity, and frequency. The control means 13 controls the crane 30 or the grab bucket 40 according to this dynamic information. The control means 13 controls at least one of the following: the amount of rotation or rotation speed of the crane 30, the amount of wire unwinding or winding up that suspends the grab bucket 40, and the opening and closing of the grab bucket 40.

[0034] The remote sensing means 19 specifically includes at least one of the following (a) to (d): (a) RGB camera (b) Depth sensor (or depth camera) (c) Stereo Camera (d) LiDAR An RGB camera is a camera that records an object as an image using the color components R (red), G (green), and B (blue). A depth sensor is a sensor that measures the distance (or depth) to an object. It is preferable to use an RGB depth sensor. Alternatively, a TOF (Time of Flight) camera equipped with a TOF sensor, a type of depth sensor, may be used, and it is even more preferable to use a 3D TOF camera. Three-dimensional spatial coordinates can be derived from the camera as the starting point, and by knowing the XYZ coordinates, it is possible to collect information such as the size and volume of the object, as well as distance and position information from the TOF camera. It is also desirable to apply pseudo-colorization to the image from the TOF camera. The data output by an RGB depth sensor or a camera equipped with a depth sensor is called RGBD data. A stereo camera is a system that uses two or more cameras to acquire depth information of an object. Like the human eye, a stereo camera can obtain information about depth or three-dimensional structure by photographing the same object from different viewpoints. The data output by a stereo camera is called 3D image data. LiDAR (Light Detection and Ranging) is a system that uses laser light to measure the distance and shape of an object and acquire three-dimensional information. LiDAR can obtain highly accurate distance data by measuring the reflection time of the laser light. Of these, a depth sensor (or depth camera), stereo camera, or LiDAR can obtain distance data to multiple points on the surface of an object. This data is a collection of three-dimensional positional information and is called point cloud data. The remote sensing means 19 obtains image data and point cloud data as sensing data using these sensors. The AI ​​12 combines this data with image processing technology to obtain dynamic information related to the dredging control system 1.

[0035] In AI12, the sensing data obtained by the remote sensing means 19 that is the target of analysis, i.e., processing, is called target data. In this example, the target data includes at least one of image data and point cloud data. If the target data includes image data, this image data is called target image data, and the image shown by the target image data is called the target image. If the target data is point cloud data, this point cloud data is called target point cloud data, and the point cloud shown by the target point cloud data is called the target point cloud. Either image data or point cloud data may be used individually, or both types of data may be used together as target data. AI12 is a software element that has a machine learning model and performs analysis using that machine learning model. Here, AI12 is a software element that includes not only the machine learning model itself, but also functions for preprocessing data in order to input data into the machine learning model, and functions for processing the data output from the machine learning model.

[0036] In one example, the remote sensing means 19 includes an RGB camera and a LiDAR. The RGB camera digitizes the captured video and transmits it to the AI ​​12 in real time. The LiDAR converts the data into point cloud data, including the distance to the measured object, and transmits it to the AI ​​12 in real time. The positional relationship between the RGB camera and the LiDAR is calibrated in advance, and the superposition relationship between points in the point cloud data and pixels in the image data is known.

[0037] The positioning means 14 measures its own position. The positioning means 14 includes a GNSS (Global Navigation Satellite System) receiver. GNSS is a general term for artificial satellite systems that provide positional information on Earth. The acquisition means 11 acquires positional information from the positioning means 14. The AI ​​12 and control means 13 can perform processing using this positional information.

[0038] Figure 3 illustrates the structure of the crane 30. The crane 30 is installed on the dredger 20. The crane 30 has a slewing base 31 and a jib 32. The slewing base 31 slewing horizontally relative to the dredger 20. The jib 32 suspends a grab bucket 40. In detail, a wire 33 is extended from the tip of the jib 32, and the grab bucket 40 is suspended from the wire 33. The crane 30 can raise and lower the wire 33, and the height from which the grab bucket 40 is suspended can be adjusted.

[0039] Figure 4 illustrates the structure of a grab bucket 40. The grab bucket 40 has an upper frame 41, a lower frame 42, support arms 44, a top plate 45, side plates 46, and a rotating shaft 47. The top plate 45 and side plates 46 are attached to the upper frame 41. A wire 33 is attached to the top plate 45. The rotating shaft 47 passes through the side plates 46, and one end of the support arm 44 is rotatably attached to it. The lower frame 42 is movable up and down relative to the upper frame 41 by a wire 34 for opening and closing operations (note that the wire 33 and wire 34 are shown overlapping on the upper side of the upper frame 41). Part of the shell 43 is rotatably supported on the lower frame 42, and another part is rotatably supported on the support arms 44. When the lower frame 42 moves down relative to the upper frame 41, the shell 43 opens, and when it moves up, the shell 43 closes.

[0040] Figure 5 illustrates the hardware configuration of the control device 25. This control device is a computer device having a CPU 101, memory 102, storage 103, input device 104, and output device 105. The CPU 101 is a processing unit that performs various processes according to a program. The memory 102 is a main memory that functions as a work area when the CPU 101 executes a program. The storage 103 is a non-volatile auxiliary storage device that stores various data and programs. The input device 104 is a device for inputting instructions or information to the control device. The input device 104 includes, for example, at least one of a keypad, buttons, touchscreen, keyboard, microphone, and receiver. The output device 105 is a device for outputting information to a human or other device. The output device 105 includes, for example, at least one of a display, speaker, and transmitter.

[0041] In this example, the data stored in the storage 103 includes a program (hereinafter referred to as the "control program") that causes the computer to function as a control device in the dredging control system 1. When the CPU 101 is executing the control program, the CPU 101 is an example of the acquisition means 11, AI 12, and control means 13.

[0042] 2. Preparation The operation of the dredging control system 1 is classified into several processes depending on the object being controlled in grab-type dredging work. Before describing each process below, the elemental technologies will be explained.

[0043] (1) Coordinate system First, we will explain the representation of position in the dredging control system 1, i.e., the coordinate system.

[0044] Figure 6 illustrates the representation of position in the dredging control system 1. Here, the crane 30, grab bucket 40, pollution control frame 50, and soil transport vessel 60 are simply illustrated. First, to represent the position of the elements in the dredging control system 1, we consider a coordinate system (referred to as the "LiDAR reference coordinate system") where the center of the rotating body of the crane 30 is the origin in the xy-plane, and the height of the LiDAR sensor (i.e., remote sensing means 19) is the origin of the z-axis. The z-axis is the vertical coordinate axis. The position of the center of the rotating body is (0, 0, 0). The position of the LiDAR is:

number

number

[0045] Figure 7 is a diagram illustrating the representation of position in the dredging control system 1. Next, consider a coordinate system with the GNSS antenna (i.e., positioning means 14) as the origin (hereinafter referred to as the "GNSS reference coordinate system"). When the position of the center of the rotating body as seen from the GNSS antenna is (a, b, c), the position of the object, according to the GNSS reference coordinate system,

number

number

[0046] (2) Object detection Object detection is a technique that detects the area in an image where an object is present and identifies the type of object. Machine learning models are used for object detection. For example, grab buckets are the objects to be detected. This machine learning model is a pre-trained model that has been trained using annotated images of various grab buckets as training data.

[0047] Figure 8 illustrates an annotated image. This image includes annotation 1001. In this example, annotation 1001 is a rectangular frame surrounding the grab bucket (more specifically, a frame that circumscribes the grab bucket and has sides parallel to the image frame).

[0048] (3) Edge detection Edge detection is a technique for detecting edges, or contours, in an image. Edges are detected, for example, based on discontinuities in brightness within the image. Examples of edge detection methods include the Canny method, the Sobel filter method, or the Laplacian method. The Canny method is an algorithm that performs the following steps in order: noise reduction, calculation of gradient intensity and direction, suppression of non-maximal values, thresholding, and edge tracking. Noise reduction is the process of removing noise from the image using filters such as a Gaussian filter. Calculation of gradient intensity and direction is the process of calculating the gradient intensity (i.e., edge strength) and gradient direction for each pixel using filters such as a Sobel filter. Gradient intensity indicates how prominent the edge is. Suppression of non-maximal values ​​is the process of suppressing pixels where the gradient intensity is not at the local maximum value, leaving only thin edges. This improves edge accuracy. Thresholding is the process of classifying pixels into strong edges, weak edges, and non-edges by setting, for example, two thresholds. Strong edges are reliably recognized as edges, while weak edges are recognized as edges only when adjacent to strong edges. Edge tracking is a process that connects weak edge pixels to strong edge pixels to form continuous edges. This allows for the detection of seamless edges. The Sobel filter method is an edge detection method that uses the Sobel filter. The Laplacian method is a method that detects edges using the second derivative of an image. The Laplacian filter is used to enhance the edges of an image. This technique uses the Laplace operator to detect abrupt changes in brightness within an image.

[0049] (4) Color identification Color identification is a technique for extracting areas within an image that fall within a specific hue range. Color identification involves analyzing the color of each pixel in an image and determining whether that color falls within a specific range.

[0050] (5) Point cloud matching Point cloud matching is a technique that compares two or more point cloud datasets to find corresponding points (feature points) or structures. In this case, point cloud data of an object (e.g., a grab bucket) is acquired in advance, and this point cloud data is compared with the target point cloud data to detect the object in the target point cloud data. Point cloud matching is particularly effective in identifying objects with distinctive shapes.

[0051] (6) Object extraction by distance Distance-based object extraction is a technique that extracts objects based on distance (or depth) data from RGBD or LiDAR data. Here, points within a specific range of distance data are identified, and objects are extracted using the identified point cloud. For example, by clustering the point cloud using distance data and removing points outside the specified range (e.g., points at infinity), it is possible to extract the target object.

[0052] Next, we will explain the processing methods used in grab-type dredging operations, depending on the type of material being dredged.

[0053] 3. Detection of the position of the pollution control frame When working near the pollution control frame 50, the grab bucket 40 may be pressed against the pollution control frame 50 from the side, pushing the frame in. This may cause the position of the pollution control frame 50 to shift. Alternatively, the pollution control frame 50 may move due to tidal currents even without pressing the grab bucket 40 against it. To perform more precise control, it is necessary to understand the changes in the position of the pollution control frame 50 in real time. Therefore, the control device 25 detects the position of the pollution control frame 50. In other words, in this process, the object of analysis by AI 12 is the pollution control frame 50, and the mechanical information output by AI 12 is displacement information. Displacement information is information that indicates the displacement of the pollution control frame 50.

[0054] Figure 9 is a flowchart illustrating the position detection process for the pollution control frame 50. In step S11, AI 12 acquires target data. AI 12 acquires point cloud data from the remote sensing means 19. This point cloud data is the target point cloud data. AI 12 also acquires video data from the remote sensing means 19. AI 12 decodes the acquired video data to obtain image data for each frame. This image data is the target image data. AI 12 may use all frames included in the acquired video data as the target image data, or it may use only some frames of the video data (for example, one frame every 30 frames) as the target image data.

[0055] In step S12, AI12 filters the point cloud data to include a portion of the point clouds that meet the distance criteria. Specifically, AI12 extracts point clouds that are within a predetermined distance range relative to the remote sensing means 19. This "predetermined distance range" is set, for example, by the user, as the range that includes the expected distance from the remote sensing means 19 to the pollution control frame 50.

[0056] In step S13, AI12 identifies the point cloud representing the pollution control frame 50 from the extracted point cloud by clustering. For example, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is used as the clustering method. DBSCAN is a method that forms clusters based on data density, identifying low-density areas as noise. Here, AI12 identifies the point cloud belonging to the cluster with the largest number of data points among the classified clusters as the point cloud representing the pollution control frame 50.

[0057] In step S14, AI12 extracts a straight line (more precisely, a line segment) from the point cloud representing the pollution prevention frame 50. This straight line is an example of a feature portion of the pollution prevention frame 50. Edge detection or color identification is used to detect the straight line. If the point cloud data contains color information, color identification is performed using this color information. If the point cloud data does not contain color information, AI12 integrates the image data and the point cloud data and adds color information to the point cloud data. Color identification identifies the color corresponding to the frame of the pollution prevention frame 50. This process is an example of an extraction means that performs predetermined image processing on the target image data to extract the feature portion of the pollution prevention frame 50. Edge detection or color identification is an example of predetermined image processing.

[0058] In step S15, AI12 calculates the position of the corners of the pollution control frame 50 in the point cloud data. The position of the corners of the pollution control frame 50 is calculated as the coordinates of the intersection points of the straight lines detected in step S14.

[0059] In step S16, AI12 estimates the position of the pollution control frame 50. AI12 can access the dimensional data of the pollution control frame 50 and estimates the position of the pollution control frame 50 by referring to this dimensional data. The dimensional data refers to data that shows the size of the pollution control frame 50 (specifically, the length × width dimensions) when viewed from above (i.e., from a position looking down at the water surface). The position of the pollution control frame 50 refers, for example, to the positions of the four corners of the pollution control frame 50 in three-dimensional space. Note that this process is an example of a position detection means that detects the position of the pollution control frame 50 using feature parts obtained by image processing of the target image data.

[0060] Figure 10 illustrates the data at each step of the position detection process. Figure 10(A) illustrates the target image in the target data. In addition to the image data showing this image, point cloud data is also acquired as the target data. Figure 10(B) shows the point cloud extracted from the target point cloud according to distance. Figure 10(C) shows the point cloud extracted as the pollution control frame 50. Figures 10(C) to (E) show the position of the point cloud in a 2D coordinate system as viewed from above with respect to the pollution control frame 50. Figure 10(D) shows the straight lines detected in the point cloud. In this example, two straight lines, L1 and L2, are detected. Figure 10(E) shows the corners detected in the point cloud. In this example, corner C1 is detected. The pollution control frame 50 is rectangular when viewed from above, so there should be four corners, but the three corners other than corner C1 are outside the shooting range and are therefore not detected in the point cloud data. Figure 10(F) shows the estimated position of the pollution control frame 50. Data indicating the size of the pollution control frame 50 is provided, and since the shape of the pollution control frame is generally square, the position of the pollution control frame 50 can be estimated from the position of the detected corners. Here, estimating the position of the pollution control frame 50 means estimating the positions of the characteristic parts of the pollution control frame 50 (for example, the four corners and four sides). In addition, AI12 estimates the position of the pollution control frame 50 as described above, and also estimates the positional changes of the pollution control frame 50 caused by the grab bucket placed inside the estimated pollution control frame in near real time. In the example in Figure 10(F), an image is shown in which the positions of the pollution control frame 50 and the grab bucket 40 are superimposed on the target image.

[0061] If the shape of the pollution control frame 50 is rectangular rather than square, for example, the control device 25 controls, for example, the crane 30 and the remote sensing means 19 to capture the entire view of the pollution control frame 50 in its field of view before dredging begins. At this time, the remote sensing means 19 photographs the entire view of the pollution control frame 50. The control device 25 determines the positional relationship between the long side and the short side from this image data. Alternatively, the positional relationship between the dredger 20 and the long side and short side of the pollution control frame 50 may be defined in advance, and the control device 25 may determine from this positional relationship whether the straight line detected in Figure 10(D) corresponds to the long side or the short side.

[0062] AI12 further calculates the displacement of the pollution control frame 50 from the time change of position calculated in step S16. This process is an example of a calculation means that calculates the displacement of the pollution control frame 50 from the time change of position detected by the position detection means.

[0063] The control means 13 controls the crane 30 and the grab bucket 40 by referring to the position of the pollution prevention frame 50 estimated in this way. If the position of the pollution prevention frame 50 is estimated in real time, the displacement of the pollution prevention frame 50 can be detected from its change over time. For example, when the grab bucket 40 is pushed toward the pollution prevention frame 50 and it is detected that the pollution prevention frame 50 has been displaced by a predetermined amount, the AI ​​12 determines that the grab bucket 40 has touched the pollution prevention frame 50. Once the grab bucket 40 has touched the pollution prevention frame 50, and the pollution prevention frame 50 has been pushed further to a predetermined position, and it is determined that the grab bucket 40 is positioned near the water surface of the area to be dredged, the control means 13 controls the crane 30 to submerge the grab bucket 40 in the water.

[0064] Here, we have described the process by which AI12 estimates the position of the pollution control frame 50 from point cloud data, but AI12 may also estimate the position of the pollution control frame 50 from image data. In this case, AI12 detects straight lines from the target image by edge detection or color identification, calculates the position of the corners from the detected straight lines, and estimates the position of the pollution control frame 50 from the position of the corners.

[0065] 4. Detection of coaming height of soil transport vessels The dredging device 10 loads the dredged mud from the seabed onto the dredger 60, but the coaming height of the dredger 60 varies greatly depending on the amount of soil loaded or the tide level. Hereinafter, the coaming of the dredger 60 will simply be referred to as "coaming". In order to load the excavated mud onto the dredger 60, the grab bucket 40 is raised up before the crane 30 is moved so that the coaming and the grab bucket 40 do not come into contact. At this time, if the position of the grab bucket 40 is too low, the grab bucket 40 will come into contact with the dredger 60. On the other hand, if the position of the grab bucket 40 is too high, unnecessary raising will occur, resulting in a loss of time. In this example, the height of the coaming is detected and the amount of raising of the grab bucket 40 is appropriately controlled. In other words, in this process, the object of analysis by AI 12 is the dredger 60, and the mechanical information output by AI 12 is height information and position information. The height information indicates the height of the coaming. The location information indicates the position of the soil transport vessel 60.

[0066] Figure 11 is a flowchart illustrating the coaming height and position detection process. In step S21, AI12 acquires target data. AI12 acquires point cloud data from remote sensing means 19. This point cloud data is the target point cloud data. AI12 also acquires video data from remote sensing means 19. AI12 decodes the acquired video data to obtain image data for each frame. This image data is the target image data. AI12 may use all frames included in the acquired video data as the target image data, or it may use only some frames of the video data (for example, one frame every 30 frames) as the target image data.

[0067] In step S22, AI12 detects the grab bucket 40 in the target image data. Object detection technology is used to detect the grab bucket 40. AI12 excludes the region detected as the grab bucket 40 from subsequent processing.

[0068] In step S23, AI12 extracts combing candidates from the target image data. Specifically, AI12 performs edge detection and color identification on the target image. The edges detected here are the combing candidates.

[0069] In step S24, AI12 extracts combing in the target image data. Specifically, AI12 detects straight lines in the target image data. AI12 extracts combing from the detected straight lines that satisfy certain conditions (for example, those whose length is greater than or equal to a threshold and that intersect with other straight lines). AI12 identifies the point cloud corresponding to the combing detected in the target image in the target point cloud data. This process is an example of a processing method for extraction means that performs predetermined image processing on point cloud data to extract combing.

[0070] Since the shape of the coaming (when viewed from above) is generally rectangular, similar to the case of the pollution control frame 50, the control device 25 controls, for example, the crane 30 and the remote sensing means 19 to capture the entire view of the coaming in its field of view before dredging begins. At this time, the remote sensing means 19 photographs the entire view of the coaming. The control device 25 determines the positional relationship between the long and short sides from this image data. Alternatively, the positional relationship between the short and long sides of the dredger 20 and the coaming (soil carrier 60) may be defined in advance, and the control device 25 may determine from this positional relationship whether the detected straight line corresponds to the long or short side.

[0071] In step S25, AI12 calculates the coaming height and position. The coaming height is a representative value of the height obtained from the point cloud data. In one example, the coaming height is a representative value of the coaming height obtained from the point cloud data. The representative value of the coaming height is, for example, the maximum value of the height (Z direction) distribution in the xy plane. Alternatively, the representative value of the coaming height may be the average value of the height (Z direction) distribution in the x or y direction. Furthermore, AI12 calculates the position of the coaming. The calculation of the position of the coaming is performed by the same process as the calculation of the position of the pollution control frame 50. These processes are examples of processing as a calculation means for calculating the height and position of the coaming from the point cloud data corresponding to the coaming.

[0072] The control means 13 controls the crane 30 and the grab bucket 40 by referring to the coaming height thus estimated. For example, the control means 13 raises the grab bucket 40 to a value that is the calculated coaming height plus a predetermined margin.

[0073] 5. Grab bucket status detection The grab bucket 40 may exhibit undesirable movements due to environmental influences or operational history. For example, sudden increases in wave size or strong winds may cause the grab bucket 40 to sway or rotate violently. Discharging soil in such conditions could lead to the grab bucket 40 coming into contact with the coaming, which is dangerous. Therefore, the state of the grab bucket 40 is detected. In other words, in this process, the object of analysis by AI12 is the grab bucket 40, and the dynamic information output by AI12 is the oscillation information of the grab bucket 40. The oscillation information indicates the rotation or swaying of the grab bucket 40.

[0074] (1) Rotation detection Figure 12 is a flowchart illustrating the rotation detection process of the grab bucket 40. In step S31, AI 12 acquires target data. AI 12 acquires point cloud data from the remote sensing means 19. This point cloud data is the target point cloud data. AI 12 also acquires video data from the remote sensing means 19. AI 12 decodes the acquired video data to obtain image data for each frame. This image data is the target image data. AI 12 may use all frames included in the acquired video data as the target image data, or it may use only some frames of the video data (for example, one frame every 30 frames) as the target image data.

[0075] In step S32, AI12 detects the grab bucket 40 in the target image data. Color identification and object detection techniques are used to detect the grab bucket 40. In step S33, AI12 calculates the aspect ratio of the detected grab bucket 40 region. This process is an example of a calculation means that performs predetermined image processing on the image data and calculates the aspect ratio of the grab bucket in the image data.

[0076] Figure 13 illustrates the aspect ratio of the grab bucket 40's region. Because the grab bucket 40 has depth (i.e., thickness), when viewed directly from the front, the width of the grab bucket 40's region is at its minimum (Do) (Figure 13(A)). However, when the grab bucket 40 rotates from this state, the width of the grab bucket 40's region in the target image (Dd) increases (Figure 13(B)). On the other hand, the height of the grab bucket 40 does not change with rotation, so rotation can be detected by monitoring the time change in the aspect ratio of the grab bucket 40's region. Furthermore, by considering the time change in the region's width, it is possible to detect whether the rotation is clockwise or counterclockwise.

[0077] Refer to Figure 12 again. In step S34, AI12 determines whether the grab bucket 40 is rotating based on its aspect ratio. Specifically, AI12 determines that the grab bucket 40 is rotating if the rate of increase in the aspect ratio exceeds a threshold. Here, the rate of increase is defined, for example, by the following equation (5).

number

[0078] Figure 14 illustrates the area of ​​the grab bucket 40 that is detected. Note that the grab bucket 40 is simplified in Figure 14 for the sake of simplicity. In this example, the threshold for the rate of increase is 0.3. Figure 14 shows three consecutive frames as an example. In frames 14(A) and (B), the rate of increase is less than the threshold, and it is determined that there is no rotation. In frame 14(C), the rate of increase exceeds the threshold, and it is determined that the grab bucket 40 is rotating.

[0079] (2) Shake detection Figure 15 is a flowchart illustrating the shaking detection process of the grab bucket 40. In step S41, AI 12 acquires target data. AI 12 acquires point cloud data from the remote sensing means 19. This point cloud data is the target point cloud data. AI 12 also acquires video data from the remote sensing means 19. AI 12 decodes the acquired video data to obtain image data for each frame. This image data is the target image data. AI 12 may use all frames included in the acquired video data as the target image data, or it may use only some frames of the video data (for example, one frame every 30 frames) as the target image data.

[0080] In step S42, AI12 detects the grab bucket 40 in the target image data. Color identification and object detection techniques are used to detect the grab bucket 40. In step S43, AI12 calculates the position (2D coordinates in the target image) of the center (or centroid) of the detected grab bucket 40 region. In step S44, AI12 recalculates the position (3D coordinates) of the center of the grab bucket 40 in the target point cloud data that has changed over time since step 43. By repeatedly performing steps 43 and 44, in step S45, AI12 determines whether the grab bucket 40 is swaying based on the time change in the center position.

[0081] Figure 16 illustrates the time evolution of the three-dimensional coordinates of the center. In Figure 16, the horizontal axis represents time, and the vertical axis represents the displacement of the center's position coordinates. The displacement Dc of the center can be calculated from the displacements Δx, Δy, and Δz of the x, y, and z coordinates, for example, by the following equation (6).

number

[0082] Furthermore, the determination of shaking is not limited to equation (6), and shaking may also be determined from the time change of only one or two components of the x, y, and z coordinates. For example, the z coordinate will always fluctuate when the grab bucket 40 is hoisted up or unloaded, so there is a risk that shaking that is not intended here may be mistakenly identified as shaking. Therefore, in one example, AI12 employs different shaking detection methods depending on whether the hoisting or unloading operation is being performed or not.

[0083] In this example, AI12 acquires information indicating the operating status of the grab bucket 40, specifically whether or not a hoisting or unwinding operation is being performed. When a hoisting or unwinding operation is being performed, AI12 determines whether the grab bucket 40 is swaying using the displacements of the x and y coordinates, regardless of the z coordinate. On the other hand, when a hoisting or unwinding operation is not being performed, AI12 determines whether the grab bucket 40 is swaying using the displacements of the z coordinate, in addition to the x and y coordinates.

[0084] (3) Summary and control As described above, AI12 can detect the rotation and sway of the grab bucket 40. AI12 may perform both rotation detection and sway detection on the same target image, or it may perform only one of the two processes, rotation detection or sway detection, on a single target image. In one example, AI12 uses different detection targets depending on the operation of the crane 30 and the grab bucket 40. For example, while the crane 30 is rotating, AI12 detects both rotation and sway. When the crane 30 has stopped and the grab bucket 40 is about to begin dredging (i.e., when the grab bucket 40 is moving toward the pollution control frame 50), AI12 detects only rotation and does not detect sway.

[0085] In one example, AI12 does not release the grab bucket 40 while rotation or swaying is detected. Alternatively, AI12 takes preventative measures against rotation or swaying when the rotation or swaying becomes excessive (for example, when the amount of rotation or swaying exceeds a set threshold). Preventative measures against rotation or swaying include, for example, reducing the slewing speed of the crane 30. AI12 reduces the slewing speed when the grab bucket 40 is rotating or swaying. Reducing the slewing speed may include stopping the slewing.

[0086] Here, we have described the process by which AI12 detects the rotation of the grab bucket 40 from image data, but AI12 may also detect the rotation of the grab bucket 40 from point cloud data. In this case, AI12 filters the target point cloud data to point clouds that satisfy the distance condition, and detects the position and orientation of the grab bucket 40 from the point cloud extracted by the filtering process. Furthermore, AI12 detects the rotation and direction of rotation of the grab bucket 40 from the time change in the orientation of the grab bucket 40.

[0087] 6. Detection of grab bucket trajectory and soil discharge location In the dredging control system 1, the crane 30 and grab bucket 40 may operate according to settings manually entered by the worker. However, due to input errors or sensor malfunctions, the grab bucket 40 may follow a different trajectory than intended and come into contact with an object, such as another device, structure, or worker. Therefore, in this example, the AI ​​12 detects the trajectory or soil discharge position of the grab bucket 40, and if it is in a defined dangerous state, i.e., a state different from what was expected, it takes action to change the trajectory.

[0088] (1) Trajectory detection of grab bucket 40 Figure 17 is a flowchart illustrating the trajectory detection process of the grab bucket 40. In this process, the object of analysis by AI12 is the grab bucket 40, and the dynamic information output by AI12 is the grab operation information of the grab bucket 40. The grab operation information is information that indicates the trajectory, rotation, or oscillation of the grab bucket 40.

[0089] In step S51, AI12 acquires target data. AI12 acquires point cloud data from remote sensing means 19. This point cloud data is the target point cloud data. AI12 also acquires video data from remote sensing means 19. AI12 decodes the acquired video data to obtain image data for each frame. This image data is the target image data. AI12 may use all frames included in the acquired video data as the target image data, or it may use only some frames of the video data (for example, one frame every 30 frames) as the target image data.

[0090] In step S52, AI12 detects the grab bucket 40 and objects (e.g., other equipment, structures, or workers) in the target image data. Object detection techniques are used to detect the grab bucket 40. Object detection techniques are also used to detect other objects, including workers, in the same way as for the grab bucket 40. In step S53, AI12 calculates the position of the center of the detected grab bucket 40 region (2D coordinates in the target image). In step S54, AI12 calculates the position of the center of the grab bucket 40 (3D coordinates) in the target point cloud data.

[0091] In step S55, AI12 determines whether the trajectory of the grab bucket 40 is in a defined hazardous state. A defined hazardous state is, for example, one of the following two: (a) If the current trajectory is maintained, there is a possibility that an object exists at the end of the predicted trajectory and that the object will come into contact with it. (i) The trajectory so far has deviated from the trajectory that AI12 had anticipated in its control, and that deviation has exceeded a threshold.

[0092] If the trajectory is determined to be in a dangerous state, the control means 13 performs a process to change the trajectory of the grab bucket 40. The process to change the trajectory may be, for example, stopping the movement of the grab bucket 40, reducing the movement speed of the grab bucket 40, rotating the crane 30, or hoisting up / out the grab bucket 40. In this example, if the trajectory of the grab bucket 40 differs from what is expected, the trajectory is changed to ensure safety.

[0093] If the trajectory is determined to be in a dangerous state, the control means 13 will not maintain the control (or operation) at that time, but will change it. The change in control may be, for example, stopping the control (e.g., stopping winding / extending, slewing, or opening / closing) or slowing down (e.g., reducing the speed in winding / extending, slewing, or opening / closing).

[0094] Here, we have described an example where the grab operation information is information indicating the trajectory of the grab bucket 40. However, the grab operation information may include information indicating rotation or oscillation in addition to, or instead of, information indicating the trajectory of the grab bucket 40. The detection of rotation or oscillation of the grab bucket 40 is as described in the chapter "Detection of the state of the grab bucket," and can be detected by repeatedly performing the processes of steps 33 and 34 or steps 43 and 44 as steps 53 and 54. In this case, AI 12 predicts the future trajectory of the grab bucket 40, including its rotation or oscillation. AI 12 determines whether the predicted trajectory, including rotation or oscillation, is in a defined dangerous state.

[0095] (2) Detection of soil discharge location Figure 18 is a flowchart illustrating the soil discharge location detection process. In this process, the object analyzed by AI12 is the mud loaded onto the hull of the soil transport vessel 60, and the mechanical information output by AI12 is mud location information indicating the location where the mud fell.

[0096] In step S61, AI12 acquires target data. AI12 acquires point cloud data from remote sensing means 19. This point cloud data is the target point cloud data. AI12 also acquires video data from remote sensing means 19. AI12 decodes the acquired video data to obtain image data for each frame. This image data is the target image data. AI12 may use all frames included in the acquired video data as the target image data, or it may use only some frames of the video data (for example, one frame every 30 frames) as the target image data.

[0097] In step S62, AI12 detects the mud being dumped from the hull of the dredger 60 in the target image data. For example, color identification is used to detect the mud. When detecting the mud, AI12 refers to the position of the coaming as the hull of the dredger 60. Specifically, AI12 first identifies the position of the coaming and then detects the mud being dumped within the coaming. The detection of the coaming position is as described in the chapter "Detection of Coaming Height of Dredger". In step S63, AI12 calculates the position of the center of the detected mud area (2D coordinates in the target image). In step S64, AI12 calculates the position of the center of the mud (3D coordinates) in the target point cloud data.

[0098] In step S65, the control means 13 determines the position where the grab bucket 40 will discharge soil into the hull of the soil carrier 60 in subsequent operations, based on the calculated position of the discharged soil. Specifically, it controls the crane 30 and the grab bucket 40 to discharge soil at a position that does not overlap with a position where a large amount of soil has already been loaded. The control means 13 may also calculate the trim and heel of the soil carrier 60 from the identified soil carrier coaming height and position (angle), and determine the discharge position based on the trim and heel of the soil carrier or one of them. Specifically, the control means 13 may control the discharge to a position that minimizes the tilt of the soil carrier 60.

[0099] When detecting the trajectory of the grab bucket 40, the AI ​​12 may also consider the depth (or height) of the grab bucket 40. That is, the AI ​​12 limits the objects to be extracted from image data, etc., according to the tide level and the depth of the grab bucket 40. In this example, the AI ​​12 acquires information indicating the depth of the grab bucket 40 (or the amount of wire 33 wound up or unwound). For example, if data is obtained indicating a tide level of +1.0m and a grab bucket 40 depth of +2.0m, the grab bucket 40 should be visible in the image, so the AI ​​12 extracts the grab bucket 40 and other structures (e.g., the pollution control frame 50) as objects from the image. Alternatively, if data is obtained showing a tide level of +1.0m and a depth of -1.0m for the grab bucket 40, the grab bucket 40 should be below the water surface and not visible in the image. In this case, AI12 will extract only other structures (e.g., the pollution control frame 50) from the image as the target object.

[0100] 7. Variations The present invention is not limited to the embodiments described above, and various modifications are possible. Several modifications are described below, and at least some of the matters described below may be applied in combination with at least some of the matters described in the embodiments described above.

[0101] In the embodiments, the object and its processing were described individually, but they may be used in combination. For example, two processes, detection of coaming height and detection of grab state, may be performed in parallel for a single object data.

[0102] The objects of control performed by the control means 13 are not limited to those exemplified in the embodiment. In addition to, or instead of, those exemplified in the embodiment, the elevation angle of the boom in the crane 30 may be controlled.

[0103] The combination of target data and processing is not limited to those exemplified in the embodiments. In the example described in the embodiments, where an object is extracted from image data, an object may be extracted from point cloud data. Conversely, in the example described in the embodiments, where an object is extracted from point cloud data, an object may be extracted from image data.

[0104] Furthermore, in the above-described embodiment, an RGB camera and LiDAR were used as the remote sensing means 19, and point cloud data and image data were used as the target data for sensing. However, a depth sensor (depth camera) or stereo camera may be used as the remote sensing means 19, or an RGB-depth sensor may be used. In that case, instead of point cloud data and image data, D data, RGBD data, or 3D image data can be used as the target data for sensing to calculate the depth to the object.

[0105] As illustrated in the chapter "Detection of Grab Bucket Trajectory and Soil Discharge Position," AI12 may perform various analyses by referring to information about the dredging device 10, such as the depth of the grab bucket 40 or operation instruction information. This information allows AI12 to narrow down the objects to be detected. For example, if the depth of the grab bucket 40 is lower than the sea surface, AI12 can exclude the grab bucket 40 from the objects to be detected for position. Alternatively, if the grab bucket 40 is in motion, AI12 can exclude the grab bucket 40 from the objects to be detected for movement. Furthermore, if the crane 30 is facing in the opposite direction from the pollution control frame 50, AI12 can exclude the pollution control frame 50 from the objects to be detected for position.

[0106] The control based on the mechanical information output from AI12 is not limited to those exemplified in the embodiment. The control means 13 may perform any control as long as it utilizes the mechanical information provided by AI12. Alternatively, the dredging control system 1 may simply present the mechanical information to the worker, and the crane 30 and grab bucket 40 may be operated manually by the worker while confirming the mechanical information. Furthermore, for safety reasons, the system may be controlled to activate only the functions for determining whether the above-defined dangerous conditions are present and for determining the position to discharge the soil into the hull of the soil transport vessel 60.

[0107] The physical location of the control device 25 on which AI12 is implemented is not limited to the examples of embodiments. For example, in the examples of embodiments, an example was described in which the control device 25 on which AI12 is implemented is installed in the driver's cab of a crane 30 or a dredger 20. However, the control device 25 may be installed, for example, on the bridge. In the embodiments, an example was described in which the control device 25 is a standalone device and is not connected to a computer network. However, the control device 25 may have communication functions and be connectable to a computer network. In this case, at least a part of the functions of AI12 may be implemented on a server on the network (a so-called cloud).

[0108] The correspondence between functional elements and hardware elements in the dredging control system 1 is not limited to those exemplified in the embodiments. The dredging control system 1 may have any hardware configuration as long as it can implement the required functional elements.

[0109] The programs executed by CPU101, etc., may be provided in a form that can be downloaded via a computer network such as the Internet, or they may be provided recorded on a computer-readable recording medium such as a CD-ROM. [Explanation of symbols]

[0110] 1…Dredging control system, 10…Dredging device, 11…Acquisition means, 12…AI, 13…Control means, 14…Positioning means, 19…Remote sensing means, 20…Dredging vessel, 21…Barge, 25…Control device, 30…Crane, 31…Slewing base, 32…Jib, 33…Wire, 34…Movable guide, 35…Wire, 40…Grab bucket, 41…Upper frame, 42…Lower frame, 43…Shell, 44…Support arm, 45…Top plate, 46…Side plate, 47…Rotating shaft, 50…Pollution prevention frame, 60…Soil transport vessel, 101…CPU, 102…Memory, 103…Storage, 104…Input device, 105…Output device, 341…Wall section, 342…Lifting device, 343…Connecting device, 1001…Annotation

Claims

1. A control system for a grab-type dredging device for performing grab-type dredging work, which involves using a crane installed on a ship for seabed dredging and a grab bucket suspended by the crane to grab mud from the seabed in an area enclosed by a pollution control frame, and loading the mud onto a soil transport vessel moored to the ship, An acquisition means that uses sensing data obtained from a remote sensing means attached to the crane, and AI that analyzes said sensing data, to acquire mechanical information including at least one of the position, height, displacement, velocity, rotation angle, angular velocity, and vibration frequency related to the crane, the grab bucket, and at least one of the objects in the grab-type dredging work, Control means for controlling the crane or the grab bucket according to the aforementioned mechanical information. A control system having

2. The object in question is the pollution prevention frame, The mechanical information includes displacement information indicating the displacement of the pollution control frame. The control system according to claim 1.

3. The remote sensing means includes an RGB camera, The sensing data includes image data of the pollution prevention frame obtained from the RGB camera. The aforementioned AI is An extraction means that performs predetermined image processing on the aforementioned image data and extracts characteristic portions of the pollution prevention frame, Position detection means for detecting the position of the pollution prevention frame using at least the feature portion obtained by the image processing, A calculation means for calculating the displacement from the time change of the position detected by the position detection means. It has, The acquisition means acquires information regarding the displacement of the pollution control frame calculated by the calculation means as the displacement information. The control system according to claim 2.

4. The image processing includes at least one of the following: edge detection processing for detecting edges from the image data, and color identification processing for extracting portions of a specific color from the image data. The control system according to claim 3.

5. The remote sensing means includes LiDAR, The sensing data includes point cloud data of the area around the pollution control frame obtained from the LiDAR. The aforementioned AI is An extraction means that performs predetermined data processing on the point cloud data and extracts the characteristic portion of the pollution control frame, A position detection means for detecting the position of the pollution prevention frame using at least the feature portion obtained by the data processing, A calculation means for calculating the displacement from the time change of the position detected by the position detection means. It has, The acquisition means acquires information regarding the displacement of the pollution control frame calculated by the calculation means as the displacement information. The control system according to claim 2.

6. The aforementioned data processing, A filtering process that filters out point clouds from the aforementioned point cloud data that satisfy the distance condition, A clustering process is performed to extract clusters where points are densely clustered from the aforementioned point cloud data. A corner identification process that identifies the position of the corners of the pollution prevention frame from the point cloud data, and Position estimation process that estimates the position of the pollution control frame from the point cloud data and the dimensional data of the pollution control frame. Includes at least one type of processing The control system according to claim 5.

7. The object in question is the soil transport vessel, The aforementioned dynamic information includes height information indicating the height of the coaming of the soil carrier, and position information indicating its location. The control system according to claim 1.

8. The remote sensing means includes an RGB camera, The sensing data includes the combing image data obtained from the RGB camera. The aforementioned AI is An extraction means for performing predetermined image processing on the image data and extracting the combing, A calculation means for calculating the height and position of the combing extracted from the image data. It has, The acquisition means acquires the information relating to the height and position of the coaming calculated by the calculation means as the height information and the position information. The control system according to claim 7.

9. The image processing includes at least one of the following: edge detection processing for detecting edges from the image data; color identification processing for extracting portions of a specific color from the image data; and line detection processing for detecting straight lines from the image data. The control system according to claim 8.

10. The remote sensing means includes LiDAR, The sensing data includes point cloud data of the combing obtained from the LiDAR. The aforementioned AI is Calculation means for calculating the height and position of the combing from the point cloud data corresponding to the combing among the point cloud data. It has, The acquisition means acquires the information relating to the height and position of the coaming calculated by the calculation means as the height information and the position information. The control system according to claim 7.

11. The object in question is the grab bucket, The mechanical information includes oscillation information indicating the rotation or oscillation of the grab bucket. The control system according to claim 1.

12. The remote sensing means includes an RGB camera, The sensing data includes image data of the grab bucket obtained from the RGB camera. The aforementioned AI is A calculation means that performs predetermined image processing on the image data and calculates the aspect ratio of the grab bucket in the image data, Rotation detection means for detecting the rotation state of the grab bucket using at least the aspect ratio obtained by the image processing, It has, The acquisition means acquires information relating to rotation detected by the rotation detection means as oscillation information. The control system according to claim 11.

13. The image processing includes a color identification process that extracts a portion of a specific color from the image data. The control system according to claim 12.

14. The remote sensing means includes LiDAR, The sensing data includes point cloud data of the area around the grab bucket obtained from the LiDAR. The aforementioned AI is A calculation means that performs predetermined data processing on the point cloud data and calculates the aspect ratio of the grab bucket in the point cloud data, Rotation detection means for detecting the rotation state of the grab bucket using at least the aspect ratio obtained in the data processing, It has, The acquisition means acquires information relating to rotation detected by the rotation detection means as oscillation information. The control system according to claim 11.

15. The data processing includes a filtering process that filters out point cloud data for points whose distance satisfies a certain condition. The control system according to claim 14.

16. The remote sensing means includes an RGB camera, The sensing data includes video data of the grab bucket obtained from the RGB camera. The aforementioned AI is A means for performing predetermined image processing on frames included in the video data and identifying the central position of the grab bucket in the frame, A vibration detection means for detecting the vibration of the grab bucket in accordance with the time change of the central position, It has, The acquisition means acquires information regarding the detected shaking as the shaking information. The control system according to claim 11.

17. The remote sensing means includes LiDAR, The sensing data includes point cloud data of the grab bucket obtained from the LiDAR. The aforementioned AI is A calculation means for calculating the position of the grab bucket according to the time change of the point cloud data, A vibration detection means for detecting the vibration of the grab bucket in accordance with the time change in the position of the grab bucket, It has, The acquisition means acquires information relating to shaking detected by the shaking detection means as shaking information. The control system according to claim 11.

18. The object in question is the grab bucket, The mechanical information includes grab operation information that indicates at least one of the trajectory of the grab bucket, the rotation of the grab bucket, or the swing of the grab bucket. The control system according to claim 1.

19. If the operation information of the grab bucket or the positional relationship between the grab bucket and the object is determined to be in a defined dangerous state along the aforementioned trajectory, the control means performs control to change the trajectory according to that positional relationship. The control system according to claim 18.

20. The AI ​​further analyzes the sensing data and outputs mud location information indicating the location where the mud is loaded on the hull of the mud transport vessel. The control means controls the crane according to the mud position information. The control system according to claim 17.

21. The control performed by the control means is at least one of the following: the amount of rotation or rotation speed of the crane, the amount of wire unwinding or winding up of the wire supporting the grab bucket, and the opening and closing of the grab bucket. The control system according to claim 1.

22. A control method for controlling a grab-type dredging device for performing grab-type dredging work, which involves using a crane installed on a ship for seabed dredging and a grab bucket suspended by the crane to grab mud from the seabed of an area enclosed by a pollution control frame and load the mud onto a soil transport vessel moored to the ship, The steps include: acquiring dynamic information including position, height, displacement, velocity, rotation angle, angular velocity, and at least one of the objects in the grab-type dredging work, using sensing data obtained from remote sensing means attached to the crane and AI that analyzes said sensing data; A step of controlling the crane or the grab bucket according to the mechanical information. A control method having

23. A computer controls a grab-type dredging device for performing grab-type dredging work, which involves using a crane installed on a ship for seabed dredging and a grab bucket suspended by the crane to grab mud from the seabed in an area enclosed by a pollution control frame and load the mud onto a soil transport vessel moored to the ship. The steps include: acquiring dynamic information including position, height, displacement, velocity, rotation angle, angular velocity, and at least one of the objects in the grab-type dredging work, using sensing data obtained from remote sensing means attached to the crane and AI that analyzes said sensing data; A step of controlling the crane or the grab bucket according to the mechanical information. A program to execute.

Images