Assisting device, assisting system and assisting program

Abstract

To provide an assisting device, an assisting system and an assisting program that assist civil engineering and construction works.SOLUTION: In an assisting system 100, an assisting device 200 includes: an image obtaining unit that obtains image data representing an image which includes an image of a sediment loading portion of a sediment carrying ship; an image processing unit which divides the image of the sediment loading portion into multiple areas, and which classifies, for each area, the image of the sediment loading portion into an image of sediments, an image of a water surface, and an image of the internal wall of the sediment loading portion; and an output unit that outputs, for each area, a determination result of whether the sediments can be loaded or not based on the classification result.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an assistance device, an assistance system, and an assistance program. [Background technology]

[0002] Conventionally, efforts have been made to reduce the burden on workers in civil engineering and construction work. One example of such efforts is a technology that automatically measures the amount of soil loaded on a soil transport ship. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 1-272908 [Patent Document 2] JP 2000-88541 A Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the aging of the workforce and labor shortages in the civil engineering and construction industries have become problems, and there is a demand for further work assistance and automation.

[0005] In one aspect, the present invention is directed to assisting civil engineering and construction operations. [Means for solving the problem]

[0006] In one aspect, the support device has an image acquisition unit that acquires image data showing an image including an image of the soil loading section of the soil transport barge, an image processing unit that divides the image of the soil loading section into a plurality of areas and classifies the image of the soil loading section for each area into an image of soil, an image of the water surface, and an image of the inner wall of the soil loading section, and an output unit that outputs a result of determining whether or not soil can be loaded into each area based on the classification result. Effect of the Invention

[0007] Can assist with civil engineering and construction work. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram explaining an overview of dredging work. [Diagram 2] FIG. 1 is a diagram illustrating an example of a system configuration of a support system. [Diagram 3] FIG. 2 illustrates an example of a hardware configuration of the support device. [Figure 4] FIG. 2 is a diagram illustrating the functions of each device included in the support system. [Diagram 5] 13 is a flowchart illustrating a process of the support device. [Figure 6] FIG. 11 is a first diagram illustrating the processing performed by the support device. [Figure 7] FIG. 11 is a second diagram illustrating the processing by the support device. [Figure 8] FIG. 11 is a third diagram illustrating the processing by the support device. [Figure 9] FIG. 4 is a fourth diagram illustrating the processing by the support device. [Figure 10] FIG. 13 is a diagram illustrating an example of a display on the control device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment will be described with reference to the drawings. A support system for dredging work according to this embodiment will be described with reference to Figs. 1 and 2. Fig. 1 is a diagram for explaining an outline of dredging work. Fig. 2 is a diagram showing an example of a system configuration of the support system.

[0010] Fig. 1 shows a grab dredging operation carried out at sea. In Fig. 1, a grab dredger 1 has a driver's cab 5, a grab bucket 6, a bridge 8, and a control device 300, and is fixed to a dredging site.

[0011] The control device 300 controls the operation of the grab dredger 1, and controls the operation of attachments including the grab bucket 6 etc. in response to the operation by the operator. The control device 300 of this embodiment has a display device that displays various information to the operator.

[0012] During grab dredging operations, an operator on board the driver's cab 5 of the grab dredger ship 1 operates the grab bucket 6 to dig up soil from the seabed within a pollution prevention frame 7 and discharge it into the soil loading section 4 of the soil transport ship 3.

[0013] When a certain amount of sediment is dumped into the sediment dump section 4, the soil transport ship 3 is pushed out by the pusher boat 2 away from the grab dredger 1 and moves to the sediment dump site. In grab dredging work, this operation is repeated to remove sediment from the seabed.

[0014] In addition, the grab dredger 1 of this embodiment is equipped with a support device 200 to support the operator's work.

[0015] Specifically, the support device 200 causes the imaging device 21 installed on the bridge 8 to capture an image of the sediment loading section 4 of the soil transport ship 3, and obtains image data showing the image of the sediment loading section 4 from the imaging device 21. The support device 200 then determines an area in which sediment can be loaded at the sediment loading section 4 from the image data of the sediment loading section 4, and notifies the operator of the determination result. The bridge 8 may be a command center for maneuvering installed at a high position on the grab dredger ship 1. The imaging device 21 may be installed in a location where it can capture an image of the sediment loading section 4 of the soil transport ship 3.

[0016] Therefore, according to this embodiment, the operator of the grab dredger 1 only needs to operate the grab bucket 6 so that sediment is dumped into the area where dumping of sediment is permitted as determined by the support device 200, and the operator is not required to check the status of the sediment dumping section 4 by himself / herself. In this way, this embodiment can support the operator's work.

[0017] The support device 200 and the control device 300 of this embodiment may be included in a support system 100, as shown in Fig. 2. In the support system 100, the support device 200 and the control device 300 can communicate with each other wirelessly or via wire. The support system 100 may also include an imaging device 21 provided on the bridge 8.

[0018] In addition, the support device 200 of this embodiment may be installed outside the driver's cab of the grab dredger 1, or may be installed at a location away from the grab dredger 1. Furthermore, the support device 200 of this embodiment may be realized by a plurality of information processing devices.

[0019] The hardware configuration of the support device 200 of this embodiment will be described below with reference to Fig. 3. Fig. 2 is a diagram showing an example of the hardware configuration of the support device.

[0020] The support device 200 of this embodiment is a computer including an input device 201, an output device 202, a drive device 203, an auxiliary storage device 204, a memory device 205, a processor 206, and an interface device 207, all of which are interconnected via a bus B.

[0021] The input device 201 is a device for inputting various types of information and is realized by, for example, a keyboard, a pointing device, etc. The output device 202 is for outputting various types of information and is realized by, for example, a display, etc. The interface device 207 includes a LAN card, etc., and is used for connecting to a network.

[0022] The assistance program possessed by the assistance device 200 is at least a part of various programs that control the assistance device 200. The assistance program is provided, for example, by distribution of a recording medium 208 or by downloading from a network. The recording medium 208 on which the assistance program is recorded can be of various types, including recording media that optically, electrically, or magnetically record information, such as a CD-ROM, a flexible disk, or a magneto-optical disk, and semiconductor memories that electrically record information, such as a ROM or a flash memory.

[0023] When the recording medium 208 on which the assistance program is recorded is set in the drive device 203, the assistance program recorded on the recording medium 208 is installed in the auxiliary storage device 204 from the recording medium 208 via the drive device 203. The assistance program downloaded from the network is installed in the auxiliary storage device 204 via the interface device 207.

[0024] The auxiliary storage device 204 realizes various storage units and the like described below, and stores the assistance program installed in the assistance device 200, as well as various files, data, and the like required by the assistance device 200. The memory device 205 reads out and stores the assistance program from the auxiliary storage device 204 when the assistance device 200 is started. The arithmetic processing device 206 realizes various processes described below in accordance with the assistance program stored in the memory device 205.

[0025] Next, the functions of each device in the support system 100 will be described with reference to FIG. FIG. 4 is a diagram illustrating the functions of each device included in the support system.

[0026] First, a description will be given of the functions of the support device 200. The support device 200 includes an image acquisition unit 210, a position detection unit 220, an image processing unit 230, an area determination unit 240, an area determination unit 250, and an output unit 260.

[0027] The image acquisition unit 210 acquires image data indicating an image captured of the soil injection unit 4 from the imaging device 21. The image data acquired by the image acquisition unit 210 from the imaging device 21 may be moving image data or still image data.

[0028] The position detection unit 220 detects, based on the image data acquired by the image acquisition unit 210, that the position of the soil boat 3 has reached a predetermined position.

[0029] When the position detection section 220 detects that the position of the soil boat 3 has reached a predetermined position, the image processing section 230 executes processing.

[0030] The image processing unit 230 of this embodiment extracts an image of the sediment injection unit 4 from the image indicated by the image data acquired by the image acquisition unit 210. Then, the image processing unit 230 divides the extracted image of the sediment injection unit 4 into a plurality of areas, and classifies the image of each of the plurality of areas into an image of sediment, an image of the water surface, and an image of the inner wall of the sediment injection unit.

[0031] The image processing unit 230 has a sediment determination model 231 , and when image data of the sediment injection unit 4 is extracted from the image data acquired by the image acquisition unit 210 , the extracted image data is input to the sediment determination model 231 .

[0032] Specifically, the image processing unit 230 extracts an image of the soil injection unit 4 from the image indicated by the image data acquired by the image acquisition unit 210, and performs projective transformation to convert it into a rectangular image.

[0033] In addition, the image processing unit 230 may set a division number for dividing a rectangular image into a plurality of areas. In this embodiment, it is assumed that the division number is set to "3". Note that the division number may be any value.

[0034] The image processing unit 230 inputs image data representing a rectangular image and the set division number to the soil / sand determination model 231 .

[0035] The sediment determination model 231 divides the image indicated by the input image data into areas whose number is indicated by the division number. Then, for each area, the sediment determination model 231 classifies the image indicated by the image data into an image of sediment, an image of the water surface, and an image of the inner wall of the sediment input section 4, and outputs the respective ratios of the area of ​​the image of the sediment in the sediment input section 4, the area of ​​the image of the water surface in the sediment input section 4, and the area of ​​the image of the inner wall of the sediment input section 4 to the area of ​​the image of the area.

[0036] The area determination unit 240 determines whether or not it is possible to dump sand into each area by referring to the ratio of the area of ​​the image of the soil and sand, the area of ​​the image of the water surface, and the area of ​​the image of the inner wall for each area, and threshold information 241. The details of the threshold information 241 will be described later.

[0037] The area determination unit 250 determines an area into which earth and sand is dumped from among a plurality of areas, according to the result of the determination by the area determination unit 240. The details of the processing by the area determination unit 250 will be described later.

[0038] The output unit 260 notifies the control device 300 of the determination result by the area determination unit 240 and the area determined by the area determination unit 250. The output unit 260 may also output an instruction signal to the grab dredger 1 to instruct the grab dredger 1 to dump soil into the area determined by the area determination unit 250.

[0039] Next, a description will be given of the functions of the control device 300. The control device 300 includes a display control unit 320 and an operation control unit 330.

[0040] The display control unit 320 causes the display device of the control device 300 to display the information output from the support device 200. Specifically, the display control unit 320 may cause the display device of the control device 300 to display the determination result by the area determination unit 240 and information indicating the area determined by the area determination unit 250.

[0041] The operation control unit 330 controls the operation of the grab dredger 1. When the operation control unit 330 of this embodiment receives an instruction signal from the support device 200 indicating an area to dump sediment, the operation control unit 330 may control the operation of the attachments including the grab bucket 6 so as to dump sediment into the area indicated by the instruction signal.

[0042] Next, the processing of the support device 200 of this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart illustrating the processing of the support device.

[0043] The support device 200 of this embodiment acquires, by the image acquisition unit 210, image data captured by the imaging device 21 installed on the bridge 8 (step S501).

[0044] Next, the support device 200 determines whether the position of the soil boat 3 has reached a predetermined position by the position detection unit 220 (step S502). The process of step S502 will be described in detail later.

[0045] In step S502, if the soil boat 3 is not in the predetermined position, the support device 200 returns to step S501.

[0046] In step S502, when the earth transport boat 3 reaches a predetermined position, the support device 200 causes the image processing unit 230 to perform projective transformation on the image indicated by the image data acquired by the image acquisition unit 210 (step S503).

[0047] Next, the image processing unit 230 of the support device 200 inputs the image data after the projective transformation to the soil / sand determination model 231 (step S504).

[0048] Next, the support device 200 acquires data output from the soil / sand determination model 231 (step S505).

[0049] Specifically, the sediment determination model 231 of this embodiment divides the image indicated by the image data after projective transformation into a number of areas according to the set division number, and outputs, for each area, the proportion of the area of ​​the image of the sediment, the image of the water surface, and the image of the inner wall to the area of ​​the image of the area.

[0050] Next, the support device 200 causes the area determination unit 240 to refer to the threshold information 241 and determine whether or not there is an area among the three areas into which earth and sand can be dumped (step S506).

[0051] In step S506, if there is no area where soil can be dumped, the support device 200 ends the process. At this time, the support device 200 may output a message to the control device 300 instructing the pusher 2 to move the soil boat 3.

[0052] In step S506, if there is an area into which earth and sand can be dumped, the support device 200 causes the area determination unit 250 to determine the area into which earth and sand is dumped (step S507).

[0053] Next, the support device 200 outputs information indicating the area into which the soil and sand is dumped to the control device 300 via the output unit 260 (step S508), and returns to step S501.

[0054] 5, after the area into which soil and sand is dumped is determined by the area determination unit 250, the output unit 260 outputs the determined result to the control device 300, but this is not limiting. When the area determination unit 240 determines that an area into which soil and sand is dumped exists, the output unit 260 of this embodiment may notify the operator of the area into which soil and sand can be dumped.

[0055] Next, the processing of the support device 200 of this embodiment will be further described with reference to Fig. 6 to Fig. 9. Fig. 6 is a first diagram for explaining the processing by the support device. In Fig. 6, the detection of the position of the soil boat 3 by the position detection unit 220 will be explained.

[0056] 6 is an example of an image captured by the imaging device 21. The image 60 shows a state in which soil DS1 and soil DS2 have been dumped into the soil dumping section 4 of the soil barge 3.

[0057] In this embodiment, the soil loading section 4 of the soil transport ship 3 is rectangular, and a GNSS (Global Navigation Satellite System) receiver (not shown) and a marker 32 are installed at the four corners. The marker 32 may be, for example, a primary color ball. In this embodiment, the grab dredger 1 may also be equipped with a GNSS receiver.

[0058] The GNSS receivers installed at the four corners of the soil depositing section 4 are an example of communication devices for acquiring position information for identifying the position of the soil depositing section 4.

[0059] In this embodiment, a predetermined position of the soil boat 3 may be preset for the position detection unit 220 in accordance with the angle of view of the imaging device 21. In the image processing unit 230 of this embodiment, for example, the positions where the markers 32 at the four corners overlap with the vertices of the rectangular area indicated by the dotted line 61 in the image 60 captured by the imaging device 21 may be preset as the predetermined position of the soil boat 3.

[0060] In this embodiment, when the position detection section 220 detects that the position of the soil boat 3 has reached a predetermined position, the image processing section 230 starts processing.

[0061] The image processing unit 230 performs projective transformation on the image of the soil supplying unit 4 (the image of the area surrounded by the dotted line 61) included in the image 60, and converts the image of the soil supplying unit 4 into a rectangular image 71 shown in FIG.

[0062] 7 is a first diagram illustrating the processing by the support device. An image 71 shown in FIG. 7 is a rectangular image obtained by performing projective transformation on the image of the soil injection unit 4 included in the image 60 shown in FIG.

[0063] In the support device 200 of this embodiment, the image processing unit 230 inputs image data indicating the image 71 to the soil / sand determination model 231 .

[0064] Then, the support device 200 determines, by the area determination unit 240, based on the output of the earth and sand determination model 231, whether or not dumping of earth and sand is possible for each area.

[0065] Image 72 in FIG. 7 is an example of an image showing the result of dividing image 71 into three areas and determining whether or not dumping of earth and sand is possible for each area.

[0066] In image 72, image 71 is divided into areas 72a, 72b, and 72c, and the judgment results for each area are shown.

[0067] Specifically, in the image 72, among the areas 72a, 72b, and 72c, it can be seen that the dumping of further soil is prohibited in the area 72a, into which soil DS1 has already been dumped, and in the area 72c, into which soil DS2 has already been dumped. Also, in the image 72, it can be seen that the dumping of soil is permitted in the area 72b.

[0068] Next, the determination as to whether or not dumping of earth and sand into each area is possible will be further described with reference to Figs.

[0069] Fig. 8 is a third diagram illustrating the processing by the support device. An image 81 shown in Fig. 8 is an example of an image obtained by performing projective transformation on an image of the soil injection unit 4 included in the image captured by the imaging device 21.

[0070] In this embodiment, for example, image data showing the image 81 is input to the soil / sand determination model 231 .

[0071] When image data showing an image 81 is input, the sediment determination model 231 of this embodiment divides the image 81 into three areas according to a set division number. Next, the sediment determination model 231 classifies the image into an image of sediment, an image of the water surface, and an image of the inner wall of the sediment feeder 4 by a method (segmentation) in which images having similar characteristics among pixels included in the image of each area are classified in the same category. Then, the sediment determination model 231 outputs the ratio of the area of ​​the image of sediment, the ratio of the area of ​​the image of the water surface, and the ratio of the area of ​​the image of the inner wall of the sediment feeder 4 to the area of ​​the image.

[0072] An image 82 shown in FIG. 8 shows the result of dividing an image 81 into three areas 83, 84, and 85, and classifying the images of each area into an image of soil and sand, an image of the water surface, and an image of the inner wall of the soil injection section 4.

[0073] Image 83a in area 83 is an image of soil, image 83b is an image of the water surface, and image 83c is an image of the inner wall of the soil feed section 4. Additionally, image 84a in area 84 is an image of soil, image 84b is an image of the water surface, and image 84c is an image of the inner wall of the soil feed section 4. Additionally, image 85a in area 85 is an image of soil, image 85b is an image of the water surface, and image 85c is an image of the inner wall of the soil feed section 4.

[0074] Therefore, the soil / sand determination model 231 outputs the ratio of the area of ​​the image 83a to the area of ​​the area 83, the ratio of the area of ​​the image 83b to the area of ​​the area 83, and the ratio of the area of ​​the image 83c to the area of ​​the area 83.

[0075] Further, the soil / sand determination model 231 outputs the ratio of the area of ​​the image 84a to the area of ​​the area 84, the ratio of the area of ​​the image 84b to the area of ​​the area 84, and the ratio of the area of ​​the image 84c to the area of ​​the area 84.

[0076] Further, the soil / sand determination model 231 outputs the ratio of the area of ​​the image 85a to the area of ​​the area 85, the ratio of the area of ​​the image 85b to the area of ​​the area 85, and the ratio of the area of ​​the image 85c to the area of ​​the area 85.

[0077] Fig. 9 is a fourth diagram illustrating the processing by the support device. An image 91 shown in Fig. 9 is another example of an image obtained by performing projective transformation on the image of the soil injection unit 4 included in the image captured by the imaging device 21.

[0078] An image 92 shown in FIG. 9 shows the result of dividing the image 91 into three areas 93, 94, and 95, and classifying the images of each area into an image of soil and sand, an image of the water surface, and an image of the inner wall of the soil injection section 4.

[0079] Image 93b in area 93 is an image of the water surface, and image 93c is an image of the inner wall of the soil feed section 4. Additionally, image 94a in area 94 is an image of soil, image 94b is an image of the water surface, and image 94c is an image of the inner wall of the soil feed section 4. Additionally, image 95b in area 95 is an image of the water surface, and image 95c is an image of the inner wall of the soil feed section 4.

[0080] Therefore, the sediment determination model 231 outputs the ratio of the area of ​​image 93b to the area of ​​area 93, and the ratio of the area of ​​image 93c to the area of ​​area 93. The sediment determination model 231 also outputs the ratio of the area of ​​image 94a to the area of ​​area 94, the ratio of the area of ​​image 94b to the area of ​​area 94, and the ratio of the area of ​​image 94c to the area of ​​area 94. The sediment determination model 231 also outputs the ratio of the area of ​​image 95b to the area of ​​area 95, and the ratio of the area of ​​image 95c to the area of ​​area 95.

[0081] In this embodiment, when the soil / sand determination model 231 outputs the respective proportions of soil / sand images, water surface images, and inner wall images for each area, the area determination unit 240 compares this with threshold information 241 to determine whether soil / sand can be dumped.

[0082] Here, the threshold information 241 of this embodiment will be described. The threshold information 241 of this embodiment may be, for example, information indicating a threshold set for the ratio of the area of ​​the image of the inner wall of the soil injection section 4 to the area of ​​the area.

[0083] In an area, when the area of ​​the image of the inner wall of the soil dumping section 4 is smaller than the threshold value, this indicates that a certain amount of soil and seawater has already been dumped into the soil dumping section 4. Therefore, the area determination section 240 of this embodiment may determine an area where dumping of soil is prohibited, where the ratio of the area of ​​the inner wall of the soil dumping section 4 to the area area output from the soil determination model 231 is equal to or less than the threshold value indicated by the threshold information 241.

[0084] For example, suppose that in image 82 of Fig. 8, the ratio of the area of ​​image 83c to the area of ​​area 83 is greater than the threshold. In this case, area 83 is determined to be an area into which earth and sand can be dumped. Also, suppose that in image 82 of Fig. 8, the ratio of the area of ​​image 84c to the area of ​​area 84 is equal to or smaller than the threshold. In this case, area 84 is determined to be an area into which earth and sand can be dumped.

[0085] Furthermore, the threshold information 241 in this embodiment may be, for example, information indicating a threshold set for the ratio of the area of ​​the image of the soil and sand in the soil depositing section 4 to the area of ​​the area.

[0086] For example, in image 82 of Fig. 8, it is assumed that the ratio of the area of ​​image 83a to the area of ​​area 83 and the ratio of the area of ​​image 85a to the area of ​​area 85 are less than the threshold. In this case, areas 83 and 85 are determined as areas into which dumping of soil and sand is permitted. Also, in image 82 of Fig. 8, it is assumed that the ratio of the area of ​​image 84a to the area of ​​area 84 is equal to or greater than the threshold. In this case, area 84 is determined as an area into which dumping of soil and sand is prohibited.

[0087] Furthermore, the threshold information 241 in this embodiment may be information indicating a threshold set for the ratio of the area of ​​the image of the sediment in the sediment loading section 4 and the area of ​​the image of the water surface to the area of ​​the area, for example.

[0088] For example, in image 92 in Fig. 9, if the ratio of the area of ​​image 93b to the area of ​​area 93 and the ratio of the area of ​​image 95b to the area of ​​area 95 are equal to or greater than the threshold value set for the area of ​​the water surface image, areas 93 and 95 are determined to be areas where dumping of soil and sand is prohibited.

[0089] In addition, the threshold information 241 in this embodiment may be information indicating thresholds set for each of the ratio of the area of ​​the image of the soil and sand to the area of ​​the area, the ratio of the area of ​​the image of the water surface, and the ratio of the area of ​​the image of the inner wall of the soil and sand loading section 4.

[0090] For example, even if the area ratio of the image of the inner wall is below a predetermined value, if the area of ​​the image of the soil and sand is small and the area ratio of the image of the water surface is large, it is possible that mainly seawater has been poured into the soil pouring section 4 and only a small amount of soil has been poured in.

[0091] In this embodiment, taking such a situation into consideration, the threshold information 241 may be information indicating thresholds set for each of the area proportion of the image of soil and sand, the area proportion of the image of the water surface, and the area proportion of the image of the inner wall.

[0092] In addition, in the threshold information 241 of this embodiment, a threshold for the proportion of the area of ​​the image of soil and sand, a threshold for the proportion of the area of ​​the image of the water surface, and a threshold for the proportion of the area of ​​the image of the inner wall may each be set individually for each area.

[0093] The image of the soil loading section 4 is captured by an imaging device 21 installed on a bridge 8 arranged at the rear of the soil barge 3. For this reason, the inner walls of the front part of the soil loading section 4 are likely to be captured within the imaging range, while the inner walls of the rear part of the soil loading section 4 are unlikely to be captured within the imaging range.

[0094] For this reason, for example, in areas 83 and 93 shown in FIG. 8 and FIG. 9, the area of ​​the image of the inner wall of the soil injection section 4 tends to be larger than in areas 85 and 95.

[0095] In this embodiment, the threshold information 241 may thus take into account the imaging range of the imaging device 21 and set a threshold for the proportion of the area of ​​the image of soil and sand, a threshold for the proportion of the area of ​​the image of the water surface, and a threshold for the proportion of the area of ​​the image of the inner wall individually.

[0096] In this embodiment, by setting the threshold information 241 in this manner, it is possible to determine whether or not dumping of earth and sand in each area is permitted.

[0097] Next, a display example of the control device 300 of this embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram showing a display example of the control device.

[0098] 10 is an example of a screen displayed on, for example, a display device of the control device 300. The screen 301 may be displayed on the display device by, for example, the processing of step S508 shown in FIG.

[0099] The screen 301 includes display areas 302, 303, and 304. In the display area 302, an image indicated by image data acquired by the image acquisition unit 210 is displayed.

[0100] In the display area 303, an image 302a of the soil injection unit 4 is extracted from the image displayed in the display area 302, and a rectangular image 303a is displayed after projective transformation. In Fig. 10, the image 71 shown in Fig. 7 is displayed as the image 303a.

[0101] Display area 304 includes display areas 304a, 304b, and 304c, and an image showing the result of judging whether or not earth and sand can be dumped into each area is displayed in each display area. In Fig. 10, an image of area 72a shown in Fig. 7 and whether or not earth and sand can be dumped into area 72a are displayed in display area 304a. Also, in Fig. 10, an image of area 72b shown in Fig. 7 and whether or not earth and sand can be dumped into area 72b are displayed in display area 304b, and an image of area 72c shown in Fig. 7 and whether or not earth and sand can be dumped into area 72c are displayed in display area 304c.

[0102] In the example of FIG. 10, it can be seen that dumping of earth and sand is prohibited in areas 72a and 72c, but dumping of earth and sand is permitted in area 72b.

[0103] In this embodiment, by displaying the judgment result on the control device 300 in this manner, the operator can understand the sediment dumping position at the sediment dumping section 4, thereby assisting the operator in operating the grab dredger 1.

[0104] In addition, for example, when there are multiple areas into which soil and sand can be dumped, the support device 200 of this embodiment may determine the area into which soil and sand is to be dumped next, depending on the operation of the grab dredger 1 so far.

[0105] The following describes a case where it is determined that dumping of earth and sand is possible in all of the areas 72a, 72b, and 72c.

[0106] In this case, the area determination unit 250 identifies the area into which sediment was dumped by the previous operation of dumping sediment by the grab dredger 1. Specifically, the area determination unit 250 may capture an image of the grab dredging operation by the imaging device 21, and identify the area into which sediment was dumped last time based on the captured image data. The area determination unit 250 may also obtain information indicating the operation history of the grab dredger 1 from the control device 300, and identify the area into which sediment was dumped last time based on the operation history of the grab dredger 1.

[0107] When the area determination unit 250 identifies the area into which earth and sand was dumped the previous time, it may determine an area other than the identified area as the area into which earth and sand will be dumped next time.

[0108] In other words, when there are a plurality of areas determined to be available for dumping of earth and sand, the area determination unit 250 may determine the area into which earth and sand is dumped so as not to dump earth and sand into the same area consecutively.

[0109] In other words, when there are multiple areas determined to be available for dumping of earth and sand, the area determination unit 250 determines an area other than the area into which earth and sand was previously dumped as the area into which earth and sand is next dumped.

[0110] The support device 200 may display the area determined by the area determination unit 250 as an area into which soil and sand are to be dumped on the screen 301. In this case, the display manner on the screen 301 may be different so that the operator can separately recognize the area into which soil and sand are determined to be dumped and the area into which soil and sand should be dumped.

[0111] In this embodiment, by determining the area into which soil and sand are dumped in this manner, it is possible to prevent soil and sand from being dumped in a concentrated manner in one part of the soil dumping section 4, which would cause the soil transport ship 3 to tilt significantly, thereby improving work efficiency and safety.

[0112] In addition, in cases where there are multiple areas into which soil and sand can be dumped, the area determination unit 250 of this embodiment may, for example, calculate the inclination of the soil transport ship 3 from the image data acquired by the image acquisition unit 210, and determine the area into which soil and sand is dumped based on the inclination of the soil transport ship 3.

[0113] Specifically, the area determination unit 250 may determine an area into which soil and sand can be dumped from a plurality of areas into which soil and sand can be dumped, so as to bring the inclination of the soil transport ship 3 closer to horizontal.

[0114] By doing so, the fluctuation in the inclination of the soil transport ship 3 can be reduced, and work efficiency and safety can be improved.

[0115] Furthermore, in this embodiment, when the area into which soil and sand is dumped is determined by the area determination unit 250, position information specifying the position of the determined area may be acquired, and an instruction signal for instructing to dump soil and sand into the determined area may be transmitted by the output unit 260 together with the position information. Note that the support device 200 may acquire position information specifying the position of the determined area based on position information acquired from GNSS receivers installed at the four corners of the soil dumping unit 4.

[0116] When the control device 300 of this embodiment receives an instruction signal from the support device 200, the operation control unit 330 acquires position information indicating the current position of the grab dredger 1. Then, based on the position information acquired together with the instruction signal and the position information indicating the current position of the grab dredger 1, the operation control unit 330 controls the operation of the attachments including the grab bucket 6 to dump sediment into the determined area.

[0117] In this embodiment, by doing this, the grab dredger 1 can automatically perform the task of dumping soil into the soil dumping section 4, thereby supporting civil engineering and construction work including grab dredging work.

[0118] In addition, the sediment determination model 231 of this embodiment classifies the image of the sediment input section 4, which is divided into multiple areas by segmentation, into an image of sediment, an image of the water surface, and an image of the inner wall, and outputs the ratio of the area of ​​each image to the area of ​​the image of the area, but is not limited to this.

[0119] The sediment determination model 231 of this embodiment may be, for example, a model that uses a method (classification) for classifying what is shown in the entire image. In this case, the sediment determination model 231 may be a model that receives image data showing the image of the sediment feeder 4 and the number of divisions as input, and outputs data showing whether each area is empty or full. In the above embodiment, the support device 200 is used in grab dredging work carried out at sea, but the support device 200 may be applied to work other than grab dredging work. Specifically, for example, the support device 200 may be applied to other dredging works, soil transport ship movement, soil dumping works, etc. Dredging work involves, for example, physically digging soil or sand from the bottom of a waterway, channel, anchorage, etc. using a grab bucket or the like, loading the soil onto a soil transport barge, and removing the soil, and the support device 200 may be applied to backhoe dredging work, which is a type of dredging work. Furthermore, in this embodiment, the soil transport ship 3 is stopped, and when the area determination unit 250 determines the area into which soil is to be dumped at the soil dumping unit 4 of the soil transport ship 3, the grab bucket 6 is rotated directly above the determined area and soil is dumped into the determined area, but this is not limited to this. In this embodiment, when the area determination unit 250 determines the area into which soil is to be dumped at the soil dumping section 4 of the soil transport ship 3, the soil transport ship 3 may be moved so that the determined area is directly below the grab bucket 6. In this case, the support device 200 may calculate the movement direction and movement distance of the soil transport ship 3 based on the relationship between the soil unloading position of the grab bucket 6 and the position of the determined area, and display it on a display device possessed by the control device 300 of the grab dredger ship 1 or on a display device installed on the push boat 2. In this embodiment, by doing so, the rotation operation of the grab bucket 6 can be kept constant, and the work efficiency of the operator of the grab dredger 1 can be improved. Soil removal and dumping involves physically removing dredged soil and sand from the mud tank of a soil transport barge using a bucket or other device to remove the soil and sand onto land or the seabed. In this case, the area determination unit 250 of the support device 200 of this embodiment may determine the area into which the soil is discharged so that the inclination of the soil ship approaches horizontal when discharging soil from the soil input section (mud tank) of the soil ship. In this way, the fluctuation of the inclination of the soil ship can be reduced, and soil can be discharged from the soil ship in a stable state. In addition, the soil lifting and dumping work may include backhoe lifting and crane ship lifting.

[0120] The disclosed technology may take the following forms as described below. (Appendix 1) an image acquisition unit that acquires image data showing an image including an image of the soil loading section of the soil transport ship; an image processing unit that divides the image of the soil injection section into a plurality of areas and classifies the image of the soil injection section for each area into an image of soil, an image of a water surface, and an image of an inner wall of the soil injection section; An support device having an output unit that outputs, for each area, a result of judging whether or not soil can be dumped based on the classification result. (Appendix 2) a determination unit that determines whether or not to dump soil and sand for each of the areas based on the classification results; The image processing unit includes: outputting, for each area, a ratio of an area of ​​the image of the soil and sand to an area of ​​the image of the soil injection portion, a ratio of an area of ​​the image of the water surface, and a ratio of an area of ​​the image of the inner wall; The determination unit is An assistance device as described in Appendix 1, which determines whether or not to dump soil into the area using at least one of the area ratio of the image of the soil, the area ratio of the image of the water surface, and the area ratio of the image of the inner wall. (Appendix 3) An assistance device as described in appendix 1 or 2, having a determination unit that, when there are multiple areas determined to be capable of being dumped with soil, determines an area other than the area into which soil was previously dumped as the area into which soil is next dumped. (Appendix 4) The output unit is A support device as described in Appendix 3, which outputs an instruction signal to a control device of a grab dredger to instruct the dumping of soil into the determined area, together with location information identifying the area determined by the determination unit. (Appendix 5) a position detection unit for detecting whether the soil transport boat is at a predetermined position; The image processing unit includes: 5. The support device according to any one of claims 1 to 4, wherein the classification is performed when it is detected that the soil transport vessel is in a predetermined position. (Appendix 6) The image processing unit includes: When image data showing an image of the soil injection section and the number of the areas are input, A support device as described in any one of appendices 1 to 5, having a model that outputs, for each area, the ratio of the area of ​​the image of the soil and sand to the area of ​​the image of the soil input section, the ratio of the area of ​​the image of the water surface, and the ratio of the area of ​​the image of the inner wall. (Appendix 7) A support system including a support device and a control device for a grab dredger, The support device includes: an image acquisition unit that acquires image data showing an image including an image of the soil loading section of the soil transport ship; an image processing unit that divides the image of the soil injection section into a plurality of areas and classifies the image of the soil injection section for each area into an image of soil, an image of a water surface, and an image of an inner wall of the soil injection section; An output unit that outputs to the control device the result of determining whether or not soil can be dumped for each area based on the classification result. (Appendix 8) Acquire image data showing an image of the soil loading section of the soil transport barge; Dividing the image of the soil injection section into a plurality of areas, and classifying the image of the soil injection section for each area into an image of soil, an image of a water surface, and an image of an inner wall of the soil injection section; A support program that causes a computer to execute a process that outputs, for each area, a result of determining whether or not soil and sand can be dumped based on the classification result. The invention is not limited to the specifically disclosed embodiments, but various modifications and variations are possible without departing from the scope of the appended claims. [Explanation of symbols]

[0121] 1. Grab Dredger 2 Push boat 3. Soil Carrier 4. Soil input section 5. Cab 6 Grab Bucket 7 Pollution Prevention Frame 100 Support System 200 Support equipment 210 Image acquisition unit 220 Position detection unit 230 Image Processing Unit 231 Sediment Judgment Model 240 Area Determination Unit 241 Threshold Information 250 Area Determination Division 260 Output section 300 Control device 330 Motion control section

Claims

1. an image acquisition unit that acquires image data showing an image including an image of the soil loading section of the soil transport ship; an image processing unit that divides the image of the soil injection section into a plurality of areas and classifies the image of the soil injection section for each area into an image of soil, an image of a water surface, and an image of an inner wall of the soil injection section; An support device having an output unit that outputs, for each area, a result of judging whether or not soil can be dumped based on the classification result.

2. a determination unit that determines whether or not to dump soil and sand for each of the areas based on the classification results; The image processing unit includes: outputting, for each area, a ratio of an area of ​​the image of the soil and sand to an area of ​​the image of the soil injection portion, a ratio of an area of ​​the image of the water surface, and a ratio of an area of ​​the image of the inner wall; The determination unit is The support device according to claim 1 , which determines whether or not to dump soil into the area using at least one of the area ratio of the image of the soil, the area ratio of the image of the water surface, and the area ratio of the image of the inner wall.

3. The support device according to claim 1 or 2, further comprising a determination unit that, when there are multiple areas determined to be capable of being dumped with soil, determines an area other than the area into which soil was previously dumped as the area into which soil will next be dumped.

4. The output unit is The support device according to claim 3, further comprising: an instruction signal for instructing a control device of a grab dredger to dump soil into the determined area, together with position information identifying the area determined by the determination unit.

5. The image processing unit includes: When image data showing an image of the soil injection section and the number of the areas are input, The support device according to claim 1 , further comprising a model that outputs, for each area, the ratio of the area of ​​the image of the soil and sand to the area of ​​the image of the soil input section, the ratio of the area of ​​the image of the water surface, and the ratio of the area of ​​the image of the inner wall.

6. A support system including a support device and a control device for a grab dredger, The support device includes: an image acquisition unit that acquires image data showing an image including an image of the soil loading section of the soil transport ship; an image processing unit that divides the image of the soil injection section into a plurality of areas and classifies the image of the soil injection section for each area into an image of soil, an image of a water surface, and an image of an inner wall of the soil injection section; An output unit that outputs to the control device the result of determining whether or not soil can be dumped for each area based on the classification result.

7. Acquire image data showing an image including an image of the soil loading section of the soil transport ship; Dividing the image of the soil injection section into a plurality of areas, and classifying the image of the soil injection section for each area into an image of soil, an image of a water surface, and an image of an inner wall of the soil injection section; A support program that causes a computer to execute a process that outputs, for each area, a result of determining whether or not soil and sand can be dumped based on the classification result.

Images