Underwater work system and underwater work method

Abstract

To provide an underwater working system and an underwater working method capable of suppressing the lowering of work speed in turbid water while grasping the accurate position of a monitoring object when visualizing the underwater working.SOLUTION: In an underwater working method, a two-dimensional image is recorded by measurement of a two-dimensional sonar 2 and a three-dimensional image is recorded by measurement of a measuring device 3 before starting work of a working device 71. Three-dimensional initial positions of identification points a, b, c, and d of the working device are calculated using the installation position of the two-dimensional sonar 2 as a reference point from the three-dimensional image before the work start, and the three-dimensional initial positions of the calculation results are associated with the two-dimensional position information of the identification points in the two-dimensional image before the work start. After starting the work of the working device, the two-dimensional image is recorded by the measurement of the two-dimensional sonar 2 and successively updated, and the updated two-dimensional image is displayed on a first monitor 6. Further, starting from the three-dimensional initial position of the identification point, the three-dimensional position information of the identification point is estimated and updated using the two-dimensional position information of the identification point in the two-dimensional image to be updated after starting the work.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an underwater work system and an underwater work method, and more particularly to an underwater work system equipped with a work device for working in water such as muddy water, and an underwater work method using the underwater work system. [Background technology]

[0002] Some work may be performed underwater, such as in rivers or oceans, or inside structures such as tanks or pools, where natural and artificial objects are present. In such cases, the work may need to be monitored visually to understand the work being performed. Observing the underwater environment with an optical camera is difficult when the water is turbid. Even if the water is clear, optical cameras are not suitable for long-distance observation. In such cases, visualization technology using ultrasound, which has higher penetration properties compared to light, is preferable.

[0003] A visualization technique using ultrasound involves transmitting an ultrasonic beam from an ultrasonic sensor into water, receiving the reflected waves from underwater objects (reflectors) with the ultrasonic sensor, and generating and displaying an image based on information about the reflected waves received by the ultrasonic sensor (more specifically, the intensity and propagation time of the reflected waves).Known ultrasound visualization techniques include a method for generating two-dimensional images called the surface viewing method and a method for generating three-dimensional images called the front viewing method.

[0004] In the surface viewing method, a fan-shaped diverging beam (hereinafter sometimes referred to as a fan beam) is transmitted, and the received reflected waves are resolved on the time axis to generate an image. A two-dimensional image is generated by scanning the fan beam one-dimensionally. Technology using the surface viewing method is disclosed in Patent Document 1, for example.

[0005] In the front-view method, an image corresponding to each position in a three-dimensional space (a three-dimensional image) is generated by transmitting and receiving a pencil-shaped ultrasonic beam similar to that of a laser range finder through two-dimensional scanning. A technology using the front-view method is disclosed in, for example, Patent Document 2. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-268192 [Patent Document 2] Japanese Patent Application Laid-Open No. 2000-64340 Summary of the Invention [Problem to be solved by the invention]

[0007] In the technology described in Patent Document 1, an ultrasonic beam is electronically scanned one-dimensionally (rotationally scanned) to generate an image by transmitting and receiving the beam a certain number of times (for example, N times), so images can be recorded at high speed. However, this technology (surface viewing method) has a weakness in that it does not have resolution in the transmission direction of the ultrasonic beam (fan beam) (resolution in the diffusion direction), and therefore cannot measure the exact position of a reflector (position in three-dimensional space).

[0008] On the other hand, the narrow multibeam sonar described in Patent Document 2 records three-dimensional images using a front-view method, making it possible to accurately measure the three-dimensional position of a reflector. In the technology described in Patent Document 2, an ultrasonic beam is electronically scanned one-dimensionally to record an image of one cross section through a certain number of transmissions and receptions (e.g., N times), and the three-dimensional image is recorded by mechanically swiveling the sonar in a direction perpendicular to the electronic scanning direction. When generating N images by swiveling the sonar, a total of N x N beam transmissions and receptions are required, which means that image recording takes longer than when images are recorded using a front-view method like the technology described in Patent Document 1.

[0009] As such, when underwater work is performed using images captured by the surface viewing method, it is difficult to accurately determine the position of reflectors, making it difficult to prevent a decrease in work speed in turbid water.When underwater work is performed using images captured by the front viewing method, it is possible to measure the accurate three-dimensional position of reflectors, but it takes time to update the images, making it difficult to prevent a decrease in work speed in turbid water.

[0010] The present invention has been made to solve the above problems, and its purpose is to provide an underwater work system and an underwater work method that can grasp the exact position of the object to be monitored when performing underwater work in a visualized manner, and can suppress a decrease in work speed in turbid water. [Means for solving the problem]

[0011] The present application includes multiple means for solving the above problems. For example, before work begins with a work device placed underwater, a two-dimensional image of the underwater work environment is captured using measurement with a two-dimensional ultrasonic sonar, and before work begins with the work device, a three-dimensional image of the underwater work environment is captured using measurement with a measuring device capable of measuring three-dimensional position information, and from the three-dimensional image captured before work begins, a three-dimensional initial position of an identification point of the work device is calculated using the installation position of the two-dimensional ultrasonic sonar as a reference point, and the calculated three-dimensional initial position of the identification point is associated with the two-dimensional position information of the identification point in the two-dimensional image captured before work begins. After work begins with the work device, two-dimensional images are captured and sequentially updated using continuous measurement with the two-dimensional ultrasonic sonar, and the sequentially updated two-dimensional images are displayed on a display device, and the three-dimensional position information of the identification point is estimated and sequentially updated using the two-dimensional position information of the identification point in the two-dimensional image that is sequentially updated after work begins. [Effects of the Invention]

[0012] According to the present invention, 2D images are captured using 2D ultrasonic sonar measurements while the work equipment is operating and displayed on a display device, thereby reducing the time required to visualize underwater work compared to capturing and displaying 3D images during work. Furthermore, the 3D position information of the identification points in the 2D images, which is updated during work starting from the 3D initial position calculated from the 3D image before work begins, is used to estimate and update the 3D position information of the identification points during work, making it possible to determine the 3D position information of identification points (monitoring targets) that cannot be measured from 2D images. These features make it possible to accurately determine the position of the monitoring target when visualizing underwater work and to prevent a decrease in work speed in turbid water. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing the configuration of an underwater work system according to a first embodiment of the present invention. [Figure 2] 2 is a diagram showing the measurement status of a monitoring target (working device) by a two-dimensional sonar and a three-dimensional sonar before work begins in the underwater work system according to the first embodiment shown in FIG. 1. FIG. [Figure 3] 2 is a diagram showing the measurement status of a monitoring target (working device) by a two-dimensional sonar during operation in the underwater work system according to the first embodiment shown in FIG. 1. FIG. [Figure 4] 2 is a flowchart showing an example of the procedure of an underwater work method using the underwater work system according to the first embodiment shown in FIG. [Figure 5] 3 is an explanatory diagram showing the scanning of ultrasonic beams of a two-dimensional sonar in the underwater work system according to the first embodiment shown in FIG. 1 on the XY plane of the world coordinate system shown in FIG. 2. FIG. [Figure 6] 1. FIG. 3 is a diagram showing the dimensions of two reflectors installed on the work arm of the work device that is the monitoring target in the underwater work system according to the first embodiment shown in FIG. 1, on the YZ cross section of the world coordinate system shown in FIG. 2. [Figure 7]7 is a diagram showing the dimensions of the working arm and the reflector of the working device shown in FIG. 6 when viewed from the arrow VII. FIG. [Figure 8] 2 is a diagram showing an example of waveform data and a two-dimensional image, which are measurement results before work begins using a two-dimensional sonar in the underwater work system according to the first embodiment shown in FIG. 1. FIG. [Figure 9] FIG. 1 is an explanatory diagram showing an example of a method for associating the measurement results of a 3D sonar and the measurement results of a 2D sonar before work begins for an identification point of a reflector to be monitored in the underwater work system according to the first embodiment shown in FIG. 1, on the YZ cross section of the world coordinate system. [Figure 10] FIG. 2 is an explanatory diagram showing an example of a method for estimating three-dimensional position information (distance and pitch angle) on the YZ cross section of the world coordinate system for an identification point of a reflector to be monitored in the underwater work system according to the first embodiment shown in FIG. 1. [Figure 11] 2A and 2B are diagrams showing an example of waveform data and a two-dimensional image, which are measurement results of a two-dimensional sonar in the underwater work system according to the first embodiment shown in FIG. 1 after work has started. [Figure 12] FIG. 2 is a block diagram showing the configuration of an underwater work system according to a modified example of the first embodiment of the present invention. [Figure 13] 10 is a flowchart showing an example of an underwater work method using an underwater work system according to a second embodiment of the present invention. [Figure 14] FIG. 14 is a block diagram showing the configuration of an underwater work system that executes the underwater work method according to the second embodiment shown in FIG. [Figure 15] 10 is a flowchart showing an example of an underwater work method using an underwater work system according to a third embodiment of the present invention. [Figure 16] FIG. 16 is a block diagram showing the configuration of an underwater work system that executes the underwater work method according to the third embodiment shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of an underwater work system and an underwater work method according to the present invention will be described with reference to the drawings.

[0015] [First embodiment] First, the schematic configuration of an underwater work system according to a first embodiment of the present invention will be described with reference to Figures 1 to 3. Figure 1 is a block diagram showing the configuration of an underwater work system according to a first embodiment of the present invention. Figure 2 is a diagram showing the measurement status of a monitored object (work equipment) by 2D sonar and 3D sonar before work begins in the underwater work system according to the first embodiment shown in Figure 1. Figure 3 is a diagram showing the measurement status of a monitored object (work equipment) by 2D sonar during work in the underwater work system according to the first embodiment shown in Figure 1.

[0016] 1, the underwater work system comprises a work implement 71 that performs work underwater, a work implement controller 72 that controls the work implement 71, and a work command device 80 that commands the operation of the work implement 71. The underwater work system according to this embodiment further comprises an underwater work monitoring system 1 that monitors the underwater work performed by the work implement 71 through visualization.

[0017] As shown in FIGS. 1 and 2, the work device 71 to be monitored by the underwater work monitoring system 1 includes at least one work arm 711, a moving mechanism 712 for moving on the floor or the like, and an optical camera 713 (shown only in FIG. 1) for observing the work environment of the work device 71. As shown in FIG. 2, a first reflector 715 and a second reflector 716 that reflect ultrasonic waves are installed on the surface of the work arm 711. The first reflector 715 and the second reflector 716 are arranged, for example, with a gap in the longitudinal direction of the work arm 711. The first reflector 715 and the second reflector 716 are parts of the work device 71 to be identified by ultrasonic measurement, of the work device 71 monitored by the underwater work monitoring system 1. The dimensions of the work arm 711 and the first reflector 715 and the second reflector 716 will be described later (see FIGS. 6 and 7). The moving mechanism 712 can be, for example, a rotating type such as a wheel or crawler, or a multi-legged walking type. When the working device 71 does not need to be moved, the working arm 711 and optical camera 713 can be installed on a structure or the like at the work site.

[0018] The drive of the work implement 71 is controlled by a work implement controller 72. The work implement controller 72 is located, for example, on a large mother ship or on land at a location away from the work implement 71 that is placed underwater. In this case, the work implement controller 72 is electrically connected to the work implement 71 via a cable (not shown). Note that the work implement controller 72 can also be configured to be located inside the work implement 71.

[0019] The working device controller 72 is a computer equipped with a storage device such as RAM or ROM and a processing device such as a CPU or MPU. The working device controller 72 has programs and various information required to control the working device 71 pre-stored in the storage device, and the processing device appropriately reads the programs and various information from the storage device and executes processing in accordance with the programs to realize various functions. The working device controller 72 has functional units such as a working device control unit 721, an arm control unit 722, a movement mechanism control unit 723, and a camera image storage unit 724. The working device control unit 721 outputs operation commands for the working arm 711 of the working device 71 based on drive commands for the working device 71 (described below) from the work command device 80, and also outputs operation commands for the movement mechanism 712 of the working device 71. The arm control unit 722 controls the operation of the working arm 711 in response to the operation commands for the working arm 711 from the working device control unit 721. The movement mechanism control unit 723 controls the operation of the movement mechanism 712 in response to the operation commands for the movement mechanism 712 from the working device control unit 721. The camera image storage unit 724 stores image data captured by the optical camera 713 of the working device 71 in a storage device, and displays the image data on the fourth monitor 73 as an external display device.

[0020] The drive of the work implement 71 is controlled via the work implement controller 72 in accordance with drive commands input from a work command device 80. The work command device 80 is an independent computer equipped with, for example, a storage device such as RAM or ROM and a processing device such as a CPU or MPU, and is located on a large mother ship or on land. The work command device 80 can be implemented as part of the functions of the work implement controller 72 described above, or as part of the functions of the work monitoring device 5 described below.

[0021] The work command device 80 realizes various functions by storing in advance in a storage device the programs and various information required to command the operation of the work device 71, and the processing device appropriately reads the programs and various information from the storage device and executes processing in accordance with the programs. The work command device 80 includes functional units, such as a work setting unit 81, a work device position information storage unit 82, and a 3D position information storage unit 83. The work setting unit 81 receives, for example, a pre-set work plan for the work device 71 (such as operation data for the work arm 711 during work and operation data for the movement mechanism 712 during work) or operation input by an operator, and generates a drive command for the work device 71 based on the received work plan and operation input. The work setting unit 81 outputs a drive command for the work device 71 to the work device controller 72. The work device position information storage unit 82 stores, in a storage device, the position information of the work device 71 output from the work device control unit 721 of the work device controller 72. The three-dimensional position information storage unit 83 stores the three-dimensional position information of the identification point (described later) of the work device 71 output from the three-dimensional position information update unit 59 (described later) of the work monitoring device 5 in a storage device.

[0022] The underwater work monitoring system 1 comprises a two-dimensional ultrasonic sonar 2 (hereinafter referred to as 2D sonar) capable of measuring two-dimensional position information of a work device 71 to be monitored using ultrasonic waves, a three-dimensional ultrasonic sonar 3 (hereinafter referred to as 3D sonar) capable of measuring three-dimensional position information of the work device 71 using ultrasonic waves, an optical camera 4 that photographs the work environment including the work device 71, and a work monitoring device 5 that monitors the work device 71 using the measurement results of the 2D sonar 2, the measurement results of the 3D sonar 3, and the video data of the optical camera 4. The underwater work monitoring system 1 further comprises a first monitor 6 that displays a two-dimensional image of the measurement results of the 2D sonar 2, a second monitor 7 that displays a three-dimensional image of the measurement results of the 3D sonar 3, and a third monitor 8 that displays an optical image of the optical camera 4. The underwater work monitoring system 1 can also be configured to include a work command device 80.

[0023] The 2D sonar 2 includes a 2D ultrasonic sensor 21 and a first transceiver 22 capable of measuring the 2D position information of a monitoring target. The 2D ultrasonic sensor 21 transmits an ultrasonic beam that diverges in a fan shape and receives reflected waves from objects (reflectors). The 2D ultrasonic sensor 21 is configured to scan in a direction perpendicular to the beam divergence direction. The 2D ultrasonic sensor 21 is, for example, a one-dimensional array sensor in which multiple transducers are arranged in one direction. The 2D ultrasonic sensor 21 can also be configured to form a fan-shaped diverging beam using an acoustic lens. The first transceiver 22 outputs a drive signal to each transducer of the 2D ultrasonic sensor 21 in response to a command from the work monitoring device 5. The drive signal vibrates each transducer to form the desired ultrasonic waves. The first transceiver 22 also captures the reflected wave signals received by each transducer of the 2D ultrasonic sensor 21 and generates waveform data and a 2D image based on the reflected wave signals. The first transceiver 22 then outputs the waveform data and the 2D image to the work monitoring device 5.

[0024] The 3D sonar 3 includes a 3D ultrasonic sensor 31 and a second transceiver 32 capable of measuring the 3D position information of the monitored object. The 3D ultrasonic sensor 31 is configured to transmit and receive ultrasonic beams using two-dimensional scanning. The 3D ultrasonic sensor 31 can be configured, for example, as a cross- or T-shaped cross-array sensor in which two one-dimensional arrays are orthogonalized, or as a matrix array in which transducers are arranged two-dimensionally. The second transceiver 32 outputs drive signals to each transducer of the 3D ultrasonic sensor 31 in response to commands from the work monitoring device 5. The drive signals vibrate each transducer to generate desired ultrasonic waves. The second transceiver 32 also captures reflected wave signals received by each transducer of the 3D ultrasonic sensor 31 and generates waveform data and a 3D image based on the reflected wave signals. The second transceiver 32 then outputs the waveform data and the 3D image to the work monitoring device 5. The 3D sonar 3 can also be configured as a multibeam sonar. The three-dimensional sonar 3 can also be configured to transmit and receive using a dynamic focusing method in which waves are transmitted by one transducer and received by a two-dimensional array sensor.

[0025] The 2D sonar 2 and the 3D sonar 3 are installed at positions different from the working device 71. The 2D sonar 2 and the 3D sonar 3 are mounted on, for example, a mobile device that moves on the water surface or underwater, or a mobile device that moves on a floor surface.

[0026] The work monitoring device 5 is a computer equipped with a storage device such as RAM or ROM and a processing device such as a CPU or MPU. The work monitoring device 5 is placed on a large mother ship or on land, and is electrically connected to the 2D sonar 2 and the 3D sonar 3 via cables (not shown). The work monitoring device 5 has programs and various information required to monitor underwater work by the work device 71 stored in advance in the storage device, and the processing device realizes various functions by reading the programs and various information from the storage device as appropriate and executing processing in accordance with the programs.

[0027] 2, before the monitored work device 71 starts work, the underwater work system of this embodiment measures (captures 2D images) the work environment including the stationary work device 71 using 2D sonar 2, and measures (captures 3D images) the work environment including the work device 71 using 3D sonar 3. The 2D position information of a specific part (e.g., reflectors 715, 716) of the work device 71, which is the measurement result of 2D sonar 2, is missing one dimension of position information (e.g., pitch angle φ, described below), compared to the 3D position information of the specific part (reflectors 715, 716) of the work device 71 obtained from the measurement result of 3D sonar 3. Therefore, the 2D position information of the specific part obtained from the measurement result of 2D sonar 2 is associated with the 3D initial position of the specific part (initial value of the 3D position information) calculated from the measurement result of 3D sonar 3. Next, when the work device 71 starts working, as shown in FIG. 3, measurements (recording of 2D images) of the work device 71 are continued using only the 2D sonar 2, thereby visually monitoring the work of the work device 71. In this case, the underwater work system updates the 2D position information of specific parts (reflectors 715, 716) of the work device 71 for each measurement by the 2D sonar 2. In addition, immediately after work begins, the 3D position information of the specific parts is estimated and updated using the updated 2D position information (measurement results of the 2D sonar 2) of the specific parts based on the 3D initial position before work begins (measurement results of the 3D sonar 3) associated with the 2D position information of the specific parts. Thereafter, the 3D position information is repeatedly estimated and updated using the updated 2D position information (measurement results of the 2D sonar 2) of the specific parts based on the updated 3D position information (estimated value) of the specific parts. As a result, recording and display of 2D images continues while the work device 71 is working, enabling real-time monitoring (visualization). Furthermore, the three-dimensional position information of the specific parts (reflectors 715, 716) of the work device 71 during work is repeatedly updated by estimation using the two-dimensional position information of the specific parts (measurement results of the two-dimensional sonar 2) that is updated for each measurement by the two-dimensional sonar 2, starting from the three-dimensional initial position of the specific parts before work began, which is calculated based on the measurement results of the three-dimensional sonar 3. This makes it possible to grasp the three-dimensional position information of the specific parts that cannot be measured by measurement using the two-dimensional sonar 2 alone.

[0028] To achieve this processing, the work monitoring device 5 includes functional units such as a sonar control unit 51, a two-dimensional image storage unit 52, a three-dimensional image storage unit 53, a camera image storage unit 54, an identification point setting unit 55, a three-dimensional initial position calculation unit 56, an identification point matching unit 57, a time series image calculation unit 58, and a three-dimensional position information update unit 59.

[0029] The sonar control unit 51 controls the transmission and reception of the 2D sonar 2 via the first transceiver 22. The sonar control unit 51 sets the scanning conditions of the ultrasonic beam according to commands from the work setting unit 81 of the work command device 80 and a monitoring range (described below) from the discrimination point setting unit 55, and outputs commands for the 2D sonar 2 to the first transceiver 22.

[0030] The two-dimensional image storage unit 52 stores the waveform data and two-dimensional image data before and during work output from the two-dimensional sonar 2 in a storage device, and displays the two-dimensional image data as a two-dimensional image (see, for example, FIGS. 8 and 11 described below) on the first monitor 6. The two-dimensional image storage unit 52 outputs the waveform data and two-dimensional image before work starts to the discrimination point setting unit 55, and outputs the waveform data and two-dimensional image during work to the time-series image calculation unit 58. The waveform data and two-dimensional image before work starts are also output to the discrimination point matching unit 57 via the discrimination point setting unit 55.

[0031] The three-dimensional image storage unit 53 stores the waveform data and three-dimensional image output from the three-dimensional sonar 3 in a storage device, and displays the three-dimensional image on the second monitor 7. Furthermore, it outputs the waveform data and three-dimensional image to the discrimination point setting unit 55 and the three-dimensional initial position calculation unit 56.

[0032] The camera image storage unit 54 stores the image data captured by the optical camera 3 in a storage device, and displays the image data on the third monitor 8.

[0033] The discrimination point setting unit 55 selects discrimination targets to be identified by ultrasonic measurement of the 2D sonar 2 in the working device 71 to be monitored in accordance with a pre-planned task (work plan). In this embodiment, a first reflector 715 and a second reflector 716 attached to the working arm 711 of the working device 71 shown in FIG. 2 are selected. Generally, specific portions (roots and tips) of the reflectors 715 and 716 are set as discrimination points for each YZ cross section (yaw angle θ) of the 3D image data before the start of the task output from the 3D image storage unit 53 and the 2D image data before the start of the task output from the 2D image storage unit 52 (see FIGS. 8 and 9 described below). Furthermore, the discrimination point setting unit 55 sets an update area of ​​the 2D image as a monitoring range during the task in accordance with the positions of the set discrimination points (the reflectors 715 and 716 of the working device 71). The discrimination point setting unit 55 outputs information about the set discrimination point to the three-dimensional initial position calculation unit 56, and also outputs information about the set update area (monitoring range) of the two-dimensional image during work to the sonar control unit 51 and the time-series image calculation unit 58. The method of setting the discrimination point by the discrimination point setting unit 55 and the method of setting the monitoring range of the discrimination point will be described in detail later.

[0034] Based on the 3D image output from the 3D image storage unit 53, the 3D initial position calculation unit 56 calculates the 3D position (for example, the 3D position in the XYZ space of the world coordinate system described below) of each discrimination point (specific portion of the reflectors 715, 716) of the working device 71 set by the discrimination point setting unit 55 before the start of work, and converts this into the 3D initial position (i.e., the distance r and pitch angle φ in each YZ plane (yaw angle θ)) of the discrimination point before the start of work, with the installation position of the 2D ultrasonic sensor 21 of the 2D sonar 2 as the reference point (origin). Note that the initial information on the pitch angle φ of each discrimination point calculated by the 3D initial position calculation unit 56 and with the installation position of the 2D ultrasonic sensor 21 as the reference point is information that cannot be acquired from the 2D image in the 2D image storage unit 52. The three-dimensional initial position calculation unit 56 outputs the calculation result, that is, the three-dimensional initial position of each discrimination point (distance r and pitch angle φ in each YZ plane (yaw angle θ)) with the installation position of the two-dimensional ultrasonic sensor 21 as the reference point, to the discrimination point matching unit 57 and the three-dimensional position information update unit 59. The calculation method of the three-dimensional initial position calculation unit 56 will be described in detail later.

[0035] The discrimination point matching unit 57 matches the 3D initial positions of the multiple discrimination points (specific parts of the reflectors 715 and 716) calculated by the 3D initial position calculation unit 56 with the 2D position information of the multiple discrimination points (specific parts of the reflectors 715 and 716) of the working device 71 in the 2D image before the start of work, which is input from the 2D image storage unit 52 via the discrimination point setting unit 55, thereby associating the 3D initial positions, including the missing position information (pitch angle), with the 2D position information of each discrimination point before the start of work. The association of the 3D initial positions of each discrimination point before the start of work with the 2D position information of each discrimination point before the start of work is performed for each YZ plane (yaw angle θ) of the 2D image. The discrimination point matching unit 57 outputs the 2D position information of each discrimination point of the working device 71 before the start of work and the 3D initial positions of each discrimination point associated with that information to the 3D position information update unit 59. The position information of each discrimination point in the 2D image before the start of work contains only distance information in each YZ plane (see FIG. 8, described later). On the other hand, the initial position of each discrimination point calculated by the three-dimensional initial position calculation unit 56 includes information on the pitch angle φ in addition to information on the distance r from the installation position of the two-dimensional ultrasonic sensor 21 as the reference point in each YZ plane (yaw angle θ) (see FIG. 9 described later).

[0036] Immediately after the start of work, the discrimination point matching unit 57 estimates the three-dimensional position information of each discrimination point during work by using update information (two-dimensional position information) of each discrimination point, which will be described later, from the time-series image calculation unit 58, based on the three-dimensional initial position before work started associated with the two-dimensional position information of each discrimination point before work started, and updates the three-dimensional position information of the estimated result by outputting it as new position information of each discrimination point to the three-dimensional position information update unit 59. Thereafter, during work, the discrimination point matching unit 57 successively estimates the three-dimensional position information of each discrimination point during work by using the two-dimensional position information of the discrimination point updated by the time-series image calculation unit 58, based on the updated three-dimensional position information (estimated value) of the discrimination point, and successively updates the three-dimensional position information by outputting it as new position information to the three-dimensional position information update unit 59. The processing method of the discrimination point matching unit 57 will be described in detail later.

[0037] The time-series image calculation unit 58 sequentially updates the two-dimensional position information of the plurality of identification points (specific portions of the reflectors 715, 716) of the working device 71 based on the two-dimensional images during work output for each measurement by the two-dimensional sonar 2. The time-series image calculation unit 58 sequentially outputs the updated two-dimensional position information of the identification points to the identification point matching unit 57. The processing method of the time-series image calculation unit 58 will be described in detail later.

[0038] Before work begins, three-dimensional position information update unit 59 outputs the three-dimensional initial position of the identification point of work device 71 output from three-dimensional initial position calculation unit 56 to three-dimensional position information storage unit 83 of work command device 80. During work, three-dimensional position information (estimated value) of the identification point during work output from identification point matching unit 57 is output to three-dimensional position information storage unit 83 of work command device 80. In this way, the process by which the identification point of work device 71 is displaced from its initial position due to work, from before work begins to during work, is recorded.

[0039] Next, the procedure of the underwater work method using the underwater work system according to the first embodiment will be explained. First, the procedure before the start of underwater work will be explained using Figures 1 to 9. Figure 4 is a flow chart showing an example of the procedure of the underwater work method using the underwater work system according to the first embodiment shown in Figure 1.

[0040] In Figure 4, the underwater work system shown in Figure 1 measures the underwater work environment before work begins (step S10) and creates a 3D map of the underwater work environment. If the water is murky at the time of measurement, the underwater work environment is measured using a 3D sonar mounted on the water surface or on a mobile device that moves underwater. On the other hand, if the impact of murky water is small, optical measurements such as those using a laser range finder, light cutting method, or stereo camera can be used instead of measurement using the 3D sonar 3. Note that if a 3D map of the underwater work environment can be obtained in advance, the processing (measurement) of step S10 can be omitted.

[0041] Next, the work device 71 is placed at a work position in the water based on the 3D map of the underwater work environment created in step S10 (step S20). The work position is determined by the 3D map of the underwater work environment and the work content (work plan).

[0042] Next, before work begins, the underwater work system uses the 2D sonar 2 to measure the underwater work environment including the work device 71 to record 2D images, and also uses the 3D sonar 3 to measure the underwater work environment including the work device 71 to record 3D images (step S30). The 2D sonar 2 outputs waveform data, which is the result of measurement using ultrasonic waves, and a 2D image generated from the waveform data to the 2D image storage unit 52 of the work monitoring device 5. The 2D image storage unit 52 stores the waveform data and 2D image from the 2D sonar 2 in a storage device and displays the 2D image on the first monitor 6. The 3D sonar 3 outputs the waveform data, which is the result of measurement using ultrasonic waves, and a 3D image generated from the waveform data to the 3D image storage unit 53 of the work monitoring device 5. The 3D image storage unit 53 stores the waveform data and 3D image from the 3D sonar 3 in a storage device and displays the 3D image on the second monitor 7. In addition, when the influence of turbid water is small, optical three-dimensional measurement such as laser range finder, light cutting method, stereo camera, etc. can be used instead of measurement by three-dimensional sonar 3.

[0043] Specifically, as shown in FIG. 2, for example, the two-dimensional sonar 2 has a pitch angle φ (vertical tilt angle with respect to the horizontal line) of angle φ 2D The ultrasonic beam that spreads within the range (range of the solid line) is transmitted toward the working device 71. On the other hand, the 3D sonar 3 transmits the ultrasonic beam at a pitch angle φ 3DThe 2D sonar 2 scans and transmits within a range indicated by a dashed line. FIG. 2 shows the ultrasonic measurement situation on a YZ cross section, with the origin O of the world coordinate system being the intersection of the Z axis passing through the transducer center (installation position) Z1 of the 2D sonar 2 and the underwater floor. Unless otherwise specified, it is preferable that the transducer centers Z1 and Z2 of the 2D sonar 2 and the 3D sonar 3 are positioned on the Z axis, as shown in FIG. 2. In this case, the 2D sonar 2 and the 3D sonar 3 share the same azimuth angle (yaw angle θ in FIG. 5) on the XY plane of the world coordinate system. Furthermore, before starting work, the 2D sonar 2 scans with an ultrasonic beam within a range of a yaw angle θs, as shown in FIG. 5, for example. FIG. 5 shows the measurement situation of the 2D sonar on the XY plane of the world coordinate system when the floor in FIG. 2 is viewed from the water surface. In the XY plane shown in FIG. 5, the working arm 711 of the working device 71 is positioned so that it extends along a yaw angle θ = 0°. In this case, the transmission and reception range of the ultrasonic beam of the two-dimensional sonar 2 is within a range of ±θs / 2 symmetrically centered on the Y axis (θ=0°), and the scanning angle interval of the ultrasonic beam is Δθ. It is essential that the two-directional scanning range of the ultrasonic beam of the three-dimensional sonar 3 on the YZ plane and XY plane of the world coordinate system includes at least the working arm 711 of the work device 71 to be monitored within the measurement range, and it is preferable to set it so as to cover as much of the transmission range of the ultrasonic beam of the two-dimensional sonar 2 as possible (see Figure 2, for example).

[0044] As shown in Figures 6 and 7, a first reflector 715 and a second reflector 716 to be identified by the 2D sonar 2 are installed on the surface of a work arm 711 of a work device 71 that is the monitoring target of the underwater work monitoring system 1. Figure 6 is a diagram showing the dimensions of the two reflectors installed on the work arm of the work device that is the monitoring target in the underwater work system according to the first embodiment, on the YZ cross section of the world coordinate system in Figure 2, and Figure 7 is a diagram showing the dimensions of the work arm and reflectors when the work arm of the work device shown in Figure 6 is viewed from arrow view VII.

[0045] Both reflectors 715, 716 reflect the ultrasonic beams transmitted from the 2D sonar 2 and 3D sonar 3 without much attenuation, allowing them to be received by the ultrasonic sensors 21, 31, making it easier to distinguish them from other structures. In particular, backscattering of ultrasonic waves is likely to occur at the bases (corners) and tips of the reflectors 715, 716, increasing the signal strength of the reflected waves. Therefore, in this embodiment, the base a and tip b of the first reflector 715 and the base c and tip d of the second reflector 716 are selected as the identification targets (identification points) for the ultrasonic measurements of the 2D sonar 2 and 3D sonar 3.

[0046] As shown in Figure 7, for example, the working arm 711 has a length L in the longitudinal direction and a width W. As shown in Figures 6 and 7, for example, the first reflector 715 arranged on the tip side of the working arm 711 has a height h1 from the base (connector) a to the tip b, and a width W that is the same as the width of the working arm 711. The second reflector 716 arranged on the base end side of the working arm 711 has a height h2 from the base (connector) c to the tip d, and a width W that is the same as the width of the working arm 711. The deviation between the height h1 of the first reflector 715 and the height h2 of the second reflector 716 is Δh, and the second reflector 716 is higher than the first reflector 715. The distance between the first reflector 715 and the second reflector 716 is k. When the three-dimensional positions of the base a and tip b (discrimination points) of the first reflector 715 and the three-dimensional positions of the base c and tip d (discrimination points) of the second reflector 716 are displaced in response to the operation (rotation) of the working arm 711, geometric constraints arise due to the dimensional settings of the above-mentioned working arm 711 and both reflectors 715, 716.

[0047] The measurement results of the 2D sonar 2 before the start of work in step S30 are, for example, waveform data as shown in the lower diagram of FIG. 8. FIG. 8 shows an example of waveform data, which is the measurement results of the 2D sonar in the underwater work system according to the first embodiment before the start of work, and a 2D image generated based on the waveform data. In the lower diagram of FIG. 8, the horizontal axis r represents distance, and the vertical axis represents the signal strength of the reflected wave. Note that (θ) on the horizontal axis r represents the beam scanning angle of the 2D sonar 2, i.e., the yaw angle. FIG. 8 shows waveform data when the yaw angle θ is 0°. By combining waveform data for yaw angles θ = 0, ±Δθ, ±2Δθ, ±3Δθ, ..., a 2D image such as that shown in the upper diagram of FIG. 8 can be generated. The 2D image shown in the upper diagram of FIG. 8 has the transducer center Z1 of the 2D ultrasonic sensor 21 of the 2D sonar 2 as the origin and is displayed according to the signal strength within the scanning range of the yaw angle θs. The 2D image includes two-dimensional position information, i.e., the yaw angle θ (scanning angle) and the distance r.

[0048] After step S30 is completed, identification points that are easy to identify by ultrasonic measurement are set on the monitored working device 71 (step S40). In this embodiment, as described above, the base portion a and tip portion b of the first reflecting plate 715 and the base portion c and tip portion d of the second reflecting plate 716 installed on the working arm 711 shown in Fig. 6 are selected as identification targets (identification points) for ultrasonic measurement.

[0049] Specifically, the discrimination point setting unit 55 of the work monitoring device 5 sets the base a and tip b of the first reflector 715 and the base c and tip d of the second reflector 716 as discrimination points based on the two-dimensional image before the start of work stored in the storage device by the two-dimensional image storage unit 52. For example, for the waveform data and two-dimensional image shown in FIG. 8, the discrimination points are set to the base a and tip b of the first reflector 715 and the base c and tip d of the second reflector 716, respectively, according to the distance r and signal strength. The waveform data shown in FIG. 8 is for a yaw angle θ = 0°. It is preferable to set such discrimination points for each cross section in the entire range where the reflected wave is received (i.e., to set multiple discrimination points on the four thick lines of the two-dimensional image in the upper diagram of FIG. 8). However, it is not necessary to set the discrimination points for the entire range of cross sections where the reflected wave is received. The above-mentioned dimensional information (see FIGS. 6 and 7) of the working arm 711 and the reflectors 715 and 716 is used to set and confirm these identification points.

[0050] Furthermore, the discrimination point setting unit 55 sets the base a and tip b of the first reflector 715 and the base c and tip d of the second reflector 716 as discrimination points based on the 3D image before the start of work stored in the storage device by the 3D image storage unit 53. For example, on the cross-sectional view of the 3D image shown in FIG. 9, the discrimination points are set to the base a and tip b of the first reflector 715 and the base c and tip d of the second reflector 716. FIG. 9 is an explanatory diagram showing an example of a method for associating the 3D sonar measurement results and the 2D sonar measurement results before the start of work for the discrimination points of the reflectors in the underwater work system according to the first embodiment on a YZ cross section of the world coordinate system. FIG. 9 shows the YZ cross section when the yaw angle θ is 0°. It is preferable to set each discrimination point in this manner for each YZ cross section covering the entire range in which reflected waves are received. However, it is not necessary to set the discrimination points for all cross sections corresponding to the entire range in which reflected waves are received. The above-mentioned dimensional information (see FIGS. 6 and 7) of the working arm 711 and both reflectors 715 and 716 is used to set and confirm these identification points.

[0051] Next, based on the 3D image that is the measurement result of the 3D sonar 3, the 3D initial position of each discrimination point on the work arm 711 before the start of work is calculated (step S50). The 3D initial position is expressed by the distance r and pitch angle φ, with the installation position Z1 of the 2D ultrasonic sensor 21 of the 2D sonar 2 as the reference point, in each YZ cross section (yaw angle θ as the beam scanning angle) of the 3D image. This is because the 2D position information of the discrimination point that is the measurement result of the 2D sonar 2 is expressed by the yaw angle θ and distance r, which are the beam scanning angle with the installation position Z1 of the 2D ultrasonic sensor 21 as the reference point.

[0052] Specifically, the three-dimensional initial position calculation unit 56 of the work monitoring device 5 first acquires three-dimensional position information in the XYZ space of the world coordinate system of each discrimination point (the base portion a and the tip portion b of the first reflector 715 and the base portion c and the tip portion d of the second reflector 716) set by the discrimination point setting unit 55 from the three-dimensional image output from the three-dimensional image storage unit 53. The acquisition of the three-dimensional position information in the XYZ space of the world coordinate system is performed for each YZ cross section of the three-dimensional image. Furthermore, the three-dimensional position information in the XYZ space of the world coordinate system of each discrimination point in each YZ cross section is converted into a distance r and a pitch angle φ with the installation position Z1 of the two-dimensional ultrasonic sensor 21 in each YZ cross section as the reference point.

[0053] 9, which shows the YZ cross section at yaw angle θ=0°, the three-dimensional position information of the base portion a and tip portion b of the first reflector 715 is (0, Ya0, Za0) and (0, Yb0, Zb0), respectively. The three-dimensional position information of the base portion c and tip portion d of the second reflector 716 is (0, Yc0, Zc0) and (0, Yd0, Zd0), respectively. By using the three-dimensional position information of these discrimination points in the XYZ space of the world coordinate system using the following equations (1) and (2), the distance rj and pitch angle φj to the discrimination points, with the transducer center (installation position) Z1 of the two-dimensional ultrasonic sensor 21 as the reference point, are calculated.

[0054]

number

[0055]

number

[0056] Here, the distance rj, pitch angle φj, coordinate Yj, and symbol j included in the coordinate Yj indicate either the base a and tip b of the first reflector 715 or the base c and tip d of the second reflector 716. That is, ra0 and φa0 indicate the distance and pitch angle of the base a of the first reflector 715 before work begins, and rb0 and φb0 indicate the distance and pitch angle of the tip b of the first reflector 715 before work begins. rc0 and φc0 indicate the distance and pitch angle of the base c of the second reflector 716, and rd0 and φd0 indicate the distance and pitch angle of the tip d of the second reflector 716 before work begins. The coordinate Yj and coordinate Zj indicate the three-dimensional position information of the base a and tip b of the first reflector 715 and the base c and tip d of the second reflector 716 before work begins.

[0057] Next, the three-dimensional initial position of each discrimination point calculated in step S60 is associated with the two-dimensional position information of each discrimination point of the work arm 711 obtained from the measurement results (waveform data or two-dimensional image) of the two-dimensional sonar 2 (step S60). The two-dimensional position information of each discrimination point obtained from the measurement results (waveform data or two-dimensional image) of the two-dimensional sonar 2 contains only distance information for each yaw angle θ indicating the scanning angle. On the other hand, the three-dimensional initial position of each discrimination point obtained from the measurement results of the three-dimensional sonar 3 contains information on the pitch angle φ in addition to the distance r for each yaw angle θ, as described above. Therefore, in this embodiment, a plurality of discrimination points set on each cross section (yaw angle θ) of the three-dimensional image that is the measurement result of the three-dimensional sonar 3 are compared with a plurality of discrimination points set on each cross section of the two-dimensional image that is the measurement result of the two-dimensional sonar 2 to identify each discrimination point, thereby associating the two-dimensional position information of each discrimination point (root a and tip b of the first reflector 715, root c and tip d of the second reflector 716) with the three-dimensional initial position (the calculation result of step S50). The association of the two-dimensional position information of each discrimination point with the three-dimensional initial position is performed for each cross section (yaw angle θ) where reflected waves from the reflectors 715 and 716 are received.

[0058] Specifically, the system identifies multiple discrimination points set in the waveform data for each cross section (yaw angle θ) of the two-dimensional image by comparing them with the multiple discrimination points set in the three-dimensional image using the distance rj calculated in step S60 for each cross section (yaw angle θ). For example, the system compares the distances ra0, rb0, rc0, and rd0, which are information on the three-dimensional initial positions of the base a and tip b of the first reflector 715 and the base c and tip d of the second reflector 716 shown in Fig. 9, with the distances ra0, rb0, rc0, and rd0 in the waveform data shown in Fig. 8, and identifies the discrimination points whose distances in the three-dimensional initial positions and the distances in the two-dimensional position information are most similar as the same discrimination point. The system then associates the two-dimensional position information (yaw angle and distance) of the identified discrimination points with the three-dimensional initial positions (distance and pitch angle at the same yaw angle). Here, by moving the working arm 711 a predetermined distance, the distance after movement of each discrimination point changes in the same way in both the 3D and 2D information cases, making it possible to confirm that each discrimination point has been correctly identified. This makes it possible to associate accurate pitch angle φ information (information missing from the 2D position information) calculated from the measurement results of the 3D sonar 3 with each discrimination point set in the waveform data of the measurement results of the 2D sonar 2. Similar processing is also performed on the waveform data of each cross section when the yaw angle θ = ±Δθ, ±2Δθ, ±3Δθ... (reception range of reflected waves from reflectors 715, 716) in the 2D image shown in the upper diagram of Figure 8.

[0059] As a final preparation before the start of underwater work, an update area for the 2D image is set as the monitoring area after work begins (step S70). During work, the identification points of the reflectors 715 and 716 are tracked by measurements from the 2D sonar 2 as the monitoring target of the work arm 711, so the update area for the 2D image is set according to the positions of the identification points of the reflectors 715 and 716. If the update area for the 2D image becomes larger, it will take that much longer to generate the 2D image. During work, it is necessary to reduce the load of generating the 2D image in order to perform real-time visualization.

[0060] Specifically, the discrimination point setting unit 55 sets the monitoring range (beam scanning range) to a range that is one cross section (Δθ) further outward from the outermost position of the cross sections (yaw angle θ of the scanning angle) at which reflected waves from the reflectors 715 and 716 are received before the start of work. The distance in the propagation direction of the ultrasonic beam for each cross section is set, for example, by expanding it to a range of ±Δri relative to the longest and shortest distances of the reflected waves measured before the start of work. The monitoring range is, for example, the shaded area shown in FIG. 8. It is also possible to expand it to a range of ±Δri according to the longest and shortest distances measured during work. It is also possible to expand it to a range of ±Δri relative to the longest and shortest distances of a predetermined operating range.

[0061] Next, the procedure after work starts will be explained using Figures 3 to 4 and 10 to 11. Figure 10 is a diagram showing the relationship between waveform data, which is the measurement result of a 2D sonar when monitoring work equipment during work in the underwater work monitoring system according to the first embodiment, and a 2D image. Figure 11 is an explanatory diagram showing a method for estimating three-dimensional position information of an identification point on a YZ cross section of a world coordinate system when monitoring work equipment during work in the underwater work monitoring system according to the first embodiment.

[0062] When underwater work begins, the underwater work system performs measurements using only the 2D sonar 2, and records and displays 2D images (step S110). As shown in Figure 3, the 2D sonar 2 performs measurements using ultrasonic waves on the work device 71, and outputs waveform data, which are the measurement results, and a 2D image generated from the waveform data, to the 2D image storage unit 52 of the work monitoring device 5. The 2D image storage unit 52 stores the waveform data and 2D image from the 2D sonar 2 in a storage device, and displays the 2D image on the first monitor 6.

[0063] Next, the underwater work system compares the multiple identification points in the current cycle (latest) two-dimensional image with the multiple identification points in the two-dimensional image from one cycle before (the most recent) and updates the two-dimensional position information of each identification point (step S120). Specifically, each time the 2D sonar 2 outputs a measurement result (updates the two-dimensional image) (each processing cycle), the time-series image calculation unit 58 of the work monitoring device 5 compares the multiple identification points (the base a and tip b of the first reflector 715 and the base c and tip d of the second reflector 716) in each cross section (yaw angle θ) of the latest two-dimensional image with the multiple identification points in each cross section (yaw angle θ) of the most recent two-dimensional image, thereby identifying each identification point in each cross section of the latest two-dimensional image. This determines the two-dimensional position information of each identified identification point (the distance r in each cross section θ), and the two-dimensional position information of each identification point is updated to the information of the latest (current cycle) measurement result. In the case of immediately after the start of work, a plurality of identification points in each cross section of the two-dimensional image before the start of work are compared with a plurality of identification points in each cross section of the two-dimensional image of the measurement results immediately after the start of work.

[0064] To clearly explain how the time-series image calculation unit 58 updates the two-dimensional position information of the discrimination points, we will use an example in which the posture of the work arm 711 clearly changes between before and immediately after the start of work. However, because the two-dimensional image update rate is less than one second, the actual range of movement of the reflectors 715 and 716 on the work arm 711 is very small. Figure 10 shows the YZ cross section of the work arm 711 rotated downward clockwise around the rotation axis from the posture shown in Figure 9. The upper diagram in Figure 11 is a two-dimensional image measured when the posture is as shown in Figure 10, and the lower diagram is waveform data for a cross section at a yaw angle θ = 0°. In Figure 11, the discrimination points and waveform data in the two-dimensional image before the start of work are indicated by dotted lines.

[0065] The distance (ra1, rb1, rc1, rd1) of work arm 711 after work starts from each discrimination point (a, b, c, d) on reflectors 715 and 716 satisfies the constraint condition shown in the following equation (3) based on the relationship between the distance (ra0, rb0, rc0, rd0) before work starts and the direction of movement of work arm 711. Furthermore, the signal strength (Sai, Sbi, Sci, Sdi) of each discrimination point (a, b, c, d) satisfies the constraint condition shown in the following equation (4). That is, this is the condition that the signal strength of the reflected wave from the corner reflection is greater than that from the tip reflection on reflectors 715 and 716. Here, "0" in the symbol i indicates before work starts, and "1" indicates after work starts.

[0066]

number

[0067]

number

[0068] Based on the constraint of equation (3), the waveform portion at the shortest distance r in the waveform data shown in the lower diagram of FIG. 11 is identified as a signal of a wave reflected from discrimination point b at the tip of the first reflector 715, and the waveform portion at the longest distance r is identified as a signal of a wave reflected from discrimination point c at the base of the second reflector 716. Based on the constraint of equation (4), of the remaining waveform portions in the waveform data shown in the lower diagram of FIG. 11, the waveform portion with a relatively high signal intensity is identified as a signal of a wave reflected from discrimination point a at the base of the first reflector 715, and the waveform portion with a relatively low signal intensity is identified as a signal of a wave reflected from discrimination point d at the tip of the second reflector 716. Information on the distances ra1, rb1, rc1, and rd1 of the identified discrimination points a, b, c, and d from the installation position Z1 of the 2D ultrasonic sensor 21 can be obtained from the waveform data shown in the lower diagram of FIG. 11. The waveform data shown in the lower diagram of FIG. 11 shows a cross section where the yaw angle θ is 0°. For the waveform data of each cross section, the constraints of equations (3) and (4) are used to obtain information on the distances ra1, rb1, rc1, and rd1 of the above-mentioned respective discrimination points a, b, c, and d.

[0069] In this way, the time-series image calculation unit 58 collates and identifies each of the identification points a, b, c, and d of the reflectors 715 and 716 in each cross section of the two-dimensional image that is updated for each measurement by the two-dimensional sonar 2, and obtains information on the distances ra1, rb1, rc1, and rd1 of each of the identified identification points a, b, c, and d from the two-dimensional image (waveform data) to update the most recent distance information. In other words, the two-dimensional position information (information on the yaw angle θ and the distance r) of each of the identification points a, b, c, and d of the reflectors 715 and 716 is updated for each measurement by the two-dimensional sonar 2.

[0070] Next, the underwater work system estimates and updates the three-dimensional position information (yaw angle θ, distance r, pitch angle φ) of each identification point a, b, c, and d of reflectors 715 and 716 using the two-dimensional position information of each identification point a, b, c, and d updated in step S120, starting from the three-dimensional initial position associated in step S60 (step S130). Immediately after the start of work, the discrimination point matching unit 57 of the work monitoring device 5 estimates the three-dimensional position information of each discrimination point a, b, c, and d of the work arm 711 during work by a convergence calculation described below, using the two-dimensional position information of each discrimination point a, b, c, and d (measurement results of the two-dimensional sonar 2) of the work arm 711 updated in step S120, starting from the three-dimensional initial position (measurement results of the three-dimensional sonar 3) associated with the two-dimensional position information of each discrimination point a, b, c, and d of the reflectors 715 and 716 before the start of work in step S60, and outputs the estimated three-dimensional position information of each discrimination point a, b, c, and d to the three-dimensional position information update unit 59. The three-dimensional position information update unit 59 updates the three-dimensional position information of each discrimination point a, b, c, and d by outputting the estimated three-dimensional position information of each discrimination point a, b, c, and d to the work command device 80 and storing it in the three-dimensional position information storage unit 83. After that (while the work is ongoing), the three-dimensional position information (estimated value) of each discrimination point a, b, c, d updated in step S130 is used as the two-dimensional position information (measurement results of two-dimensional sonar 2) of each discrimination point a, b, c, d updated in step S120, and the three-dimensional position information of each discrimination point a, b, c, d is repeatedly estimated and updated by convergent calculation.

[0071] Specifically, immediately after work begins, discrimination point matching unit 57 performs geometric calculations (trajectory calculations for each discrimination point a, b, c, d) in accordance with the movement of work arm 711, starting from the three-dimensional initial positions of each discrimination point a, b, c, d of reflectors 715, 716 before work begins, and predicts distances Ra1, Rb1, Rc1, Rd1 from installation position Z1 of two-dimensional ultrasonic sensor 21 to each discrimination point a, b, c, d. For example, by performing geometric calculations in accordance with the movement of work arm 711 on the YZ cross section of the three-dimensional image before work begins shown in Figure 9, the position of work arm 711 including each discrimination point a, b, c, d of reflectors 715, 716 after work begins on the YZ cross section shown in Figure 10 is predicted. Based on the position information (Y coordinates Ya1, Yb1, Yc1, Yd1 and Z coordinates Za1, Zb1, Zc1, Zd1) of each discrimination point a, b, c, d on the YZ cross section immediately after the start of work (the calculation results shown in Figure 10), distances Ra1, Rb1, Rc1, Rd1 from each discrimination point a, b, c, d to installation position Z1 of the 2D ultrasonic sensor 21 are calculated. This is to predict the position of each discrimination point a, b, c, d immediately after the start of work, using the three-dimensional initial position on the YZ cross section before the start of work as the starting point. In the geometric calculation, multiple rotation angles of the work arm 711 are predicted using the shape and dimensions of the work arm 711 and reflectors 715 and 716 shown in Figures 6 and 7.

[0072] Furthermore, as shown in the following formula (5), the sum of errors (differences) between the distance information of each discrimination point a, b, c, d predicted by the above-mentioned geometric calculation with the installation position Z1 of the two-dimensional ultrasonic sensor 21 as the reference point, starting from the three-dimensional initial position of each discrimination point a, b, c, d, and the distance information of each discrimination point a, b, c, d obtained from the two-dimensional image updated in step S120 is Sum err Calculate the sum of the calculated errors. err The convergence calculation is performed to minimize the sum of the errors, err The distances Ra1, Rb1, Rc1, Rd1 and pitch angles φa1, φb1, φc1, φd1 of each discrimination point a, b, c, d on the YZ cross section relative to the installation position Z1 of the two-dimensional ultrasonic sensor 21 when is smallest are estimated as the three-dimensional position information immediately after the start of work. This makes it possible to grasp information on the pitch angle φ of each discrimination point a, b, c, d that cannot be obtained from the measurement results of the two-dimensional sonar 2.

[0073]

number

[0074] The YZ cross section of the three-dimensional image shown in Figure 9 is for the case where the yaw angle θ = 0°. On each cross section that receives reflected waves from reflectors 715, 716 at yaw angles other than θ = 0°, the above-mentioned geometric calculations and convergence calculations using equation (5) are performed to estimate the distance R and pitch angle φ of each discrimination point a, b, c, and d on each cross section, with the installation position Z1 of the two-dimensional ultrasonic sensor 21 as the reference point, as the three-dimensional position information after the start of work.

[0075] The discrimination point matching unit 57 outputs the estimation results of the three-dimensional position information of each discrimination point a, b, c, and d after the start of work to the three-dimensional position information update unit 59. The three-dimensional position information update unit 59 outputs the estimated three-dimensional position information of each discrimination point a, b, c, and d to the work command device 80, which stores it in the three-dimensional position information storage unit 83. In this way, the three-dimensional position information of each discrimination point a, b, c, and d after the start of work (the distance R and the pitch angle φ at each cross section θ of the three-dimensional image) is updated.

[0076] The underwater work system determines whether the work has been completed (step S140 shown in FIG. 4). If it is determined that the work has not been completed (NO), the process returns to step S110 and steps S110 to S140 are repeated until it is determined in step S140 that the work has been completed (YES). In other words, until the work is completed, the underwater work system continues to record and display the two-dimensional images that are the measurement results of the two-dimensional sonar 2 at set time intervals, while updating the three-dimensional position information (distance R and pitch angle φ at each cross section θ of the three-dimensional image) of each identification point a, b, c, and d.

[0077] As described above, in this embodiment, two-dimensional and three-dimensional images are captured by ultrasonic measurement before work begins, and the two-dimensional position information of each discrimination point a, b, c, and d of the monitoring target is associated with the three-dimensional initial position. As a result, after work begins, two-dimensional ultrasonic images are captured at regular time intervals, and the three-dimensional position information (distance R and pitch angle φ at each cross section θ) of discrimination points a, b, c, and d that changes during this time interval can be calculated by convergence calculation in three-dimensional space using the information from the captured two-dimensional images and the information from the three-dimensional initial positions associated before work began (measurement results from the three-dimensional sonar 3). Therefore, even when working in turbid water, three-dimensional information of the monitoring target (discrimination points a, b, c, and d) can be grasped while checking the two-dimensional images captured by ultrasonic measurement. This allows work to be performed without reducing the work speed.

[0078] As described above, the underwater work method of the first embodiment involves recording a two-dimensional image of the underwater work environment by measurement using a two-dimensional sonar 2 before work begins with a work device 71 placed underwater, recording a three-dimensional image of the underwater work environment by measurement using a three-dimensional sonar 3 (measuring device) that can measure three-dimensional position information before work begins with the work device 71, calculating the three-dimensional initial positions θ, r, and φ of the identification points a, b, c, and d of the work device 71 from the three-dimensional image before work begins, using the installation position Z1 of the two-dimensional sonar 2 as the reference point, and matching the three-dimensional initial positions of the identification points a, b, c, and d obtained as a result of the calculation with the two-dimensional position information of the identification points a, b, c, and d in the two-dimensional image before work begins. After the work device 71 starts working, two-dimensional images are recorded and sequentially updated by continuous measurement from the two-dimensional sonar 2, and the sequentially updated two-dimensional images are displayed on the first monitor 6 (display device). Starting from the three-dimensional initial position that is associated with the two-dimensional position information of the discrimination points a, b, c, and d, the three-dimensional position information of the discrimination points a, b, c, and d is estimated and sequentially updated by using the two-dimensional position information of the discrimination points a, b, c, and d in the two-dimensional images that are sequentially updated after the work starts.

[0079] As described above, the underwater work system according to the first embodiment comprises a work device 71 that performs work underwater, and an underwater work monitoring system 1 that visualizes and monitors the underwater work of the work device 71. The underwater work monitoring system 1 comprises a 2D sonar 2 that can measure 2D position information by emitting ultrasonic waves, a 3D sonar 3 (measuring device) that can measure 3D position information, a first monitor 6 (display device) that displays 2D images captured by measurement by the 2D sonar 2, and a work monitoring device 5 that calculates 3D position information of identification points a, b, c, and d of the work device 71 based on the measurement results of the 2D sonar 2 and the 3D sonar 3 (measuring device). The work monitoring device 5 is configured to calculate the three-dimensional initial positions of the discrimination points a, b, c, and d of the work device 71 from the three-dimensional image recorded by measurement with the three-dimensional sonar 3 (measuring device) before the work device 71 starts working, with the installation position Z1 of the two-dimensional sonar 2 as the reference point, and to correspond the three-dimensional initial positions of the discrimination points a, b, c, and d obtained as a result of the calculation to the two-dimensional position information of the discrimination points a, b, c, and d in the two-dimensional image recorded by measurement with the two-dimensional sonar 2 before the work device 71 starts working, and to estimate and sequentially update the three-dimensional position information of the discrimination points a, b, c, and d by using the two-dimensional position information of the discrimination points a, b, c, and d in the two-dimensional image that is sequentially updated by measurement with the two-dimensional sonar 2 after the work device 71 starts working, starting from the three-dimensional initial positions corresponded to the two-dimensional position information of the discrimination points a, b, c, and d.

[0080] According to this method and configuration, 2D images are captured by 2D sonar 2 during operation of the work device 71 and displayed on the first monitor 6 (display device). This reduces the time required to visualize underwater work compared to capturing and displaying 3D images during operation. Furthermore, the 3D position information of identification points a, b, c, and d during operation is estimated and updated using the 2D position information of identification points a, b, c, and d in the 2D image, which is updated during operation starting from the 3D initial position calculated from the 3D image before operation begins. This allows the 3D position information of identification points a, b, c, and d (monitoring targets) that cannot be measured from the 2D image to be determined. These factors enable the accurate determination of the position of the monitoring target when performing underwater work while minimizing a decrease in work speed in turbid water.

[0081] Furthermore, in the underwater work method and underwater work system according to this embodiment, the measuring device is a 3D sonar 3 that can measure 3D position information by emitting ultrasonic waves. With this method and configuration, it is possible to measure 3D position information of the monitored object whether the underwater environment before work begins is clear water or muddy water.

[0082] [Modification of the first embodiment] Next, an underwater work system and an underwater work method according to a modified example of the first embodiment will be described with reference to Figure 12. Figure 12 is a block diagram showing the configuration of an underwater work system according to a modified example of the first embodiment of the present invention. In Figure 12, the same reference numerals as those shown in Figures 1 to 11 indicate similar parts, and therefore detailed description thereof will be omitted.

[0083] The underwater work system according to a modification of the first embodiment shown in Figure 12 differs from the first embodiment in that the underwater work monitoring system 1A does not include both 2D sonar 2 and 3D sonar 3 as in the first embodiment, but includes a dual-purpose sonar 2A that combines the functions of both 2D and 3D sonar. Specifically, the dual-purpose sonar 2A is equipped with a cross array sensor 21A in which one 1D array sensor is arranged in a cross or T shape so that another 1D array sensor is orthogonal to another 1D array sensor, and the array sensor is divided into a transmitting and receiving array sensor, and a transceiver 22A corresponding to the cross array sensor 21A. With this configuration, unlike the first embodiment, only one transceiver 22A is required.

[0084] In this modified example, the method of monitoring underwater work is the same as in the first embodiment, except that the dual-purpose sonar 2A measures two-dimensional images of the underwater work environment and two-dimensional position information of the monitored object (identification points a, b, c, d of reflectors 715, 716), as well as three-dimensional images of the underwater work environment and three-dimensional position information of the monitored object (identification points a, b, c, d of reflectors 715, 716).

[0085] In the underwater operation method according to a modified example of the first embodiment described above, a dual-purpose sonar 2A (ultrasonic sonar) having a cross array sensor in which a first one-dimensional array in which multiple transducers are arranged in one direction and a second one-dimensional array in which multiple transducers are arranged in one direction are arranged perpendicular to each other performs two-dimensional sonar measurements as well as three-dimensional sonar measurements.

[0086] In addition, in the underwater work system of this modified example, a dual-purpose sonar 2A (ultrasonic sonar) having a cross array sensor in which a one-dimensional array in which multiple transducers are arranged in one direction and another one-dimensional array in which multiple transducers are arranged in one direction are arranged perpendicular to each other is configured to function as both a two-dimensional ultrasonic sonar and a three-dimensional ultrasonic sonar.

[0087] According to this method and configuration, the dual-purpose sonar 2A can be configured to include only one transceiver 22A, making it possible to make the system more compact and reduce costs than in the first embodiment.

[0088] [Second embodiment] Next, an underwater work system and an underwater work method according to a second embodiment of the present invention will be described with reference to Figures 13 and 14. Figure 13 is a flowchart showing an example of an underwater work method using the underwater work system according to the second embodiment of the present invention. Figure 14 is a block diagram showing the configuration of an underwater work system that executes the underwater work method according to the second embodiment shown in Figure 13. In Figures 13 and 14, parts that are the same as those shown in Figures 1 to 12 have the same reference numerals, so detailed description will be omitted.

[0089] The underwater work method using the underwater work system according to the second embodiment shown in Figures 13 and 14 differs from the first embodiment in that an error judgment is performed to judge the magnitude of the error in the three-dimensional position information of identification points a, b, c, and d of the monitored reflectors 715 and 716, which is updated after work has started. In this embodiment, it is assumed that the work device 71B has an attitude sensor 717 that detects the attitude of the work arm 711 (see Figure 14). Information from an estimation calculation of the three-dimensional position information of identification points a, b, c, and d of the work arm 711, which is performed using the measurement results of the 2D sonar 2 during work (information from the calculation process in step S130 shown in Figure 13), is compared with the position information of the work arm 711 calculated from the detection value of the attitude sensor 717, to judge whether the error in the estimated three-dimensional position information of identification points a, b, c, and d is within a preset allowable error range (step S135 shown in Figure 13). If the error in the calculation result is within the allowable error range (YES in step S135 shown in FIG. 13), the work command device 80B causes the work device 71 to continue working until the work is completed. If the error in the calculation result is outside the allowable error range (NO in step S135 shown in FIG. 13), the work is either suspended or a warning of error growth is issued (step S150 shown in FIG. 13). When a warning of error growth is issued, it is possible to configure the system to determine whether to suspend or continue the work based on the presence or absence of an input from the operator monitoring the work. If the work is suspended, the work is resumed after performing the processing of steps S30 to S70 shown in FIG. 13, which are the processing procedures before the start of work. That is, two-dimensional images are recorded by measurement with the two-dimensional sonar 2 and three-dimensional images are recorded by measurement with the three-dimensional sonar 3, and based on the results of these remeasurements, the two-dimensional position information and three-dimensional position information of the identification points a, b, c, and d of the reflectors 715 and 716 are re-associated with each other. This makes it possible to reduce errors in the three-dimensional position information of the discrimination points a, b, c, and d relative to the detection values ​​of the orientation sensor 717.

[0090] Next, the configuration of an underwater work system for executing the flow of the underwater work method shown in FIG. 13 will be described.

[0091] The working apparatus 71B shown in Figure 14 is equipped with an attitude sensor 717 which detects attitude information (relative position information) of the working arm 711. The attitude sensor 717 detects, for example, the rotation angle of each joint of the working arm 711 as attitude information of the working arm 711. The attitude sensor 717 outputs the detected attitude information (rotation angle) of the working arm 711 to the working apparatus controller 72 (arm control unit 722). The working apparatus controller 72 outputs the attitude information (rotation angle) of the working arm 711 detected by the attitude sensor 717 to the work command device 80B.

[0092] The underwater work monitoring system 1B includes a work command device 80B. The work command device 80B receives attitude information (rotation angle) of the work arm 711 detected by the attitude sensor 717 from the work device controller 72. The work command device 80B has an error determination unit 84 in addition to the work setting unit 81B, work device position information storage unit 82B, and three-dimensional position information storage unit 83 of the first embodiment. The function of the three-dimensional position information storage unit 83 is the same as in the first embodiment, so a description thereof will be omitted.

[0093] The working device position information storage unit 82B stores the position information of the working device 71, including the attitude information (rotation angle) of the working arm 711 detected by the attitude sensor 717, in a storage device.

[0094] The error determination unit 84 compares the information of the estimated calculation of the three-dimensional position information of the discrimination points a, b, c, and d of the work arm 711, which is performed using the measurement results of the 2D sonar 2 during work (information of the calculation processing in step S130 shown in Figure 13), with the position information (posture information) of the work arm 711 calculated from the detection value of the posture sensor 717, and performs error determination to determine whether the error in the estimated three-dimensional position information of the discrimination points a, b, c, and d is within a preset allowable error range (see step S135 shown in Figure 14). For example, the error determination unit 84 calculates the difference (error) between the posture information of working arm 711 (for example, the rotation angle of each joint of working arm 711) which is the detection value of posture sensor 717 stored in working device position information storage unit 82B and the posture information of working arm 711 (for example, the rotation angle of each joint of working arm 711) obtained during an estimation calculation (convergence calculation) of the three-dimensional position information of identification points a, b, c, d of reflectors 715, 716 stored in three-dimensional position information storage unit 83, and determines whether the error in the calculation result is below a preset threshold. Error determination unit 84 outputs the determination result to the task setting unit 81B.

[0095] When the error determination unit 84 determines that the error in the calculation result is equal to or greater than the threshold value (NO in step S135 shown in FIG. 13), the work setting unit 81B outputs a command to suspend the work or to issue a warning of error expansion (step S150 shown in FIG. 13). Furthermore, the work setting unit 81B issues a command to execute the processing procedure before the start of the work (steps S30 to S70 shown in FIG. 13). That is, the work setting unit 81B outputs a command to redo the association between the two-dimensional position information and the three-dimensional position information of the discrimination points a, b, c, and d of the reflectors 715 and 716. Thereafter, the work setting unit 81B outputs a command to resume the work.

[0096] According to the underwater work method and underwater work system of the second embodiment, as in the first embodiment, 2D images are captured by 2D sonar 2 during operation of the work device 71B and displayed on the first monitor 6 (display device). This reduces the time required to visualize underwater work compared to capturing and displaying 3D images during operation. Furthermore, the 3D position information of identification points a, b, c, and d during operation is estimated and updated using 2D position information of identification points a, b, c, and d in the 2D image, which is updated during operation starting from the 3D initial position calculated from the 3D image before operation begins. This allows the 3D position information of identification points (monitoring targets) that cannot be measured from 2D images to be determined. These factors enable the accurate determination of the position of the monitoring target when performing underwater work while minimizing a decrease in work speed in turbid water.

[0097] Furthermore, in the underwater work method according to this embodiment, while the work device 71B is working, the detected value of the attitude information of the work device 71B is compared with the information of the estimated calculation of the three-dimensional position information of the identification points a, b, c, and d of the work device 71B, and an error determination is performed to determine whether the error in the estimated value of the three-dimensional position information of the identification points a, b, c, and d is within the allowable range.

[0098] Furthermore, in the underwater work system according to this embodiment, the work apparatus 71B has an attitude sensor 717 that detects attitude information of the work apparatus 71B. Furthermore, the underwater work monitoring system 1B is configured to compare the attitude information detected by the attitude sensor 717 while the work apparatus 71B is working with information obtained by the work monitoring device 5 in estimating the three-dimensional position information of the identification points a, b, c, and d of the work apparatus 71B, thereby performing an error determination to determine whether or not the error in the estimated value of the three-dimensional position information of the identification points a, b, c, and d is within an allowable range.

[0099] According to this method and configuration, by performing error judgment on the estimated values ​​of the three-dimensional position information of the discrimination points a, b, c, and d during work, it is possible to grasp the positions of the discrimination points a, b, c, and d (monitoring targets) more accurately.

[0100] Furthermore, in the underwater work method according to this embodiment, the measuring device is a 3D sonar 3 capable of measuring 3D position information by transmitting ultrasonic waves. Furthermore, if the error determination determines that the error exceeds the allowable range, the work of the work device 71B is interrupted, a 2D image is recorded by measurement with the 2D sonar 2, and the 3D image is re-recorded by measurement with the 3D sonar 3. From the 3D image after the work interruption, 3D position information after the work interruption of discrimination points a, b, c, and d is calculated using the installation position Z1 of the 2D sonar 2 as a reference point, and the 3D position information after the work interruption of discrimination points a, b, c, and d obtained as a result of the calculation is associated with the 2D position information of discrimination points a, b, c, and d in the 2D image after the work interruption, and the work of the work device 71B is resumed.

[0101] Furthermore, in the underwater work system according to this embodiment, the above-mentioned measuring device is a 3D sonar 3 capable of measuring 3D position information by transmitting ultrasonic waves. Furthermore, when work by the work device 71B is interrupted because the error judgment determines that the error exceeds the allowable range, the work monitoring device 5 is configured to calculate 3D position information after the work interruption of discrimination points a, b, c, and d from 3D images re-recorded by measurement with the 3D sonar 3 after the work interruption, with the installation position Z1 of the 2D sonar 2 as the reference point, and to associate the calculated 3D position information after the work interruption of discrimination points a, b, c, and d with the 2D position information of discrimination points a, b, c, and d in the 2D images recorded by measurement with the 2D sonar 2 after the work interruption.

[0102] According to this method and configuration, if there is a large error in the estimated values ​​of the three-dimensional position information of the identification points a, b, c, and d during work, the two-dimensional position information of the identification points a, b, c, and d in the two-dimensional image captured after the work was interrupted is again associated with the three-dimensional position information of the identification points a, b, c, and d calculated from the three-dimensional image re-captured after the work was interrupted, making it possible to grasp the exact position of the monitored object (identification points a, b, c, and d) after work is resumed.

[0103] [Third embodiment] Next, an underwater work system and an underwater work method according to a third embodiment of the present invention will be described with reference to Figures 15 and 16. Figure 15 is a flowchart showing an example of an underwater work method using the underwater work system according to the third embodiment of the present invention. Figure 16 is a block diagram showing the configuration of an underwater work system that executes the underwater work method according to the third embodiment shown in Figure 15. In Figures 15 and 16, parts that are the same as those shown in Figures 1 to 14 have the same reference numerals, so detailed description will be omitted.

[0104] The underwater work method using the underwater work system according to the third embodiment shown in Figures 15 and 16 differs from the first embodiment in that, if an obstacle that may hinder the work is measured in the underwater work environment before work begins, an interference determination is made to determine whether the work device 71 will interfere with the obstacle during work. In this embodiment, if an obstacle is measured before work begins, three-dimensional position information of the obstacle is calculated from the measurement results of the 3D sonar 3, and the distance between the obstacle and the work arm 711 of the work device 71 before work begins is calculated (step S55 shown in Figure 15). After work begins, the distance between the work arm 711 and the obstacle is calculated for each estimation calculation of three-dimensional position information of identification points a, b, c, and d of the work arm 711, and the calculated distance is compared with a preset threshold to determine whether there is a possibility of the work arm 711 interfering with (contacting) the obstacle (step S135C shown in Figure 15). If it is determined that there is no possibility of interference with an obstacle by the work arm 711 (NO in step S135C), the work continues (steps S110 to S140 shown in FIG. 15). On the other hand, if it is determined that there is a possibility of interference with an obstacle (YES in step S135C), the work is either suspended or a warning of interference with an obstacle is issued (step S150C shown in FIG. 15). If a warning of interference is issued, it is possible to configure the system to determine whether to suspend or continue the work depending on whether or not there is an input from the operator monitoring the work.

[0105] If work is interrupted, the same procedure as before work began is repeated, recalculating the distance between the work arm 711 and the obstacle using the measurement results of the 3D sonar 3, and reconfirming whether or not there is a possibility of the work arm 711 interfering with the obstacle (steps S160 to S200 shown in FIG. 15). That is, 2D images are captured by remeasurement with the 2D sonar 2, and 3D images are captured by remeasurement with the 3D sonar 3 (step S160 shown in FIG. 15). Then, similar to step S40 before work began, the discrimination points of the reflectors 715 and 716 to be monitored are reset in the 2D images and the 3D images (step S170 shown in FIG. 15). Then, similar to step S50 before work began, the 3D position information of the set discrimination points a, b, c, and d, i.e., the distance r and pitch angle φ at each cross section θ of the 3D image, are recalculated (step S180 shown in FIG. 15). Next, as in step S55 before work starts, the distance between the work arm 711 and the obstacle is recalculated using the three-dimensional position information of the obstacle calculated from the measurement results of the 3D sonar 3 and the recalculated three-dimensional position information of the discrimination points a, b, c, and d (step S190 shown in FIG. 15). The result of this recalculation uses the three-dimensional position information of the discrimination points based on the measurement results of the 3D sonar 3, and can provide a more accurate result than a distance calculation using estimated values ​​of the three-dimensional position information of the discrimination points a, b, c, and d based on the measurement results of the 2D sonar 2. Furthermore, as in step S135C after work starts, the distance to the obstacle calculated in step S190 is compared with a threshold value to re-determine whether or not there is a possibility of interference of the work arm 711 with the obstacle (step S200 shown in FIG. 15).

[0106] If it is determined that there is a possibility that the working arm 711 will interfere with an obstacle (YES in step S135C), the working position of the working device 71 or working arm 711 is changed (step S210 shown in Figure 15). Then, the process returns to steps S160 to S200 shown in Figure 15, and it is reconfirmed whether or not there is a possibility that the working arm 711 will interfere with an obstacle after the working position of the working device 71 or working arm 711 has been changed. If it is determined that there is no possibility that the working arm 711 will interfere with an obstacle (NO in step S200), the process returns to steps S110 to S140 shown in Figure 15, and work is resumed.

[0107] Next, the configuration of an underwater work system for executing the flow of the underwater work method shown in FIG. 15 will be described.

[0108] 16 includes a work command device 80C in addition to a 2D sonar 2, a 3D sonar 3, a work monitoring device 5C, etc. Of the functional units of the work monitoring device 5C, the sonar control unit 51, 2D image storage unit 52, 3D image storage unit 53, camera video storage unit 54, discrimination point setting unit 55, discrimination point matching unit 57, and time-series image calculation unit 58 are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0109] As in the first embodiment, the 3D initial position calculation unit 56C of the work monitoring device 5C calculates the 3D initial positions (distance r in each YZ plane θ and pitch angle φ) of the discrimination points of the reflectors 715 and 716 before the start of work (step S50 shown in FIG. 15), and calculates the 3D position information (distance r in each YZ plane θ and pitch angle φ) of the discrimination points a, b, c, and d of the reflectors 715 and 716 based on the results of remeasurements by the 2D sonar 2 and the 3D sonar 3 after the start of work (step S180 shown in FIG. 15). In addition, the 3D position information of the obstacle is calculated based on the measurement results by the 3D sonar 3 before the start of work, and the 3D position information of the obstacle is calculated based on the remeasurements by the 2D sonar 2 and the 3D sonar 3 after the start of work. The 3D initial position calculation unit 56C outputs the calculated 3D initial positions of the discrimination points a, b, c, and d and the 3D position information of the obstacle to the 3D position information update unit 59C.

[0110] The three-dimensional position information update unit 59C outputs the three-dimensional initial positions of the discrimination points a, b, c, and d before the start of work and the three-dimensional position information of obstacles to a three-dimensional position information storage unit 83C of the work command device 80C. After the start of work, the three-dimensional position information (estimated values) of the discrimination points a, b, c, and d that are updated with each measurement by the two-dimensional sonar 2 is output to the three-dimensional position information storage unit 83C of the work command device 80C, and the three-dimensional position information of the discrimination points a, b, c, and d and the three-dimensional position information of obstacles calculated when the two-dimensional sonar 2 and three-dimensional sonar 3 are remeasured is output to the three-dimensional position information storage unit 83C of the work command device 80C.

[0111] The work command device 80C has an interference determination unit 85 in addition to the work setting unit 81, work device position information storage unit 82, and three-dimensional position information storage unit 83C of the first embodiment. The functions of the work setting unit 81 and work device position information storage unit 82 are the same as those of the first embodiment, so a description thereof will be omitted.

[0112] As in the first embodiment, the three-dimensional position information storage unit 83C stores in a storage device the three-dimensional initial positions of the identification points output from the three-dimensional position information update unit 59C before the start of work and the three-dimensional position information of the identification points that is successively output and updated from the three-dimensional position information update unit 59C after the start of work. Furthermore, the three-dimensional position information of the obstacles output from the three-dimensional position information update unit 59C before the start of work, and the three-dimensional position information of the identification points and the three-dimensional position information of the obstacles that are obtained by remeasurement by the two-dimensional sonar 2 and the three-dimensional sonar 3 and that are output from the three-dimensional position information update unit 59C after the start of work, in a storage device.

[0113] The interference determination unit 85 performs interference determination to determine whether or not the work arm 711 will interfere with an obstacle. Specifically, it calculates the distance between the work arm 711 and the obstacle using the three-dimensional position information of the obstacle before the start of work stored in the three-dimensional position information storage unit 83C and the three-dimensional initial position of the discrimination point (step S55 shown in Figure 15). Furthermore, it calculates the distance between the work arm 711 and the obstacle using the three-dimensional position information of the obstacle before the start of work stored in the three-dimensional position information storage unit 83C and the three-dimensional position information of the discrimination point updated after the start of work, and performs interference determination to determine whether or not the work arm 711 will interfere with the obstacle by comparing the calculated distance with a preset threshold (step S135C shown in Figure 15). The three-dimensional position information of the obstacle obtained by remeasurement by 2D sonar 2 and 3D sonar 3 after the start of work, which is stored in the three-dimensional position information storage unit 83C, and the three-dimensional initial position of the discrimination point are used to calculate the distance between the work arm 711 and the obstacle (step S190 shown in FIG. 15), and the calculated distance is compared with a threshold value to perform an interference determination to determine whether the work arm 711 will interfere with the obstacle (step S200 shown in FIG. 15). The interference determination unit 85 outputs the determination result to the work setting unit 81C.

[0114] If the interference determination unit 85 determines that there is a possibility of interference (YES in step S135C shown in FIG. 15), the work setting unit 81C outputs a command to suspend the work or issue an interference warning (step S150C shown in FIG. 13). Furthermore, the work setting unit 81C issues a command to re-determine the possibility of interference by re-measuring the 2D sonar 2 and 3D sonar 3 (steps S160 to S200 shown in FIG. 15).

[0115] According to the underwater work method and underwater work system of the third embodiment, as in the first embodiment, 2D images are captured by the 2D sonar 2 during operation of the work equipment 71 and displayed on the first monitor 6 (display device). This reduces the time required to visualize underwater work compared to capturing and displaying 3D images during operation. Furthermore, the 3D position information of the identification points a, b, c, and d during operation is estimated and updated using the 2D position information of the identification points a, b, c, and d in the 2D image, which is updated during operation starting from the 3D initial position calculated from the 3D image before operation begins. This allows the 3D position information of the identification points (monitoring targets) that cannot be measured from the 2D image to be determined. These factors enable the accurate determination of the position of the monitoring target and the suppression of a decrease in work speed in muddy water when performing underwater work in a visualized manner.

[0116] Furthermore, in the underwater work method according to this embodiment, if an obstacle that will hinder the work of the work device 71 is measured by measurements using the 2D sonar 2 and 3D sonar 3 (measuring devices) before work begins, the distance from the work device 71 to the obstacle is calculated using the 3D position information of the obstacle obtained from the 3D image before work begins and the information obtained by estimating the 3D position information of the identification points a, b, c, and d after work begins, and an interference judgment is performed to determine whether the calculated distance is below a preset threshold value.

[0117] Furthermore, the underwater work monitoring system 1C in the underwater work system of this embodiment is configured so that, when an obstacle that will hinder the work of the work equipment 71 is measured by the 2D sonar 2 and 3D sonar 3 (measuring devices) before work begins, the underwater work monitoring system 1C calculates the distance from the work equipment to the obstacle during work by using the 3D position information of the obstacle obtained from the 3D image recorded before work begins and the information obtained by estimating the 3D position information of the identification points a, b, c, and d after work begins, and performs an interference judgment to determine whether the calculated distance is below a preset threshold.

[0118] According to this method and configuration, the distance between the work device 71 and the obstacle during work is calculated using information obtained by estimating the three-dimensional position information of the identification points a, b, c, and d after work has started, thereby determining whether the work device 71 will come into contact with the obstacle, making it possible to avoid contact between the work device 71 and the obstacle.

[0119] Furthermore, in the underwater work method according to this embodiment, the measuring device is a 3D sonar 3 capable of measuring 3D position information by transmitting ultrasonic waves. If it is determined in the interference determination that the distance calculated as a result is equal to or less than the threshold, the work of the work device 71 is suspended, 2D images are recorded by measurement with the 2D sonar 2, and 3D images are re-recorded by measurement with the 3D sonar 3. Using the 3D position information of the obstacle obtained from the 3D image after the work is suspended and the 3D position information of the discrimination points a, b, c, and d, the distance from the work device 71 to the obstacle after the work is suspended is calculated, and an interference re-determination is performed to determine whether the calculated distance after the work is suspended is equal to or less than the threshold.

[0120] Furthermore, in the underwater work system according to this embodiment, the measuring device is a 3D sonar 3 capable of measuring 3D position information by transmitting ultrasonic waves. Furthermore, when the work of the work device 71 is interrupted because it is determined in the interference determination that the distance calculated as a result is equal to or less than a threshold, the underwater work monitoring system 1C is configured to calculate the distance from the work device 71 to the obstacle after the work is interrupted by using the 3D position information of the obstacle and the 3D position information of the discrimination points a, b, c, and d obtained from the 3D image re-recorded by measurement by the 3D sonar 3 after the work is interrupted, and to perform an interference re-determination to determine whether the calculated distance after the work is interrupted is equal to or less than a threshold.

[0121] According to this method and configuration, the distance between the work device 71 and the obstacle is calculated using the three-dimensional position information of the obstacle and identification points a, b, c, and d obtained from the three-dimensional image re-captured after work was interrupted, and interference between the work device 71 and the obstacle is re-determined, thereby making it possible to make an accurate interference determination and appropriately determine whether or not work can be resumed.

[0122] Furthermore, in the underwater work method according to this embodiment, if it is determined in the interference re-assessment that the distance after work interruption in the calculation result exceeds the threshold, the work of the work implement 71 is resumed. On the other hand, if it is determined in the interference re-assessment that the distance after work interruption in the calculation result is equal to or less than the threshold, the work plan for the work implement 71 is changed.

[0123] According to this method, whether or not work can be resumed is determined based on the results of interference re-determination using the 3D position information of obstacles and identification points a, b, c, and d obtained from the 3D images re-recorded after work was interrupted, enabling efficient operation through appropriate determinations.

[0124] [others] It should be noted that the present invention is not limited to the above-described first to third embodiments and their modifications, but includes various modifications. The above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. For example, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0125] For example, a configuration and method that combine the second and third embodiments described above are also possible. That is, a configuration and method that executes both the error determination by the error determination unit 84 of the work command device 80B shown in FIG. 14 in the second embodiment (step S135 shown in FIG. 13) and the interference determination by the interference determination unit 85 of the work command device 80C shown in FIG. 16 in the third embodiment (step S135C shown in FIG. 15) are also possible. In this case, the work command device has the functional units of the error determination unit 84 and the interference determination unit 85, and combines the flowchart shown in FIG. 13 and the flowchart shown in FIG. 15. However, it is preferable to execute the error determination in step S135 shown in FIG. 13 before the interference determination in step S135C shown in FIG. 15.

[0126] In the above-described embodiment, an example was shown in which two flat reflectors 715, 716 were installed on the work arm 711 to be monitored. However, three or more reflectors are also possible. The reflectors can also be shaped like a cross, a lattice, a sphere, or a hemisphere, in addition to flat plates. The reflectors can also be configured so that the internal medium is air, which increases the ratio of acoustic impedance to the material of the reflectors and thereby increases the reflectivity.

[0127] Furthermore, in the above-described embodiment, an example has been shown in which there is one 2D sonar 2 and one 3D sonar 3. However, it is also possible to configure the 2D sonar 2 and the 3D sonar 3 to include two or more units. [Explanation of symbols]

[0128] 1, 1A, 1B, 1C... Underwater work monitoring system, 2... 2D ultrasonic sonar, 2A... Dual-purpose sonar (ultrasonic sonar), 3... 3D ultrasonic sonar (measuring device), 5, 5C... Work monitoring device, 6... First monitor (display device), 80, 80B, 80C... Work command device, 71, 71B... Work device, 717... Attitude sensor

Claims

1. Before the work begins on the underwater work equipment, a two-dimensional image of the underwater work environment is recorded using two-dimensional ultrasonic sonar measurements. Before the work device starts working, a three-dimensional image of the underwater work environment is recorded by a measurement device capable of measuring three-dimensional position information; calculating a three-dimensional initial position of an identification point of the work device from a three-dimensional image before the start of work, with the installation position of the two-dimensional ultrasonic sonar as a reference point; Corresponding the three-dimensional initial position of the discrimination point obtained as a result of the calculation to two-dimensional position information of the discrimination point in the two-dimensional image before the start of the work; After the work of the work device starts, two-dimensional images are recorded and sequentially updated by continuous measurement of the two-dimensional ultrasonic sonar, and the sequentially updated two-dimensional images are displayed on a display device; Using a three-dimensional initial position associated with the two-dimensional position information of the identification point as a starting point, the three-dimensional position information of the identification point is estimated and sequentially updated by using the two-dimensional position information of the identification point in the two-dimensional image that is sequentially updated after the start of work. An underwater work method characterized by:

2. The underwater work method according to claim 1, The measuring device is a three-dimensional ultrasonic sonar capable of measuring three-dimensional position information by transmitting ultrasonic waves. An underwater work method characterized by:

3. 3. The underwater work method according to claim 2, An ultrasonic sonar having a cross array sensor in which a first one-dimensional array in which a plurality of transducers are arranged in one direction and a second one-dimensional array in which a plurality of transducers are arranged in one direction are arranged so as to be orthogonal to each other performs the measurement of the two-dimensional ultrasonic sonar and also performs the measurement of the three-dimensional ultrasonic sonar. An underwater work method characterized by:

4. The underwater work method according to claim 1, While the work device is working, a detected value of the attitude information of the work device is compared with information obtained by estimating and calculating the three-dimensional position information of the identification point of the work device, thereby performing an error determination to determine whether or not an error in the estimated value of the three-dimensional position information of the identification point is within an allowable range. An underwater work method characterized by:

5. 5. The underwater work method according to claim 4, the measuring device is a three-dimensional ultrasonic sonar capable of measuring three-dimensional position information by transmitting ultrasonic waves, When it is determined in the error determination that the error exceeds an allowable range, the operation of the work device is interrupted; A two-dimensional image is recorded by the measurement of the two-dimensional ultrasonic sonar, and a three-dimensional image is re-recorded by the measurement of the three-dimensional ultrasonic sonar; calculating, from the three-dimensional image after the work is interrupted, three-dimensional position information of the identification point after the work is interrupted, with the installation position of the two-dimensional ultrasonic sonar as a reference point; Correlating the two-dimensional position information of the discrimination point in the two-dimensional image after the work interruption with the three-dimensional position information of the discrimination point after the work interruption, which is the calculation result; The work of the work device is resumed. An underwater work method characterized by:

6. The underwater work method according to claim 1, When an obstacle that hinders the work of the work device is detected by the measurement of the two-dimensional ultrasonic sonar and the measurement device before the work starts, calculating a distance from the work device to the obstacle by using information obtained by estimating three-dimensional position information of the obstacle obtained from the three-dimensional image before the start of work and information obtained by estimating three-dimensional position information of the identification point after the start of work; Collision detection is performed to determine whether the calculated distance is equal to or less than a preset threshold. An underwater work method characterized by:

7. 7. The underwater work method according to claim 6, the measuring device is a three-dimensional ultrasonic sonar capable of measuring three-dimensional position information by transmitting ultrasonic waves, When it is determined in the interference determination that the distance calculated is equal to or less than the threshold value, the operation of the work device is suspended; A two-dimensional image is recorded by the measurement of the two-dimensional ultrasonic sonar, Re-recording a three-dimensional image using the three-dimensional ultrasonic sonar measurement; calculating a distance from the work device to the obstacle after the work is interrupted by using three-dimensional position information of the obstacle and three-dimensional position information of the identification point obtained from the three-dimensional image after the work is interrupted; The interference re-determination is performed to determine whether the calculated distance after the work interruption is equal to or less than the threshold value. An underwater work method characterized by:

8. 8. The underwater work method according to claim 7, When it is determined in the interference re-determination that the distance after the work interruption calculated as a result of the calculation exceeds the threshold, the work of the work device is resumed, When it is determined in the interference re-determination that the distance after the work interruption in the calculation result is equal to or less than the threshold value, the work plan of the work device is changed. An underwater work method characterized by:

9. A work device for performing work underwater; an underwater work monitoring system that visualizes and monitors the underwater work of the work device, The underwater work monitoring system includes: A two-dimensional ultrasonic sonar that can measure two-dimensional position information by transmitting ultrasonic waves; a measuring device capable of measuring three-dimensional position information; a display device that displays a two-dimensional image recorded by the measurement of the two-dimensional ultrasonic sonar; a work monitoring device that calculates three-dimensional position information of an identification point of the work device based on the measurement results of the two-dimensional ultrasonic sonar and the measurement results of the measuring device, The work monitoring device calculating a three-dimensional initial position of the identification point of the working device, using the installation position of the two-dimensional ultrasonic sonar as a reference point, from a three-dimensional image recorded by measurement by the measurement device before the working device starts working; before the start of work by the work device, two-dimensional position information of the discrimination point in the two-dimensional image recorded by measurement of the two-dimensional ultrasonic sonar is associated with a three-dimensional initial position of the discrimination point obtained as a result of the calculation; The apparatus is configured to estimate and sequentially update the three-dimensional position information of the identification point by using the two-dimensional position information of the identification point in the two-dimensional image that is sequentially updated by the measurement of the two-dimensional ultrasonic sonar after the work device starts, starting from a three-dimensional initial position that is associated with the two-dimensional position information of the identification point. An underwater work system characterized by:

10. 10. The underwater work system according to claim 9, The measuring device is a three-dimensional ultrasonic sonar capable of measuring three-dimensional position information by transmitting ultrasonic waves. An underwater work system characterized by:

11. The underwater work system according to claim 10, The ultrasonic sonar has a cross array sensor in which a one-dimensional array in which a plurality of transducers are arranged in one direction and another one-dimensional array in which a plurality of transducers are arranged in one direction are arranged orthogonally, and is configured to serve as both the two-dimensional ultrasonic sonar and the three-dimensional ultrasonic sonar. An underwater work system characterized by:

12. 10. The underwater work system according to claim 9, the working device has a posture sensor that detects posture information of the working device, The underwater work monitoring system is configured to perform an error judgment to determine whether an error in the estimated value of the three-dimensional position information of the identification point is within an allowable range by comparing the attitude information detected by the attitude sensor while the work device is working with information obtained by an estimation calculation of the three-dimensional position information of the identification point of the work device performed by the work monitoring device. An underwater work system characterized by:

13. The underwater work system according to claim 12, the measuring device is a three-dimensional ultrasonic sonar capable of measuring three-dimensional position information by transmitting ultrasonic waves, When the error determination determines that the error exceeds an allowable range and the work of the work device is interrupted, the work monitoring device calculating three-dimensional position information of the identification point after the interruption of work, using the installation position of the two-dimensional ultrasonic sonar as a reference point, from a three-dimensional image re-recorded by measurement of the three-dimensional ultrasonic sonar after the interruption of work by the work device; and a method for detecting the three-dimensional position of the identification point after the interruption of work, the three-dimensional position of the identification point after the interruption of work is associated with the two-dimensional position of the identification point in the two-dimensional image recorded by the measurement of the two-dimensional ultrasonic sonar after the interruption of work by the work device. An underwater work system characterized by:

14. 10. The underwater work system according to claim 9, When an obstacle that may hinder the operation of the work device is detected by the two-dimensional ultrasonic sonar and the measurement device before the start of work, the underwater work monitoring system: calculating a distance from the work device to the obstacle during work by using information obtained by estimating the three-dimensional position information of the obstacle obtained from the three-dimensional image recorded before work begins and the three-dimensional position information of the identification point after work begins; The apparatus is configured to perform interference determination by determining whether the distance calculated is equal to or less than a preset threshold value. An underwater work system characterized by:

15. 15. The underwater work system according to claim 14, the measuring device is a three-dimensional ultrasonic sonar capable of measuring three-dimensional position information by transmitting ultrasonic waves, When the distance calculated in the interference determination is determined to be equal to or less than the threshold value and the work of the work device is suspended, the underwater work monitoring system calculating a distance from the work device to the obstacle after the work is interrupted by using three-dimensional position information of the obstacle and three-dimensional position information of the identification point obtained from a three-dimensional image re-recorded by measurement with the three-dimensional ultrasonic sonar after the work is interrupted; The apparatus is configured to perform interference re-determination to determine whether the distance after the work interruption based on the calculation result is equal to or less than the threshold value. An underwater work system characterized by:

Images