Method and system for creating three-dimensional model of floating body

Abstract

To provide a method and system for creating a three-dimensional model of a floating body, capable of accurately creating a three-dimensional model of a preset model creation target area in a floating body that is floating in a body of water.SOLUTION: A method for creating a three-dimensional model of a floating body comprises: arranging targets 2 in three or more locations in a model creation target area 10s of a floating body 10; acquiring an original photographic image data group DG1 consisting of a plurality of original photographic image data D1 capturing at least a part of the model creation target area 10s using an imaging device 3 from above the floating body 10 floating in a body of water, and including the model creation target area 10s and the three or more targets 2; generating an extracted image data group DG2 consisting of a plurality of extracted image data D2 with the model creation target area 10s extracted in each original photographic image data D1; performing image processing to match the extracted image data D2 based on feature points or targets 2 present in the model creation target area 10s; and creating a three-dimensional model M.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a method and system for creating a three-dimensional model of a floating body, and more specifically to a method and system for creating a three-dimensional model of a floating body that can more accurately create a three-dimensional model of a predetermined model creation target area on a floating body floating in a water area. [Background technology]

[0002] Various methods have been proposed for creating a three-dimensional model of a subject from multiple pieces of photographed image data by photographing the subject from multiple directions and performing image processing based on the photographing positions (photographing directions) relative to the subject in each piece of photographed image data (see, for example, Patent Document 1). When creating a three-dimensional model (3D model) of the Earth's surface, as in the invention described in Patent Document 1, when photographing the Earth's surface with a photographing device mounted on an unmanned aerial vehicle, a GPS receiver is used to measure the absolute position coordinates of the unmanned aerial vehicle (photographing device), thereby easily identifying the photographing position of the photographing device relative to the Earth's surface when each piece of photographed image data was acquired. By using the absolute position coordinates of the photographing positions where each piece of photographed image data was acquired, image processing can be easily performed to match photographed image data that depict a common area.

[0003] On the other hand, when creating a 3D model of a floating object in water, such as a ship or caisson, the floating object is photographed from above using a camera while the object is moving and swaying due to the influence of water currents and waves. Because a floating object in water moves and sways, the relative position between the camera and the floating object changes from moment to moment, even when the camera is stationary in the air. Therefore, simply measuring the absolute position coordinates of the camera using a GPS receiver or other device cannot accurately determine the relative position of the camera to the floating object when each piece of photographed image data was acquired. Therefore, using methods similar to those used to create 3D models of stationary objects such as the Earth's surface, it is difficult to accurately match photographed image data that capture a common area, and it is therefore not possible to accurately create a 3D model of a floating object in water. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-214294 Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to provide a method and system for creating a three-dimensional model of a floating body that can create a three-dimensional model of a predetermined model creation target area of ​​a floating body floating in a water area with higher accuracy. [Means for solving the problem]

[0006] In order to achieve the above object, the present invention provides a method for creating a three-dimensional model of a floating body, which creates a three-dimensional model of a preset model creation target area of ​​a floating body floating in a water area, by placing targets in at least three or more locations in the model creation target area, and performing a photographing step of acquiring an original photographed image data group consisting of a plurality of original photographed image data images that capture at least a part of the model creation target area by using a photographing device from above the floating body floating in the water area, so that the original photographed image data group includes the entire model creation target area and at least three or more of the targets placed in the model creation target area, and in a subsequent extraction image processing step, By generating a plurality of extracted image data that extracts the range in which the model creation target area is captured in the original captured image data, a group of extracted image data is generated that captures the entire model creation target area and at least three or more of the targets placed in the model creation target area, and in the subsequent 3D model creation step, image processing is performed to match the extracted image data in the group of extracted image data that capture common ranges of the model creation target area based on feature points or the targets present in the model creation target area captured in the extracted image data, thereby creating a 3D model of the model creation target area from the group of extracted image data.

[0007] The system for creating a three-dimensional model of a floating body of the present invention is a system for creating a three-dimensional model of a floating body floating in a water area, which creates a three-dimensional model of a preset model creation target area on a floating body floating in a water area, and includes targets placed in three or more locations in the model creation target area, a photographing device that photographs the model creation target area from above the floating body floating in the water area, a moving means that moves the photographing device above the floating body, and a computing device to which original photographed image data acquired by the photographing device is input, and the photographing device and the moving means acquire an original photographed image data group consisting of a plurality of original photographed image data that capture at least a part of the model creation target area, and generate a three-dimensional model of the entire model creation target area and at least three or more of the targets placed in the model creation target area. The computing device is configured to input the group of original photographed image data generated by the input processing to the computing device, and the computing device generates a plurality of extracted image data by extracting the range in which the model creation target area is captured in each of the input original photographed image data, thereby generating a group of extracted image data that captures the entire model creation target area and at least three or more of the targets placed in the model creation target area, and performs image processing to match the extracted image data in the group of extracted image data that capture a common range of the model creation target area based on feature points or the targets that are present in the model creation target area captured in the extracted image data, thereby creating a three-dimensional model of the model creation target area from the group of extracted image data. [Effects of the Invention]

[0008] According to the present invention, an original image data group consisting of multiple original image data images capturing at least a portion of a model creation target area of ​​a floating body in a body of water is acquired by an imaging device from above the floating body, so that the original image data group includes the entire model creation target area and at least three or more targets placed in the model creation target area. Then, multiple extracted image data sets are generated by extracting the range of the model creation target area captured in each original image data set, to generate an extracted image data group capturing the entire model creation target area and at least three or more targets placed in the model creation target area. Each extracted image data set does not include any subject other than the model creation target area captured in the original image data that could become noise, and further, the extracted image data group captures three or more targets that indicate the reference position of a coordinate system based on the floating body. This enables accurate image processing to be performed to match extracted image data sets capturing a common range of the model creation target area in a coordinate system based on the floating body, thereby enabling accurate creation of a 3D model of the model creation target area of ​​a floating body floating in a body of water. [Brief explanation of the drawings]

[0009] [Figure 1] This is an explanatory diagram illustrating, in a plan view, a situation in which a photographing step is being performed on a floating work vessel using a photographing device mounted on a drone. [Figure 2] This is an explanatory diagram illustrating a side view of a situation in which a photographing step is being performed on a floating work vessel using a photographing device mounted on a drone. [Figure 3] This is an explanatory diagram illustrating, in plan view, the relative position coordinates of the camera mounted on a drone with respect to a work boat (coordinate reference position) and the shooting range of the camera when the camera is used to photograph a floating work boat. [Figure 4] 3 is an explanatory diagram illustrating original photographed image data acquired by the photographing device of FIG. 1 and FIG. 2. [Figure 5]5 is an explanatory diagram illustrating extracted image data obtained by extracting a range in which a model creation target area of ​​a floating body (work boat) is captured from the original photographed image data of FIG. 4. FIG. [Figure 6] FIG. 10 is an explanatory diagram illustrating an example of an extracted image data group made up of a plurality of extracted image data. [Figure 7] FIG. 1 is an explanatory diagram illustrating an example of three-dimensional model image data of a floating body (work boat). [Figure 8] FIG. 10 is an explanatory diagram illustrating, in plan view, a situation in which a photographing step is being performed on a carrier ship, which is a floating body, using a photographing device installed on a crane. [Figure 9] FIG. 10 is an explanatory diagram illustrating, in cross section, a situation in which a photographing step is being performed on a floating body, a carrier ship, using a photographing device installed on a crane. [Figure 10] 10 is an explanatory diagram illustrating original photographed image data acquired by the photographing device of FIGS. 8 and 9. FIG. [Figure 11] 11 is an explanatory diagram illustrating extracted image data obtained by extracting a range in which a model creation target area of ​​a floating body (carrier ship) is captured in the original photographed image data of FIG. 10. FIG. [Figure 12] FIG. 1 is an explanatory diagram illustrating an example of three-dimensional model image data of a floating body (carrier ship). [Figure 13] This is an explanatory diagram illustrating a plan view of a situation in which a photographing step is being performed on a floating caisson using a photographing device mounted on a drone. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, a method and system for creating a three-dimensional model of a floating body will be described based on the embodiment shown in the drawings.

[0011] The embodiment of the method and system 1 for creating a 3D model of a floating body exemplified in FIGS. 1 and 2 is a method and system for creating a 3D model M of a preset model creation target area 10s of a floating body 10 floating in a body of water as exemplified in FIG. 7. In this embodiment, the floating body 10 is a work vessel 10A, and the model creation target area 10s is the entire area of ​​the floating body 10 (work vessel 10A). Examples of the floating body 10 include various ships, caissons, and wind power generation equipment floating in a body of water (such as the sea, lakes, or rivers). The model creation target area 10s can be set arbitrarily; for example, a partial area of ​​the floating body 10 can be set as the model creation target area 10s.

[0012] As illustrated in Figures 1 and 2, this creation system 1 includes targets 2 placed at three or more locations in a model creation target area 10s, a photographing device 3 that photographs the model creation target area 10s from above a floating body 10 floating in a water area, a moving means 4 that moves the photographing device 3 above the floating body 10, and a computing device 5 to which the original photographed image data D1 acquired by the photographing device 3 is input.

[0013] As shown in FIG. 1, the target 2 is a mark that indicates the reference position of a coordinate system based on the floating body 10. For example, a sign with a marker figure or lettering on a plate-like or sheet-like member is used as the target 2. The sign is placed on the floating body 10 as the target 2. The target 2 (sign) has a figure obtained by dividing a rectangle into four crosses, and is designed so that adjacent sections of the figure are colored in different colors. The shape, design, color, etc. of the target 2 are not limited to this embodiment, and various other configurations are possible.

[0014] The size of the target 2 is determined appropriately depending on the size of the float 10, etc., and the actual length and width dimensions of the target 2 in a plan view are, for example, approximately 200 mm to 500 mm. For example, a known anti-aircraft marker can be used as the target 2. The target 2 can be made from a plate-like or sheet-like member with a magnet or adhesive attached, and can be configured to be attachable to and detachable from the float 10. For example, a figure or letter that serves as a marker can be painted on the float 10 as the target 2. Furthermore, for example, if the float 10 has a distinctive part, component, pattern, printing, etc. that serves as a marker, the marker that is originally on the float 10 can also be used as the target 2.

[0015] In this embodiment, targets 2 are placed at three locations on the deck of the floating body 10 (work vessel 10A). The installation positions of the targets 2 relative to the model creation target area 10s are not particularly limited, but each target 2 is placed in a position where it can be photographed from above the floating body 10 (model creation target area 10s). For example, targets 2 can be placed at four or more locations relative to the model creation target area 10s. In this embodiment, targets 2 with the same design are used, but targets 2 with different, distinguishable designs can also be used.

[0016] For example, a digital camera is used as the imaging device 3. For example, a drone 4A or a crane is used as the transportation means 4. In this embodiment, a drone 4A is used as the transportation means 4, and the imaging device 3 is installed below the drone 4A. The imaging direction of the imaging device 3 is set to face downward.

[0017] A computer or the like is used as the arithmetic device 5. The location of the arithmetic device 5 is not particularly limited. Although a notebook-type personal computer having a monitor 6 is used as the arithmetic device 5 in this example, a desktop-type or tablet-type personal computer can also be used as the arithmetic device 5, for example.

[0018] Using this creation system 1, a three-dimensional model M of a model creation target area 10s on a floating body 10 floating in a body of water is created by performing a photographing step, an extracted image processing step, and a three-dimensional model creation step. In this embodiment, an example is shown in which a three-dimensional model M of a floating body 10 (model creation target area 10s) on which a GNSS receiver 11 (a receiver of a global navigation satellite system) is placed is created.

[0019] As illustrated in FIGS. 1 to 3, in the photographing step, targets 2 are placed in at least three or more locations in a model creation target area 10s. Then, an original photographed image data group DG1 consisting of multiple original photographed image data D1 capturing at least a portion of the model creation target area 10s is acquired by an imaging device 3 from above a floating body 10 floating in a body of water. The original photographed image data group DG1 includes the entire model creation target area 10s and at least three or more targets 2 placed in the model creation target area 10s. In other words, in the photographing step, multiple original photographed image data D1 capturing at least a portion of the model creation target area 10s are acquired under different imaging conditions of the imaging device 3 relative to the floating body 10 (model creation target area 10s), thereby acquiring an original photographed image data group DG1 in which the model creation target area 10s and at least three or more targets 2 placed in the model creation target area 10s are captured.

[0020] The number of original photographed image data D1 constituting the original photographed image data group DG1 can be determined appropriately depending on the size of the model creation target area 10s and the shooting range (angle of view) of the shooting device 3. It is sufficient that at least three or more targets 2 are captured in the original photographed image data group DG1, and it is not necessary that the target 2 is captured in each original photographed image data D1. In other words, the original photographed image data group DG1 may include not only original photographed image data D1 that captures the target 2, but also original photographed image data D1 that does not capture the target 2.

[0021] When creating a three-dimensional model M of a model creation target area 10s in which a GNSS receiver 11 is placed, the GNSS receiver 11 is placed in the model creation target area 10s, and an original photographed image data group DG1 is obtained in which the model creation target area 10s, the target 2, and the GNSS receiver 11 are photographed in the photographing step.

[0022] In Figure 3, the shooting range of each piece of original photographed image data D1 acquired by the photographing device 3 is indicated by a dashed-dotted rectangular frame. As illustrated in Figure 3, in the photographing step, the original photographed image data D1 is acquired so that the shooting ranges of the photographing device 3 at adjacent shooting locations have an overlapping range. The overlapping range of adjacent original photographed image data D1 can be determined appropriately depending on the shooting range of the photographing device 3, the size of the floating body 10, etc. In Figure 3, for ease of understanding, the overlapping range between the shooting range of the photographing device 3 and adjacent pieces of original photographed image data D1 is depicted as relatively narrow, but in reality, it is preferable to set the overlapping range between the shooting range of the photographing device 3 and adjacent pieces of original photographed image data D1 to be relatively wide.

[0023] The original photographic image data group DG1 (plurality of original photographic image data D1) acquired by the photographing device 3 is stored in a memory built into the photographing device 3, and after the photographing step is completed, the original photographic image data group DG1 stored in the memory of the photographing device 3 is input to the arithmetic device 5. For example, the photographing device 3 and the arithmetic device 5 may be connected to be able to communicate with each other, and the original photographic image data D1 acquired by the photographing device 3 may be sequentially transmitted to the arithmetic device 5, and the arithmetic device 5 may store the input plural original photographic image data D1 as the original photographic image data group DG1.

[0024] 4 shows an example of original photographic image data D1, one of the original photographic image data group DG1, displayed on the monitor 6 of the calculation device 5. The original photographic image data D1 in the present invention is photographic image data acquired by the photographing device 3, but is referred to as original photographic image data D1 because it is photographic image data before image processing that serves as the basis for extracted image data D2 generated in an extracted image processing step described later.

[0025] 1 and 2, in this embodiment, a drone 4A, which is a means of transportation 4, is flown above a floating body 10 (a model creation target area 10s), and an imaging device 3 attached to the drone 4A captures images of the model creation target area 10s, three targets 2 located in the model creation target area 10s, and a GNSS receiver 11. In FIG. 1, the flight path of the drone 4A is indicated by a solid arrow, and each imaging position coordinate APn (AP1 to AP21) where the imaging device 3 acquires the original captured image data D1 is indicated by a filled-in circle. Each imaging position coordinate APn shown in FIG. 1 indicates an absolute position coordinate.

[0026] While moving the drone 4A and the photographing device 3 above the floating body 10, original photographed image data D1 is acquired, which captures at least a portion of the model creation target area 10s at each photographing position coordinate APn. Then, by acquiring the original photographed image data D1 at multiple photographing position coordinates APn, an original photographed image data group DG1 is acquired, in which the entire model creation target area 10s, the three targets 2, and the GNSS receiver 11 are photographed. The movement means 4 (drone 4A) and the photographing device 3 can be remotely controlled by an operator, or can be automatically controlled based on a pre-entered computer program, etc. When the drone 4A is automatically controlled, the movement path of the drone 4A can be set based on absolute position coordinates.

[0027] The photographing height of the photographing device 3 relative to the floating body 10 (the model creation target area 10s) can be determined appropriately depending on the size of the model creation target area 10s and the angle of view of the photographing device 3. It is preferable to maintain a roughly constant photographing height (flight height of the drone 4A) of the photographing device 3 at each photographing position coordinate APn. However, because the floating body 10 sways up and down due to the influence of waves and the drone 4A may also sway up and down due to the influence of wind, the photographing height of the photographing device 3 relative to the floating body 10 (the model creation target area 10s) does not need to be maintained strictly constant. While this embodiment illustrates a case in which the original photographed image data D1 is acquired at a total of 21 photographing position coordinates AP1 to AP21, the number of photographing locations (photographing position coordinates APn) from which the original photographed image data D1 is acquired, the movement path of the photographing device 3 by the transportation means 4, the locations of the photographing locations, and the number of original photographed image data D1 acquired at each photographing location can be determined appropriately depending on the size of the model creation target area 10s and the photographing range (angle of view) of the photographing device 3. For example, it is possible to obtain a plurality of original photographed image data D1 by changing the photographing height, photographing direction, photographing range, etc. at the same photographing location.

[0028] In this embodiment, the photographing step is performed by one photographing device 3, but for example, the photographing step can also be performed by a plurality of photographing devices 3. In that case, each photographing device 3 that has finished photographing is collected, and the original photographed image data D1 stored in the memory of each photographing device 3 is input to the arithmetic device 5. The arithmetic device 5 stores the multiple original photographed image data D1 input from the multiple photographing devices 3 together as an original photographed image data group DG1.

[0029] In this way, the photographing device 3 and the moving means 4 that constitute the creation system 1 are configured to acquire an original photographed image data group DG1 consisting of multiple original photographed image data D1 that capture at least a portion of the model creation target area 10s, and input the original photographed image data group DG1 that includes the entire model creation target area 10s and at least three or more targets 2 placed in the model creation target area 10s to the calculation device 5.

[0030] The coordinate reference position CP shown in Figures 1 to 3 is a reference point in a three-dimensional coordinate system based on the floating body 10. In this embodiment, the coordinate reference position CP is set at the front (bow) of the floating body 10 (work boat 10A). In the figures, the coordinate reference position CP is indicated by a black square. The coordinate reference position CP can be set at any position on the floating body 10. In Figure 3, the relative position coordinates FPn (FP1 to FP21) of the photographing device 3 mounted on the drone 4A with respect to the floating body 10 (coordinate reference position CP) at the time when each original photographed image data D1 was acquired by the photographing device 3 mounted on the drone 4A are indicated by black triangles.

[0031] When photographing a floating body 10 floating on the water, even if the floating body 10 is fixed in place, the floating body 10 sways in all directions and up and down due to the influence of water currents and waves, and the relative positional relationship between the floating body 10 and the photographing device 3 (drone 4A) changes from moment to moment during the photographing step. Therefore, as can be seen by comparing the photographing position coordinates APn shown in absolute position coordinates in Figure 1 with the relative position coordinates FPn of the photographing device 3 with respect to the coordinate reference position CP in Figure 3, a discrepancy occurs between the photographing position coordinates APn, which show the photographing position by the photographing device 3 in absolute position coordinates, and the relative position coordinates FPn, which show the photographing position of the photographing device 3 relative to the floating body 10.

[0032] For example, when creating a 3D model of a stationary subject on land, it is relatively easy to determine the relative position coordinates of the camera device 3 with respect to the subject when the subject was photographed, and therefore it is relatively easy to perform image processing to match multiple photographed image data of the subject and create a 3D model of the subject.In contrast, when creating a 3D model M of a floating body 10 floating in water, it is difficult to accurately determine the relative position coordinates FPn of the camera device 3 with respect to the floating body 10 at the time each original photographed image data D1 was acquired.

[0033] Furthermore, when creating a 3D model of a stationary subject, the background of the subject captured in the photographed image data of the subject can be kept constant, making it easy to match multiple photographed image data sets together, but as illustrated in Figure 4, when photographing a floating body 10 floating in a body of water, the original photographed image data D1 acquired by the photographing device 3 not only captures part of the model creation target area 10s of the floating body 10 and the target 2, but also captures waves W and splashes on the water surface around the floating body 10, and the state of the waves W and splashes differs in each original photographed image data D1. If there are buoys B (e.g., buoys), mooring ropes, ships other than the floating body 10, land (e.g., quays), etc. floating in the water around the floating body 10, these may also be captured in the original photographed image data D1.

[0034] Therefore, when creating a 3D model M of a floating body 10 floating in a water area, there are many elements that become noise in the image processing for matching multiple original photographed image data D1 obtained by photographing the model creation target area 10s, and simply matching multiple original photographed image data D1 does not allow for the creation of a highly accurate 3D model M. Therefore, in order to accurately generate a 3D model M of a floating body 10 floating in a water area, this creation method and creation system 1 perform an extracted image processing step and a 3D model creation step, which will be described below.

[0035] 5 and 6, in the extracted image processing step performed after the photographing step, the arithmetic device 5 generates a plurality of extracted image data D2 by extracting the range in each original photographed image data D1 in which the model creation target area 10s is captured, thereby generating an extracted image data group DG2 in which the entire model creation target area 10s and at least three targets 2 arranged in the model creation target area 10s are captured. More specifically, in the extracted image processing step, as illustrated in FIG. 5, the arithmetic device 5 performs image processing on the original photographed image data D1 to remove the range in the original photographed image data D1 in which the model creation target area 10s is not captured (removal target range R), thereby generating extracted image data D2 in which the range in which the model creation target area 10s is captured is extracted. Then, by performing the above-described image processing on each original photographed image data D1, an extracted image data group DG2 in which the entire model creation target area 10s and at least three targets 2 arranged in the model creation target area 10s are captured is generated, as illustrated in FIG. As in this embodiment, when the GNSS receiver 11 is placed in the model creation target area 10s, the GNSS receiver 11 also appears in the extracted image data group DG2.

[0036] 5 and 6, the removal target area R removed from the original captured image data D1 is indicated by diagonal lines. In Fig. 5, a boundary line L between the area containing the extracted model creation target area 10s and the removal target area R is indicated by a two-dot chain line. Image processing for generating extracted image data D2 from the original captured image data D1 may involve, for example, image processing for trimming (cutting out) and extracting the area containing the model creation target area 10s in the original captured image data D1, or image processing for masking (filling in) the area in the original captured image data D1 where the model creation target area 10s is not included (removal target area R) and extracting the area containing the model creation target area 10s.

[0037] The task of setting the boundary line L of the range in which the model creation target area 10s to be extracted from the original photographed image data D1 is captured may be configured to be performed automatically by a computer program, or may be configured to be performed manually by an operator operating input means such as a mouse or keyboard connected to the computing device 5 while visually checking the original photographed image data D1 displayed on the monitor 6.

[0038] When the task of setting the boundary line L is performed automatically by a computer program, a computer program that automatically identifies the area in the original photographed image data D1 in which the model creation target area 10s is captured is stored in the arithmetic device 5, and when the original photographed image data D1 is input to the arithmetic device 5, the extracted image data D2 is automatically generated by the computer program. The computer program that identifies the area in which the model creation target area 10s is captured can be created by having artificial intelligence (AI) learn the information using a large amount of image data of the floating body 10 as training data. Examples of the machine learning techniques include deep learning, neural networks, regression, clustering, and pattern matching.

[0039] It is preferable that the range (boundary line L) containing the model creation target area 10s extracted from the original photographed image data D1 in the extracted image processing step be set so as not to include, as much as possible, areas other than the model creation target area 10s (the water area outside the floating body 10), but it is difficult to ensure that no areas other than the model creation target area 10s are included in the extracted image data D2. Therefore, in the extracted image processing step, it is preferable to extract the range containing the model creation target area 10s from the original photographed image data D1 so that the area other than the model creation target area 10s included in the extracted image data D2 is, for example, 10% or less, more preferably 5% or less, and even more preferably 3% or less of the area of ​​the range of the model creation target area 10s included in the extracted image data D2.

[0040] As illustrated in FIGS. 6 and 7 , in the 3D model creation step performed after the extracted image processing step, image processing is performed to match extracted image data D2 that depict common areas of the model creation target area 10s in the extracted image data group DG2 based on feature points or targets 2 that exist in the model creation target area 10s and that are captured in the extracted image data D2, thereby creating a 3D model M of the model creation target area 10s from the extracted image data group DG2. Feature points that exist on the float 10 include characteristic parts, parts, accessories, mounted objects, patterns, printing, scratches, unevenness, etc. that exist on the float 10. By using three or more targets 2 captured in the extracted image data group DG2 as reference positions in a coordinate system based on the float 10 (coordinate reference position CP), it is possible to perform image processing to match extracted image data D2 that depict common areas of the model creation target area 10s in a coordinate system based on the float 10 (coordinate reference position CP).

[0041] The three-dimensional model M of the model creation target area 10s of the floating body 10 also includes the target 2. If a GNSS receiver 11 is placed in the model creation target area 10s, the three-dimensional model M also includes the GNSS receiver 11. Figure 7 shows an example of the created three-dimensional model M of the model creation target area 10s of the floating body 10 (work boat 10A) displayed on the monitor 6 as three-dimensional model image data D3.

[0042] Image processing for generating a 3D model M of the model creation target area 10s from the extracted image data group DG2 (plurality of extracted image data D2) can be performed using known SfM (Structure from Motion) software. In the image processing (SfM processing) described above, a point cloud is generated by multi-viewpoint image measurement based on the extracted image data group DG2, thereby generating a 3D model M of the model creation target area 10s.

[0043] An image processing computer program that automatically detects targets 2 appearing in the extracted image data group DG2 (extracted image data D2) and matches the extracted image data D2 using the detected targets 2 can be created, for example, by machine learning using image data of the targets 2 as training data. Examples of the aforementioned machine learning techniques include deep learning, neural networks, regression, clustering, and pattern matching.

[0044] As described above, in the creation system 1 and creation method, an original photographed image data group DG1 consisting of a plurality of original photographed image data D1 capturing at least a portion of a model creation target area 10s of the floating body 10 is acquired by an imaging device 3 from above the floating body 10 floating in a body of water. The original photographed image data group DG1 includes the entire model creation target area 10s and at least three or more targets 2 arranged in the model creation target area 10s. Then, a plurality of extracted image data D2 are generated by extracting the range in each of the original photographed image data D1 that captures the model creation target area 10s, thereby generating an extracted image data group DG2 that captures the entire model creation target area 10s and at least three or more targets 2 arranged in the model creation target area 10s.

[0045] Each extracted image data D2 does not include any subject other than the model creation target area 10s that was captured in the original captured image data D1 that could become noise, and furthermore, the extracted image data group DG2 contains three or more targets 2 that indicate reference positions in a coordinate system based on the floating body 10 (coordinate reference position CP). Therefore, by using the three or more targets 2 captured in the extracted image data group DG2, it is possible to accurately perform image processing to match extracted image data D2 that capture common areas of the model creation target area 10s in the extracted image data group DG2 in the coordinate system based on the floating body 10. Therefore, it is possible to accurately create a three-dimensional model M of the model creation target area 10s in the floating body 10 floating in the water.

[0046] That is, since the floating body 10 floating in the water body is swaying, simply measuring the absolute position coordinates (photographing position coordinates APn) of the photographing device 3 by installing a GPS receiver on a mobile means 4 such as a drone 4A and positioning the absolute position coordinates of the photographing device 3 (photographing position coordinates APn) as in the conventional technology cannot accurately identify the photographing position (relative position coordinates FPn) of the photographing device 3 relative to the floating body 10 when each original photographed image data D1 was acquired. Therefore, simply measuring the absolute position coordinates (photographing position coordinates APn) of the photographing device 3 when each original photographed image data D1 was acquired as in the conventional technology cannot accurately create a 3D model M of the floating body 10 floating in the water body.

[0047] In contrast, the creation system 1 and creation method remove noise-causing objects other than the model creation target area 10s that appear in the original photographed image data group DG1, thereby minimizing noise factors when matching extracted image data D2. Furthermore, by using three or more targets 2 arranged in the model creation target area 10s as reference positions in a coordinate system based on the floating body 10 (coordinate reference position CP), it is possible to accurately create a 3D model M of the model creation target area 10s on the floating body 10, even when the floating body 10 floating in the water is moving or swaying. Furthermore, by using extracted image data D2 from which noise-causing elements have been removed from the original photographed image data D1, it is possible to significantly reduce the amount of calculation required for image processing to match extracted image data D2 in the 3D model creation step, making it possible to quickly create the 3D model M.

[0048] In a conventional method for creating a 3D model of a stationary object on land, the 3D model is created by using information in an absolute coordinate system, such as absolute position coordinate information of a camera obtained using a Global Navigation Satellite System (GNSS) or shooting angle information of a camera obtained using an angle sensor, etc. In contrast, a floating body 10 floating in a body of water sways, and the absolute coordinates of the floating body 10 are not constant. Therefore, the system 1 and method for creating a 3D model of the floating body 10 do not use information measured in a coordinate system (absolute coordinate system) different from the coordinate system based on the floating body 10 as described above, thereby improving the accuracy of creating the 3D model M of the floating body 10 floating in a body of water.

[0049] This creation system 1 and creation method are not limited to cases where the floating body 10 is fixed in a fixed location, but can also be used when the floating body 10 is being intentionally moved, by using the moving means 4 to move the camera 3 to follow the floating body 10 while taking pictures, thereby making it possible to create a 3D model M of the model creation target area 10s on the floating body 10. Therefore, this is very useful to those skilled in the art.

[0050] The imaging step can be performed very efficiently by configuring the imaging device 3 mounted on the drone 4A to acquire the original image data D1. In this creation system 1 and creation method, the swinging of the drone 4A in the sky does not affect the creation accuracy of the 3D model M of the floating body 10, so the use of the drone 4A makes it possible to create the 3D model M efficiently and with high accuracy.

[0051] As in this embodiment, a GNSS receiver 11 is placed in a model creation target area 10s, and the original captured image data group DG1 includes the GNSS receiver 11 in the photographing step. Then, in the 3D model creation step, a 3D model M of the model creation target area 10s in which the GNSS receiver 11 is placed is created. This accurately displays the relative positional relationship between the floating body 10 (model creation target area 10s) and the GNSS receiver 11 in the 3D model M, making it possible to accurately determine the installation position of the GNSS receiver 11 relative to the floating body 10. This improves the positioning accuracy when determining the position coordinates of the floating body 10 using the GNSS receiver 11 placed on the floating body 10. Conventionally, because the floating body 10 sways, it has been difficult to accurately measure the installation position of the GNSS receiver 11 relative to the floating body 10 when the GNSS receiver 11 is placed on the floating body 10. However, by employing this creation system 1 and creation method, the installation position of the GNSS receiver 11 relative to the floating body 10 can be easily and accurately determined.

[0052] As with the GNSS receiving device 11, for example, if a measuring device (sensor) is placed in the model creation target area 10s and a three-dimensional model M of the model creation target area 10s in which the measuring device is placed is created, the relative positional relationship between the floating body 10 (model creation target area 10s) and the measuring device is accurately displayed on the three-dimensional model M, making it possible to accurately determine the installation position of the measuring device relative to the floating body 10.

[0053] In another embodiment illustrated in Figures 8 to 12, the floating body 10 is a carrier ship 10B carrying exposed loads 12, and the model creation target area 10s is a part of the floating body 10. In this embodiment, the carrier ship 10B is a soil carrier, and the loads 12 are earth and sand. In this embodiment, the model creation target area 10s is the hold in which the loads 12 are loaded, located on the bow side (left side of the paper) of the dotted line shown in Figure 8, and the part of the hull on the stern side (right side of the paper) of the dotted line is not set as the model creation target area 10s.

[0054] 8 and 9, in this embodiment, the photographing step is performed on a carrier ship 10B that is anchored near land G (quay). A crane 4B placed on land G is used as the moving means 4, and the photographing device 3 is moved above the floating body 10 (carrier ship 10B).

[0055] The crane 4B is equipped with a tiltable boom 4c, and a hoisting device 4e is attached to the lower end of a hoisting wire 4d suspended from the tip of the boom 4c. In this embodiment, a camera 3 is attached to the hoisting device 4e, and the camera 3 is moved by the rotation of the crane 4B and the movement of the boom 4c. When using the crane 4B, for example, the camera 3 can be installed at the tip of the boom 4c of the crane 4B, or the camera 3 can be attached to the lower end of the hoisting wire 4d. For example, the hoisting device 4e can be attached to the lower end of the hoisting wire 4d, and the camera 3 can be attached to the load held by the hoisting device 4e.

[0056] As shown in Fig. 8, in this embodiment, targets 2 with different designs that are identifiable are used, and the targets 2 are arranged in six locations on the floating body 10 (carrier ship 10B). In the photographing step, as in the above-described embodiment, an original photographed image data group DG1 consisting of multiple original photographed image data D1 that capture at least a portion of the model creation target area 10s is obtained by the photographing device 3 from above the floating body 10 (carrier ship 10B) floating in the water area. The original photographed image data group DG1 then includes the entire model creation target area 10s and at least three or more (six in this embodiment) targets 2 arranged in the model creation target area 10s.

[0057] 10, in this embodiment, original photographed image data D1 is acquired at each photographing position, with the angle of view including a part of the model creation target area 10s and three or more targets 2. It is not necessary for all targets 2 placed on the floating body 10 to be captured in each original photographed image data D1.

[0058] 11, in the extraction image processing step, similar to the embodiment described above, a plurality of extracted image data D2 are generated by extracting the range in each of the original captured image data D1 in which the model creation target area 10s is captured, thereby generating an extracted image data group DG2 in which the entire model creation target area 10s and at least three (six in this embodiment) targets 2 arranged in the model creation target area 10s are captured. In Fig. 11, the removal target area R removed from the original captured image data D1 is indicated by diagonal lines, and the boundary line L between the range in which the part of the model creation target area 10s to be extracted is captured and the removal target area R is indicated by a two-dot chain line.

[0059] As in this embodiment, when targets 2 are placed in three or more locations in the model creation target area 10s and the photographing step is performed so that three or more of the targets 2 placed in the model creation target area 10s are included in the angle of view of each original photographed image data D1, the three-dimensional model creation step can also be configured as described below.

[0060] As illustrated in Figure 12, in this embodiment, as in the previously described embodiment, in the three-dimensional model creation step, image processing is performed to match extracted image data D2 that depict common areas of the model creation target area 10s in the extracted image data group DG2 based on feature points or targets 2 that exist in the model creation target area 10s depicted in the extracted image data D2, thereby creating a three-dimensional model M of the model creation target area 10s from the extracted image data group DG2.

[0061] However, in this embodiment, when image processing is performed to match the extracted image data D2 in the three-dimensional model creation step, the relative position coordinates FPn of the camera 3 with respect to the coordinate reference position CP preset for the floating body 10 at the time when the original photographed image data D1 for each extracted image data D2 was acquired are identified based on the three or more targets 2 captured in each extracted image data D2 and the feature points present in the model creation target area 10s captured in each extracted image data D2. The arrangement and size of each target 2 with respect to the floating body 10 (model creation target area 10s) can be known in advance. Therefore, based on the relative positional relationships between three or more targets 2 captured in the extracted image data D2, the display size of each target 2, the relative size differences between the targets 2, and the feature points present in the model creation target area 10s captured in the extracted image data D2, it is possible to determine the relative position coordinates FPn (Xn, Yn, Zn) of the photographing device 3 with respect to the coordinate reference position CP (X0, Y0, Z0) at the time the original photographed image data D1 used to generate the extracted image data D2 was acquired.

[0062] In the image processing for matching the extracted image data D2, by using the relative position coordinates FPn of the photographing device 3 for each of the extracted image data D2 described above, it is possible to perform the image processing for matching the extracted image data D2 with higher accuracy. Figure 12 shows an example of a state in which the created 3D model M of the model creation target area 10s of the floating body 10 (carrier ship 10B) is displayed on the monitor 6 as 3D model image data D3.

[0063] As in this embodiment, targets 2 are placed in three or more locations in the model creation target area 10s, and in the photographing step, original photographed image data D1 is obtained at each photographing position, with a portion of the floating body 10 and three or more targets 2 captured in the field of view.In the three-dimensional model creation step, the above-mentioned relative position coordinate identification work is performed.This makes it possible to perform image processing to match the extracted image data D2 with each other with higher accuracy, which is more advantageous for creating a three-dimensional model M of the floating body 10 with high accuracy.

[0064] As in this embodiment, by placing targets 2 each having a different identifiable design in the model creation target area 10s, it becomes easier to identify the relative position coordinates FPn of the photographing device 3 for each extracted image data D2 in the above-mentioned relative position coordinate identification task, and to perform image processing to match the extracted image data D2 with each other.

[0065] When creating a 3D model M of a floating body 10 located close to land G, the photographing step can be performed very efficiently by configuring the original photographed image data D1 to be acquired by a photographing device 3 installed on a crane 4B. In this creation system 1 and creation method, the accuracy of creating the 3D model M is not affected by the swinging of the photographing device 3 suspended by the crane 4B, so by using the crane 4B, the 3D model M can be created efficiently with high accuracy.

[0066] If the floating body 10 is a carrier ship 10B carrying exposed loads 12, creating a three-dimensional model M of the carrier ship 10B makes it possible to accurately determine the loading position and amount of the loads 12 loaded on the carrier ship 10B, as shown in Fig. 12. In particular, if the carrier ship 10B is a soil carrier and the loads 12 are soil and sand, it has conventionally been difficult to accurately determine the amount (volume) of soil loaded on the soil carrier, but by creating a three-dimensional model M of the soil carrier (hold) according to the present invention, it becomes possible to easily and accurately determine the amount (volume) of soil and sand loaded on the soil carrier.

[0067] For example, even if the carrier 10B is a bulk carrier (dry bulk carrier) and the goods 12 are grain, coal, iron ore, etc., creating a 3D model M of the bulk carrier makes it possible to easily and accurately grasp the amount (volume) of the grain, coal, iron ore, etc. loaded on the bulk carrier. Also, even if the carrier 10B is a container ship and the goods 12 are containers, creating a 3D model M of the container ship makes it possible to easily and accurately grasp the loading position and number of containers relative to the container ship. This is very useful for those skilled in the art.

[0068] As illustrated in Figure 13, this creation system 1 and creation method can be used not only when the floating body 10 is a ship such as a work boat 10A or a carrier ship 10B, but also when the floating body 10 is a floating body 10 other than a ship, such as a caisson 10C. For example, if the floating body 10 is a caisson 10C, it is possible to create a 3D model M of the caisson 10C by placing targets 2 in three or more locations on the top surface (top cover) of the caisson 10C and similarly performing the photographing step, the extracted image processing step, and the 3D model creation step.

[0069] 13, by placing the GNSS receiver 11 on a caisson 10C and creating a three-dimensional model M of the caisson 10C on which the GNSS receiver 11 is placed, the relative positional relationship between the caisson 10C and the GNSS receiver 11 is accurately displayed on the three-dimensional model M, making it possible to accurately grasp the installation position of the GNSS receiver 11 relative to the caisson 10C. This improves the positioning accuracy when measuring the position coordinates of the caisson 10C using the GNSS receiver 11 placed on the caisson 10C.

[0070] In the above embodiment, the entire area of ​​the work ship 10A, which is the floating body 10, is set as the model creation target area 10s, but for example, a portion of the area of ​​the work ship 10A can also be set as the model creation target area 10s. Also, the embodiment has been described in which a portion of the carrier ship 10B, which is the floating body 10, is set as the model creation target area 10s, but for example, the entire area of ​​the carrier ship 10B can also be set as the model creation target area 10s. [Explanation of symbols]

[0071] 1. Crafting System 2. Target 3. Imaging equipment 4. Transportation 4A Drone 4B Crane 4c Boom 4d Hanging Wire 4e Hanging equipment 5 Computing device 6 monitors 10 Floating Body 10A Workboat 10B Carrier 10C Caisson 10s model creation target area 11 GNSS receiver 12 Transported goods D1 Original image data DG1 Original image data set D2 Extracted image data DG2 Extracted image data set D3 3D model image data R Removal range M 3D model L (trimming) border APn Shooting position coordinates CP coordinate reference position FPn relative position coordinates W wave B Buoy G land

Claims

1. A method for creating a three-dimensional model of a floating body in a water body, which creates a three-dimensional model of a predetermined model creation target area of ​​the floating body, a photographing step of placing targets in at least three locations in the model creation target area, and acquiring an original photographed image data group consisting of a plurality of original photographed image data images that capture at least a portion of the model creation target area by a photographing device from above the floating body floating in the water area, so that the original photographed image data group includes the entire model creation target area and the at least three or more targets placed in the model creation target area; In the subsequent extracted image processing step, a plurality of extracted image data are generated by extracting the range in which the model creation target area is captured in each of the original photographed image data, thereby generating a group of extracted image data that captures the entire model creation target area and at least three or more of the targets placed in the model creation target area; In the subsequent three-dimensional model creation step, image processing is performed to match the extracted image data that depict a common range of the model creation target area in the extracted image data group based on feature points or the targets that exist in the model creation target area depicted in the extracted image data, thereby creating a three-dimensional model of the model creation target area from the extracted image data group.

2. In the photographing step, each of the original photographed image data includes three or more of the targets arranged in the model creation target area within an angle of view, A method for creating a three-dimensional model of a floating body as described in claim 1, wherein, in the three-dimensional model creation step, when performing the image processing to match the extracted image data, the relative position coordinates of the photographing device with respect to a coordinate reference position previously set for the floating body at the time the original photographed image data of each of the extracted image data was acquired are identified based on the three or more targets and feature points captured in each of the extracted image data, and the image processing is performed using the identified relative position coordinates of each of the extracted image data.

3. The method for creating a three-dimensional model of a floating body according to claim 1 or 2, wherein the original photographed image data is acquired by the photographing device mounted on a drone.

4. A method for creating a three-dimensional model of a floating body as described in claim 1 or 2, wherein a GNSS receiving device is placed in the model creation target area, and in the photographing step, the original photographed image data group includes the GNSS receiving device, and in the three-dimensional model creation step, a three-dimensional model of the model creation target area in which the GNSS receiving device is placed is created.

5. 3. The method for creating a three-dimensional model of a floating body according to claim 1, wherein the floating body is a carrier ship carrying an exposed load.

6. A system for creating a three-dimensional model of a floating body that creates a three-dimensional model of a predetermined model creation target area of ​​a floating body floating in a water area, The system comprises targets arranged at three or more locations in the model creation target area, a photographing device that photographs the model creation target area from above the floating body floating in a water area, a moving means that moves the photographing device above the floating body, and a computing device to which original photographed image data acquired by the photographing device is input, the photographing device and the moving means are configured to acquire an original photographed image data group consisting of a plurality of original photographed image data images each showing at least a part of the model creation target area, and input the original photographed image data group including the entire model creation target area and at least three or more of the targets arranged in the model creation target area to the arithmetic device; The calculation device generates a plurality of extracted image data by extracting the range in which the model creation target area is captured from each of the input original captured image data, generates a group of extracted image data that captures the entire model creation target area and at least three or more of the targets placed in the model creation target area, and performs image processing to match the extracted image data in the group of extracted image data that capture a common range of the model creation target area based on feature points or the targets present in the model creation target area captured in the extracted image data, thereby creating a three-dimensional model of the model creation target area from the group of extracted image data.

Images