Information processing program, information processing method, and information processing device

Abstract

A role of instructing a ship docking at a harbor is to give instruction regarding direction and movement for enabling the ship to dock while visually confirming the distance and angle to a quay. However, due to various kinds of unevenness at quays, including bumpers, and the inability of range finders to discern that unevenness, points to be measured cannot be identified, and it is difficult to accurately ascertain the distance and angle between a ship and a quay. According to the present invention, a computer acquires an image including a docking area photographed from a ship, uses a segmentation AI to identify the docking area in the acquired image, measures the distance to the docking area by using a distance measuring device, and calculates at least one of the relative distance or relative angle between the docking area and the ship from the measurement result from the distance measuring device.

Description

Information processing program, information processing method, and information processing device

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device.

[0002] In major ports around the world, including Japan, many ships, especially large ships, that dock within the port are docked by pushers. In this case, the person in charge gives instructions to the pusher, such as pushing and stopping, during the docking operation. More specifically, for example, the person in charge visually checks the distance and angle to the quay, and instructs the pusher on the direction and actions to dock the large ship.

[0003] As such, instructions to the pusher boat are given visually, and are therefore difficult to give, as they largely depend on the intuition and experience of the person in charge of giving instructions. For this reason, it is thought that instructions to the pusher boat can be assisted by calculating the distance and angle between the ship and the quay using a rangefinder such as LiDAR (Light Detection and Ranging) or LRF (Laser Range Finder) and presenting this information to the person in charge of giving instructions.

[0004] JP 2020-059403 A JP 2005-180949 A JP 2021-136466 A

[0005] However, for example, a quay has various irregularities, including fenders, and the distance measuring device cannot distinguish between these, so it is not possible to identify the docking area to be measured, making it difficult to accurately grasp the distance and angle between the ship and the quay. The docking area may be a quay, a floating pier connected to the quay, etc.

[0006] In one aspect, an object of the present invention is to provide an information processing program, an information processing method, and an information processing device that can more accurately grasp the distance and angle between a ship and a quay.

[0007] In one aspect, the information processing program causes a computer to perform a process of acquiring an image including a docking area photographed from a ship, identifying the docking area in the acquired image using segmentation AI (Artificial Intelligence), measuring the distance to the docking area using a ranging device, and calculating at least one of the relative distance between the docking area and the ship or the relative angle, which is the azimuth angle difference between the quay and the parallel body of the ship, from the measurement results obtained by the ranging device.

[0008] On the one hand, the relative distance and relative angle between the ship and the quay can be grasped more accurately.

[0009] FIG. 1 is a diagram for explaining an example of a method for measuring a relative distance and a relative angle to a quay. FIG. 2 is a diagram for explaining an example of a camera image of a quay taken from a ship. FIG. 3 is a diagram for explaining an example of the configuration of an information processing system 1 according to the first embodiment. FIG. 4 is a diagram for explaining an example of the configuration of an information processing device 10 according to the first embodiment. FIG. 5 is a diagram for explaining an example of quay surface extraction by segmentation according to the first embodiment. FIG. 6 is a diagram for explaining an example of distance measurement using a rangefinder according to the first embodiment. FIG. 7 is a diagram for explaining an example of registration processing according to the first embodiment. FIG. 8 is a diagram for explaining an example of relative distance and relative angle calculation according to the first embodiment. FIG. 9 is a diagram for explaining a distance measurement range when a quaying area and a ship approach each other. FIG. 10 is a diagram for explaining calculation of a relative distance. FIG. 11 is a flowchart for explaining the flow of a relative distance and relative angle calculation process according to the first embodiment. FIG. 12 is a diagram for explaining rolling correction. FIG. 13 is a diagram for explaining calculation of a relative distance by an information processing device according to a third example. FIG. 14 is a diagram for explaining an example of the hardware configuration of the information processing device 10.

[0010] Examples of the information processing program, information processing method, and information processing device according to the present embodiment will be described in detail below with reference to the accompanying drawings. Note that the present embodiment is not limited to these examples. Furthermore, the examples can be combined as appropriate within a consistent range.

[0011] First Embodiment First, a method for measuring the relative distance and relative angle from a large ship to a quay will be described. FIG. 1 is a diagram for explaining an example of a method for measuring the relative distance and relative angle to a quay. As shown in FIG. 1 , for example, the distances from two locations, one in front of and one behind the large ship, to the quay can be measured using range finders such as LiDAR and LRF installed on the ship, thereby determining the relative distance and relative angle from the large ship to the quay. Note that while FIG. 1 illustrates a large ship as an example, this embodiment is not limited to large ships and may include other small ships. Furthermore, in the description of each embodiment, the term "ship" may refer to both large ships and other small ships.

[0012] An example of a method for measuring the relative distance and relative angle from a ship to a quay has been explained using Figure 1, but there are problems with actual measurements. Figure 2 shows an example of a camera image of a quay taken from on board a ship. As shown in Figure 2, various objects such as fenders are present around an actual quay. Therefore, in measurements using a range finder such as LiDAR, measuring the quay surface, which is the vertical surface of the quay that forms the docking area, leads to a more accurate understanding of the relative distance and relative angle between the ship and the quay. Here, the relative angle is the difference in azimuth angle between the quay and the parallel body of the ship, and for example, the angle θ in Figure 1 ０ Note that the azimuth angle of the parallel body is usually the same as the bow direction of the ship, but in this embodiment the azimuth angle of the parallel body may be different from the bow direction of the ship.

[0013] However, measurements using distance meters such as LiDAR can measure the relative distance to each point in space using a set of data of vertices in a three-dimensional space called point cloud data or point cloud, but cannot determine what each point in space is. Therefore, it is not possible to determine whether the relative distance of each point in the point cloud data is, for example, the relative distance to the quay surface or the relative distance to an object other than the quay surface, such as a fender. Therefore, one of the objectives of this embodiment is to more accurately grasp the relative distance and relative angle between the ship and the quay.

[0014] Next, an information processing system 1 according to this embodiment will be described. FIG. 3 is a diagram showing an example of the configuration of the information processing system 1 according to the first embodiment. The information processing system 1 is a system that calculates the relative distance and relative angle between a ship and its docking area. The docking area for the ship may be a quay as shown in FIGS. 1 and 2, or a floating pier floating on the sea surface and connected to the quay as shown in FIGS. 6 and 8, which will be described later. As shown in FIG. 3, the information processing system 1 is a system in which an information processing device 10, a camera device 100, and distance measuring devices 200-1 and 200-2 (hereinafter collectively referred to as "distance measuring devices 200") are connected to each other via a network 50 so as to be able to communicate with each other.

[0015] The network 50 may be any of a variety of communication networks, such as an intranet, whether wired or wireless. The network 50 may not be a single network, but may instead be configured such that an intranet and the Internet are connected via a network device such as a gateway or other device (not shown).

[0016] The information processing device 10 may be, for example, a server computer installed on a ship or the like, or an information processing device such as a desktop PC (Personal Computer) or a notebook PC. While the information processing device 10 is shown as a single computer in FIG. 3 , it may also be a distributed computing system made up of multiple computers. Furthermore, the information processing device 10 may also be a cloud computing device managed by a service provider that provides cloud computing services.

[0017] The information processing device 10 acquires an image including a docking area captured by the camera device 100 from a ship, for example, and identifies the docking area from the acquired image using segmentation AI. Note that the segmentation AI may be, for example, semantic segmentation or instance segmentation, and extracts the docking area from the image using a machine learning model. The machine learning model is, for example, a machine learning model generated by machine learning using images including a docking area tagged with information as training data.

[0018] In addition, the information processing device 10 measures the distance to the docking area, for example, using a ranging device 200, and calculates the relative distance and relative angle between the docking area extracted from the image captured by the camera device 100 and the ship from the measurement results by the ranging device 200.

[0019] The camera device 100 is, for example, a digital camera that is installed on a ship and captures images of the ship's surroundings, such as a quay, from the ship. Images captured by the camera device 100 are transmitted to the information processing device 10.

[0020] The ranging device 200 is a range finder such as LiDAR or LRF that is installed at multiple locations, for example, at the front and rear of the ship, and measures the distance to objects around the ship. Point cloud data, which is the measurement results measured by the ranging device 200, is transmitted to the information processing device 10. Note that the distance to an object measured by the ranging device 200 is, strictly speaking, the distance between the ranging device 200 and the object, but as described with reference to FIG. 1 , in this embodiment, the ranging device 200 is installed on the ship, and therefore, the distance is processed as the distance between the ship and the object.

[0021] (Functional configuration of information processing device 10) Next, the functional configuration of the information processing device 10 that executes this embodiment will be described. Fig. 4 is a diagram showing an example of the configuration of the information processing device 10 according to the first embodiment. As shown in Fig. 4, the information processing device 10 has a communication unit 20, a storage unit 30, and a control unit 40.

[0022] The communication unit 20 is a processing unit that controls communication with other devices such as the camera device 100, and is, for example, a communication interface such as a network interface card.

[0023] The storage unit 30 has a function of storing various data and programs executed by the control unit 40, and is realized by a storage device such as a memory or a hard disk. The storage unit 30 also stores various information such as image information 31, measurement information 32, and model information 33.

[0024] The image information 31 stores, for example, images captured by the camera device 100. Furthermore, the image information 31 may store, for example, information regarding a docking area identified from the images captured by the camera device 100 using segmentation AI.

[0025] The measurement information 32 stores, for example, point cloud data that is the measurement results measured by the distance measuring device 200. Furthermore, the measurement information 32 may store, for example, calculation result data such as the relative distance and relative angle between the docking area and the ship calculated from the point cloud data.

[0026] The model information 33 stores, for example, information about a machine learning model for extracting a docking area from an image captured by the camera device 100, and model parameters of the machine learning model. The machine learning model is generated by machine learning using training data in which the image captured by the camera device 100 is used as an explanatory variable and the docking area in the image is used as a target variable.

[0027] The above information stored in the storage unit 30 is merely an example, and the storage unit 30 can store various information other than the above information.

[0028] The control unit 40 is a processing unit, such as a processor, that controls the entire information processing device 10. The control unit 40 includes an acquisition unit 41, an identification unit 42, a measurement unit 43, and a calculation unit 44. Each processing unit is an example of an electronic circuit included in the processor or an example of a process executed by the processor.

[0029] The acquisition unit 41 acquires, for example, an image including a docking area photographed by the camera device 100 from the ship from the image information 31. The acquisition unit 41 also acquires position information of the docking area in each of the acquired image and the measurement results measured by the ranging device 200, for example, by performing calibration on the acquired image and the measurement results measured by the ranging device 200. Here, the measurement results are three-dimensional information (two-dimensional coordinates + distance) assuming LiDAR. The calibration process is a process of acquiring position information in each of the image photographed by the camera device 100 and the measurement results measured by the ranging device 200 in order to align the positions in different coordinate systems in the image photographed by the camera device 100 and the measurement results measured by the ranging device 200.

[0030] The identification unit 42 identifies the docking area in the image acquired by the acquisition unit 41, for example, by using segmentation AI. Note that the identification of the docking area may use a machine learning model generated by machine learning using training data in which the image captured by the camera device 100 is used as an explanatory variable and the docking area in the image is used as a target variable. Information on the machine learning model is stored, for example, in the model information 33. Note that the identification of the docking area by the identification unit 42 may be performed based on markers installed on the quay surface, which is the docking area.

[0031] Furthermore, the process of identifying the docking area may include a process of identifying a quay wall surface or a floating pier as the docking area. FIG. 5 is a diagram illustrating an example of quay wall surface extraction by segmentation according to the first embodiment. FIG. 5 illustrates an example of identifying and extracting a quay wall surface as the docking area. The image shown on the left side of FIG. 5 is, for example, an image of a quay wall photographed from a ship as shown in FIG. 2 , i.e., an image of the quay wall and its surroundings photographed by the camera device 100. As shown on the right side of FIG. 5 , by using segmentation AI, at least the quay wall surface is extracted from the image. Note that, for example, as shown in FIG. 5 , fenders, holes, and other installations may exist on the quay wall surface, and trucks and port facilities may exist around the quay wall. However, these objects are distinguished from the quay wall surface, and only the quay wall surface is extracted from the image.

[0032] Furthermore, the identification unit 42 identifies and aligns the position of the docking area in the image captured by the camera device 100 and the measurement results measured by the distance measuring device 200 by performing registration of the position information in the image and the measurement results. The registration process will be described later, but the identification unit 42 aligns the positions of different coordinate systems in the image captured by the camera device 100 and the measurement results measured by the distance measuring device 200 by the registration process.

[0033] The measurement unit 43 measures the distance to the docking area, for example, using the distance measuring device 200. As described above, although the distance measuring device 200 can measure the distance to each point in space, it cannot determine what each point in space is. Therefore, strictly speaking, the measurement unit 43 measures the distance to objects around the ship, including the docking area. Then, the measurement unit 43 extracts the distance to the docking area in the measurement results using the image aligned by the identification unit 42 and the position of the docking area in the measurement results.

[0034] The calculation unit 44 calculates the relative distance and relative angle between the docking area and the ship, for example, from the measurement results obtained by the distance measuring device 200. The calculated relative distance and relative angle may be used, for example, to provide maneuvering instructions to the pusher.

[0035] The calculation of the relative distance and the relative angle by the calculation unit 44 will be described in more detail. FIG. 6 is a diagram showing an example of distance measurement using a range finder according to the first embodiment. As shown in FIG. 6, the distance measuring device 200 installed on the vessel is usually located higher than the quay, and therefore the ray from the distance measuring device 200 (denoted as "LiDAR irradiation" in FIG. 6) is oblique irradiation. Therefore, it is necessary to calculate the horizontal relative distance, denoted as "measured distance" in FIG. 6. Therefore, the calculation unit 44 calculates the relative distance and the relative angle from the distance to the quay, which is the measurement result, assuming that the quay is vertical, for example. The calculation process of the relative distance and the relative angle will be described later.

[0036] (Function Details) Next, the registration process will be described in more detail. Fig. 7 is a diagram showing an example of the registration process according to the first embodiment. Fig. 7 shows an example in which a point P, which is a docking area, is photographed by the camera device 100 and measured by the distance measuring device 200. In Fig. 7, the position on the image photographed by the camera device 100 that corresponds to the point P is P Ｃ , the position on the point cloud data measured by the distance measuring device 200 is P Ｌ , respectively.

[0037] The distance from the ship to the docking area, point P, is calculated by Ｌ However, it is not possible to determine what each point in space is based on point cloud data alone, so P Ｌ Therefore, the information processing device 10 determines the positions in the coordinate systems different from those of the image captured by the camera device 100 and the point cloud data, and in the example of FIG. Ｃ and P Ｌ By combining this, P Ｃ From P Ｌ More specifically, if the point P is an object that is represented as a single point on a map, such as an edge of a quay corner or a mooring post, the world coordinates (X Ｗ , Y Ｗ , Z Ｗ ) can be specified in advance from map data, etc. Also, the position P Ｃ coordinates (X Ｃ , Y Ｃ ) can also be identified by identifying the docking area using segmentation AI. Ｗ , Y Ｗ , Z Ｗ ) and the position of the docking area on the image P Ｃ coordinates (X Ｃ , Y Ｃ ) but can be aligned.

[0038] Then, the world coordinates of the point P and the position P on the point cloud data measured by the distance measuring device 200 are calculated. ＬThe coordinates of point P can be converted into the world coordinates (X Ｗ , Y Ｗ , Z Ｗ ) to P Ｌ coordinates (X Ｌ , Y Ｌ ) can be calculated.

[0039]

[0040] In formula (1), r １１ ~r ３３ is the amount of rotation of the origin of the coordinate system by the distance measuring device 200 from the origin of the coordinate system by the camera device 100, and t １ ~t ３ is the amount of movement of the origin of the coordinate system by the distance measuring device 200 from the origin of the coordinate system by the camera device 100. １１ ~r ３３ and t １ ~t ３ is assumed to be specified in advance based on the installation positions of the two devices, and is stored in advance in the storage unit 30, etc. By the registration process using the formula (1), the position P Ｌ , the point P which is a world coordinate and an absolute coordinate, and further, the position P on the image captured by the camera device 100. Ｃ will be aligned with each other.

[0041] Next, the calculation of the relative distance and relative angle between the ship and the docking area will be described in more detail. Fig. 8 is a diagram showing an example of the calculation of the relative distance and relative angle according to the first embodiment. Fig. 8 shows an example in which the docking area is the vertical surface of a quay, but the docking area may also be the vertical surface of a floating pier, a fender installed on the vertical surface of a quay, etc.

[0042] First, as shown in Fig. 8, segmentation AI is used to detect the vertical surface of the quay wall, which is the docking area, from an actual image captured by the camera device 100. In addition, before, after, or in parallel with the image capture of the docking area by the camera device 100, the distance between the ship and objects around the docking area, including the docking area, is measured by the distance measuring device 200. Note that the distance measurement by the distance measuring device 200 may be performed by distance measuring devices 200 set at at least two locations, one in front of and one in back of the parallel body, which is the hull, as shown in Fig. 8.

[0043] Here, it is possible that the docking area and the ship are approaching each other, and the horizontal distance from the docking area to the camera device 100 is small or negative. Fig. 9 is a diagram showing the distance measurement range when the docking area and the ship are approaching each other. Fig. 9 also shows the angle of view 101 captured by the distance measurement device 200. In Fig. 9, the distance measurement devices 200 are arranged side by side, overlapping each other, in the depth direction of the page (the fore-and-aft direction of the ship). Within the angle of view 101, part of the quay wall 111 and part of the sea 112 are captured.

[0044] The distance measuring device 200 cuts out a part of the range 102 of the angle of view 101 and uses it for distance measurement. As shown in FIG. 9 , if the horizontal distance from the docking area to the camera device 100 is small or negative, it is conceivable that the range 102 used for distance measurement will hardly or not include an image of the vertical surface of the quay wall 111. In this case, it becomes difficult for the information processing device 10 to detect the vertical surface. Therefore, the information processing device 10 uses segmentation AI to detect the horizontal surface of the quay wall 111 and detect the quay wall line. Then, the distance measuring device 200 may measure the distance by considering the position of the detected quay wall line as the position of the vertical surface.

[0045] 7, the image captured by the camera device 100 is aligned with the measurement results measured by the distance measuring device 200 through calibration and registration processes. This allows data on the docking area to be extracted from the point cloud data, and the distance between the ship and the docking area to be obtained.

[0046] Next, as shown in FIG. 8 , the information processing device 10 calculates the inclination between the vertical plane of the quay and the parallel body of the ship, i.e., the relative angle between the ship and the docking area, by geometric calculation. The information processing device 10 can use, for example, inverse trigonometric functions as the geometric calculation. The information processing device 10 then calculates the relative distance between the ship and the docking area, for example, from the relative angle calculated by performing the geometric calculation and the distance between the ranging device 200 and the docking area acquired from the point cloud data. FIG. 10 is a diagram for explaining the calculation of relative distance. When the angle at which the ranging device 200 looks down on the quay, i.e., the incident angle from the ranging device 200 to the distance measurement target with respect to the horizontal plane, is Φ, and the distance between the ranging device 200 and the distance measurement target point in the docking area is L#1, the information processing device 10 calculates the relative distance as L#2 = L#1 × cos(Φ).

[0047] (Processing Flow) Next, a flow of a process for calculating the relative distance and relative angle between the ship and the docking area, which is executed by the information processing device 10, will be described. Fig. 11 is a flowchart showing the flow of the relative distance / relative angle calculation process according to the first embodiment. The relative distance / relative angle calculation process shown in Fig. 11 may be executed, for example, when the information processing device 10 detects that the ship has entered within a predetermined range from the docking area.

[0048] 11 , the information processing device 10 acquires an image of the docking area captured by the camera device 100 and stored in the image information 31 (step S101). The camera device 100 may capture an image in response to an instruction sent from the information processing device 10 to the camera device 100, or may capture an image at any time by the camera device 100. The image captured by the camera device 100 is sent to the information processing device 10 and stored in the image information 31.

[0049] Next, the information processing device 10 identifies a docking area from the image acquired in step S101, for example, using segmentation AI (step S102). The identified docking area may be, for example, a vertical surface of a quay or floating pier in the docking area, or a fender installed on the vertical surface of the quay, and is determined in advance for each docking area. Then, a machine learning model to be used by the segmentation AI is generated in advance by machine learning using images including the docking area to which information tags have been added as training data.

[0050] Next, the information processing device 10 measures the distance to the docking area using distance measuring devices 200 installed at multiple locations, for example, at the front and rear of the ship (step S103). As described above, the measurement using the distance measuring devices 200 cannot determine the identity of each point in space, so strictly speaking, the information processing device 10 measures the distance to objects around the ship, including the docking area. Point cloud data, which is the measurement result measured by the distance measuring devices 200, is transmitted to the information processing device 10 and stored in the measurement information 32. The measurement in step S103 may be performed before steps S101 and S102, or in parallel with steps S101 and S102.

[0051] Next, the information processing device 10 aligns the image acquired in step S101 with the measurement results measured in step S103 (step S104) by performing calibration and registration processes, for example, as described with reference to Fig. 7. This allows the information processing device 10 to extract data on the docking area from the measurement results measured in step S103, and to obtain the distance between the ship and the docking area.

[0052] Next, the information processing device 10 performs, for example, a geometric calculation process to calculate the relative distance between the ship and the docking area, and also calculates the relative angle as described with reference to Fig. 8 (step S105). After execution of step S105, the relative distance / relative angle calculation process shown in Fig. 11 ends.

[0053] (Effect) As described above, the information processing device 10 acquires an image including a docking area photographed from a ship, identifies the docking area from the acquired image using segmentation AI, measures the distance to the docking area using a ranging device, and calculates the relative distance and relative angle between the docking area and the ship from the measurement results from the ranging device.

[0054] In this way, by using segmentation AI, the information processing device 10 can identify the quay surface from the captured image, and calculate the relative distance and relative angle to the quay surface from the distance to the quay surface measured using a rangefinder, thereby more accurately grasping the distance and angle between the ship and the quay.

[0055] Furthermore, the process of specifying a docking area executed by the information processing device 10 includes a process of specifying a quay surface or a floating pier as the docking area.

[0056] This allows the information processing device 10 to more accurately grasp not only the quay as the docking area, but also the distance and angle between the ship and the floating pier.

[0057] In addition, the process of calculating the relative distance and relative angle executed by the information processing device 10 includes a process of calculating the relative distance and relative angle from the distance to the docking area, which is the measurement result, assuming that the docking area is vertical.

[0058] This allows the information processing device 10 to more accurately grasp the distance and angle between the ship and the quay.

[0059] In addition, the information processing device 10 performs calibration on the image and the measurement results to obtain position information of the docking area in each of the image and the measurement results, and performs registration on the position information in each of the image and the measurement results to identify and align the position of the docking area in the image and the measurement results.

[0060] This allows the information processing device 10 to more accurately grasp the distance and angle between the ship and the quay.

[0061] Second Embodiment Next, a second embodiment will be described. An information processing device 10 according to this embodiment is also represented by the block diagram of Fig. 4. In the following description, description of the operation of each unit similar to that of the first embodiment will be omitted.

[0062] A ship equipped with a camera device 100 and a distance measuring device 200 undergoes rolling, pitching, yawing, and heaving at sea. Rolling is rotation around the longitudinal axis of the ship. Pitching is rotation around the lateral axis of the ship. Yawing is rotation around the vertical axis. Heaving is vertical displacement of the ship. The angle of view of the camera device 100 fixed to the hull is affected by rolling, pitching, yawing, and heaving. Therefore, the information processing device 10 according to this embodiment performs distance measurement by correcting the angle of view in consideration of the effects of rolling, pitching, yawing, and heaving.

[0063] The ship according to this embodiment has rotation measuring instruments such as inclinometers that measure rolling, pitching, and yawing. The ship also has draft measuring instruments such as draft gauges that measure heaving. Furthermore, the accuracy of heaving can be improved by correcting with tide information, which is sea level height.

[0064] The calculation unit 44 calculates the relative angle with respect to the docking area, for example, assuming that the docking area is vertical. Next, the calculation unit 44 obtains the values ​​of rolling, pitching, and yawing from the rotation amount measuring instrument. The calculation unit 44 also obtains the value of heaving from the draft measuring instrument.

[0065] The calculation unit 44 corrects the angle of incidence from the distance measuring device 200 using information on rolling, pitching, yawing, and heaving. Then, the calculation unit 44 calculates the relative distance using the distance to the docking area, which is the measurement result, and the corrected relative angle and relative height.

[0066] FIG. 12 is a diagram for explaining rolling correction. Here, the rolling state of the ship is shown in accordance with the distance measuring device 200. Here, the quay line is the target point for distance measurement. Here, Ld is the distance detected by the distance measuring device 200. Furthermore, θd is the inclination angle of the quay line from the distance measuring device 200 calculated from the angle of view of the camera device 100. Furthermore, L10 is the horizontal relative distance between the distance measuring device 200 and the quay line.

[0067] State 301 shows a state in which the ship is not rolling. In this case, the inclination angle of the quay line from the distance measuring device 200 calculated from the angle of view of the camera device 100 matches the horizontal direction of the distance measuring device 200 and the angle θ connecting the distance measuring device 200 and the quay line. Therefore, the calculation unit 44 calculates the relative distance L10 as Ld × cos(θ).

[0068] In contrast, in state 302, the ship is rolling by an angle θs. In this case, the sum of the angle θd of the distance measuring device 200 and the rolling angle θs matches θ, which is the angle between the horizontal direction of the distance measuring device 200 and the quay line. In other words, θ = θd + θs. Therefore, the calculation unit 44 calculates the relative distance L10 = Ld × cos(θ) = Ld × cos(θd + θs).

[0069] Although the easy-to-understand rolling correction has been described as an example here, the calculation unit 44 can also calculate other elements using geometric correction in a similar manner. The calculation unit 44 can also geometrically calculate the position of the ship's side, which is the subject of the relative distance between the position where the distance measuring device 200 is installed and the quay.

[0070] (Effect) In this way, by calculating the relative distance by correcting the angle of incidence from the distance measuring device 200 and the mounting height of the distance measuring device 200 based on rolling, pitching, yawing, and heaving, it is possible to improve the accuracy of the calculated relative distance.

[0071] Third Embodiment Next, a third embodiment will be described. An information processing device 10 according to this embodiment is also represented by the block diagram of Fig. 4. In the following description, description of the operation of each unit similar to that of the first embodiment will be omitted.

[0072] In the first embodiment, when the identification unit 42 identifies the docking area from the image acquired by the acquisition unit 41 using segmentation AI, it distinguishes fenders, holes, and other installations on the quay surface from the quay surface and extracts the quay surface from the image. In this case, if a fender is present at the position on the quay closest to the distance measurement origin, which serves as the ship's reference for relative distance, that position is excluded from the survey target. Therefore, the length from the ship to the closest position is not used as the relative distance, which may result in a decrease in accuracy.

[0073] Therefore, when the position of the quay where the fender is provided is the closest, the information processing device 10 according to this embodiment calculates the relative distance to the position of the fender using the relative distance to the position of the other quay, and sets this as the relative distance between the ship and the quay surface. The calculation process of the relative distance to the position of the fender by the information processing device 10 will be described below. Figure 13 is a diagram for explaining the calculation of the relative distance by the information processing device according to the third example.

[0074] The calculation unit 44 obtains the distance to the docking area, which is the measurement result by the distance measuring device 200. In this case, the calculation unit 44 obtains the distances to multiple points on the docking surface. Then, using the image of the docking surface obtained by the camera device 100 and the distance to the docking area, which is the measurement result by the distance measuring device 200, the calculation unit 44 determines whether or not the distance to the position of the nearest quay wall surface is included. Hereinafter, the position of the nearest quay wall surface will be referred to as the "nearest point." If the distance to the nearest point is included, the calculation unit 44 uses that distance to calculate the relative distance.

[0075] On the other hand, if the distance to the nearest point is not included in the acquired distance because the fender 211 is located at the closest position, the calculation unit 44 uses the distance to an appropriate position 201 to calculate a relative distance L11 to the position 201. The calculation unit 44 also uses the distance to another appropriate position 202 to calculate a relative distance L12 to the position 202.

[0076] Next, the calculation unit 44 calculates the angle θ of the ship facing the quay using the relative distances L11 and L12. １１ and θ １２Next, the calculation unit 44 calculates the relative distances L11 and L12 and the angle θ １１ and θ １２ Based on this, a geometric calculation process is performed to calculate the relative distance L1 from the distance measurement origin 210 to the nearest point 203.

[0077] Here, the thickness of the fender 211 is predetermined according to specifications. Therefore, the calculation unit 44 stores in advance a length L3 indicating the thickness of the fender 211. The calculation unit 44 subtracts the length L3, which is the thickness of the fender 211, from the calculated relative distance L1 to calculate the relative distance L2 from the vessel to the tip of the fender 211.

[0078] Conversely, if the relative distance L2 from the measurement origin 210 to the tip of the fender 211 can be obtained by another method, the calculation unit 44 can also determine the deterioration of the fender 211 by the following method. For example, the calculation unit 44 adds the obtained relative distance L2 and the length L3, which is the specified thickness of the fender 211, and compares the result with the relative distances L11 and L12 and the relative distance L1 to the nearest point 203, which is calculated based on the angle of the quay with respect to the ship. Then, if the sum of the relative distance L2 and the length L3 is shorter than the relative distance L1, the calculation unit 44 can determine that the fender 211 has deteriorated by that amount.

[0079] (System) The information, including the processing procedures, control procedures, specific names, various data, and parameters shown in the above documents and drawings, may be changed as desired unless otherwise specified. Furthermore, the specific examples, distributions, and numerical values ​​described in the embodiments are merely examples and may be changed as desired.

[0080] Furthermore, the specific form of distribution or integration of the components of each device is not limited to that shown in the drawings. That is, all or some of the components may be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions of each device may be realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0081] (Hardware) Fig. 14 is a diagram illustrating an example of the hardware configuration of the information processing device 10. As shown in Fig. 14, the information processing device 10 has a communication interface 10a, a hard disk drive (HDD) 10b, a memory 10c, and a processor 10d. The components shown in Fig. 14 are connected to each other via a bus or the like.

[0082] The communication interface 10a is a network interface card or the like, and communicates with other information processing devices. The HDD 10b stores programs and data for operating the functions shown in FIG.

[0083] The processor 10d is a hardware circuit that operates a process that executes each function described in FIG. 4 and other figures by reading a program that executes the same processing as each processing unit shown in FIG. 4 from the HDD 10b or the like and expanding the program into the memory 10c. That is, this process executes the same functions as each processing unit of the information processing device 10. Specifically, the processor 10d reads a program having the same functions as the acquisition unit 41, the identification unit 42, and the like from the HDD 10b or the like. Then, the processor 10d executes a process that executes the same processing as the acquisition unit 41, the identification unit 42, and the like.

[0084] In this way, the information processing device 10 operates as an information processing device that executes operation control processing by reading and executing a program that executes processing similar to that of each processing unit shown in Figure 4. The information processing device 10 can also realize functions similar to those of the above-mentioned embodiment by reading a program from a recording medium using a medium reading device and executing the read program. Note that the program in these other embodiments is not limited to being executed by the information processing device 10. For example, this embodiment may also be applied to cases where another computer executes a program or where the information processing device 10 and another computer cooperate to execute a program.

[0085] 4 can be distributed via a network such as the Internet. This program can be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and can be read out from the recording medium and executed by a computer.

[0086] The following additional notes are provided regarding the embodiments including the above examples.

[0087] (Supplementary Note 1) An information processing program that causes a computer to execute the following processes: acquire an image including a docking area photographed from a ship; identify the docking area from the acquired image using segmentation AI (Artificial Intelligence); measure the distance to the docking area using a ranging device; and calculate at least one of the relative distance or relative angle between the docking area and the ship from the measurement results obtained by the ranging device.

[0088] (Appendix 2) The information processing program according to Appendix 1, characterized in that the process of identifying the docking area includes a process of identifying a quay surface or a floating pier as the docking area.

[0089] (Appendix 3) The information processing program described in Appendix 1, characterized in that the process of calculating at least one of the relative distance or the relative angle includes a process of calculating at least one of the relative distance or the relative angle from the distance to the docking area, which is the measurement result, assuming that the docking area is vertical.

[0090] (Appendix 4) The information processing program described in Appendix 1, characterized in that the computer is caused to perform the following process: by performing calibration on the image and the measurement results, position information of the docking area in each of the image and the measurement results is obtained; and by performing registration on the position information in each of the image and the measurement results, the position of the docking area in the image and the measurement results is identified and aligned.

[0091] (Appendix 5) The information processing program described in Appendix 1, characterized in that the process of calculating at least one of the relative distance or the relative angle includes the processes of acquiring information on the amount of rotation and draft of the ship, correcting the angle of incidence based on the information on the amount of rotation and the information on the draft, and calculating the relative distance based on the corrected angle of incidence.

[0092] (Appendix 6) The information processing program described in Appendix 1, characterized in that the process of identifying the docking area includes a process of identifying the docking area from the acquired image excluding fenders, and the process of calculating the relative distance includes a process of calculating the distance from the ship to the fenders based on the relative distances between two points on the docking area among the calculated relative distances.

[0093] (Appendix 7) An information processing method characterized in that a computer executes the following processes: acquiring an image including a docking area photographed from a ship; identifying the docking area in the acquired image using segmentation AI; measuring the distance to the docking area using a ranging device; and calculating at least one of the relative distance or relative angle between the docking area and the ship from the measurement results obtained by the ranging device.

[0094] (Appendix 8) The information processing method according to appendix 7, wherein the process of identifying the docking area includes a process of identifying a quay surface or a floating pier as the docking area.

[0095] (Appendix 9) The information processing method described in Appendix 7, characterized in that the process of calculating at least one of the relative distance or the relative angle includes a process of calculating at least one of the relative distance or the relative angle from the distance to the docking area, which is the measurement result, assuming that the docking area is vertical.

[0096] (Appendix 10) The information processing method described in Appendix 7, characterized in that the computer performs the following processes: calibrating the image and the measurement results to obtain position information of the docking area in each of the image and the measurement results; and registering the position information in each of the image and the measurement results to identify and align the position of the docking area in the image and the measurement results.

[0097] (Appendix 11) The information processing method described in Appendix 7, characterized in that the process of calculating at least one of the relative distance or the relative angle includes the processes of acquiring information on the amount of rotation and draft of the ship, correcting the angle of incidence based on the information on the amount of rotation and the information on the draft, and calculating the relative distance based on the corrected angle of incidence.

[0098] (Appendix 12) The information processing method described in Appendix 7, characterized in that the process of identifying the docking area includes a process of identifying the docking area from the acquired image excluding fenders, and the process of calculating the relative distance includes a process of calculating the distance from the ship to the fenders based on the relative distances between two points on the docking area among the calculated relative distances.

[0099] (Appendix 13) An information processing device characterized by comprising a control unit that executes the following processes: acquiring an image including a docking area photographed from a ship; identifying the docking area in the acquired image using segmentation AI; measuring the distance to the docking area using a ranging device; and calculating at least one of the relative distance or relative angle between the docking area and the ship from the measurement results obtained by the ranging device.

[0100] (Supplementary Note 14) The information processing device according to Supplementary Note 13, characterized in that the process of identifying the docking area includes a process of identifying a quay surface or a floating pier as the docking area.

[0101] (Appendix 15) The information processing device described in Appendix 13, characterized in that the process of calculating at least one of the relative distance or the relative angle includes a process of calculating at least one of the relative distance or the relative angle from the distance to the docking area, which is the measurement result, assuming that the docking area is vertical.

[0102] (Appendix 16) The information processing device described in Appendix 13, characterized in that the control unit performs a process of obtaining position information of the docking area in each of the image and the measurement results by performing calibration on the image and the measurement results, and identifying and aligning the position of the docking area in the image and the measurement results by performing registration on the position information in each of the image and the measurement results.

[0103] (Appendix 17) The information processing device described in Appendix 13, characterized in that the process of calculating the relative angle includes the processes of acquiring information on the amount of rotation and draft of the ship, correcting the angle of incidence based on the information on the amount of rotation and the information on the draft, and calculating the relative distance based on the corrected angle of incidence.

[0104] (Appendix 18) The information processing device described in Appendix 13, characterized in that the process of identifying the docking area includes a process of identifying the docking area from the acquired image excluding fenders, and the process of calculating the relative distance includes a process of calculating the distance from the ship to the fenders based on the relative distances between two points on the docking area among the calculated relative distances.

[0105] (Supplementary Note 19) An information processing device comprising a processor and a memory operably connected to the processor, wherein the processor executes the following processes: acquire an image including a docking area photographed from a ship; identify the docking area in the acquired image using segmentation AI; measure the distance to the docking area using a ranging device; and calculate at least one of the relative distance or relative angle between the docking area and the ship from the measurement results obtained by the ranging device.

[0106] REFERENCE SIGNS LIST 1 Information processing system 10 Information processing device 10a Communication interface 10b HDD 10c Memory 10d Processor 20 Communication unit 30 Storage unit 31 Image information 32 Measurement information 33 Model information 40 Control unit 41 Acquisition unit 42 Identification unit 43 Measurement unit 44 Calculation unit 50 Network 100 Camera device 200 Distance measuring device

Claims

1. An information processing program that causes a computer to execute the following process: acquire an image including a docking area photographed from a ship; identify the docking area in the acquired image using segmentation AI (Artificial Intelligence); measure the distance to the docking area using a distance measuring device; and calculate at least one of the relative distance or relative angle between the docking area and the ship from the measurement results obtained by the distance measuring device.

2. The information processing program according to claim 1, characterized in that the process of identifying the docking area includes a process of identifying a quay surface or a floating pier as the docking area.

3. The information processing program according to claim 1, characterized in that the process of calculating at least one of the relative distance or the relative angle includes a process of calculating at least one of the relative distance or the relative angle from the distance to the docking area, which is the measurement result, on the assumption that the docking area is vertical.

4. The information processing program according to claim 1, characterized in that the computer is caused to perform the following process: by performing calibration on the image and the measurement results, position information of the docking area in each of the image and the measurement results is obtained; and by performing registration on the position information in each of the image and the measurement results, the position of the docking area in the image and the measurement results is identified and aligned.

5. The information processing program according to claim 1, characterized in that the process of calculating at least one of the relative distance or the relative angle includes the process of acquiring information on the amount of rotation and draft of the ship, correcting the angle of incidence based on the information on the amount of rotation and the information on draft, and calculating the relative distance based on the corrected angle of incidence.

6. The information processing program according to claim 1, characterized in that the process of identifying the docking area includes a process of identifying the docking area from the acquired image excluding fenders, and the process of calculating the relative distance includes a process of calculating the distance from the ship to the fenders based on the calculated relative distances to two points on the docking area.

7. An information processing method characterized by a computer executing the following processes: acquiring an image including a docking area photographed from a ship; identifying the docking area in the acquired image using segmentation AI; measuring the distance to the docking area using a ranging device; and calculating at least one of the relative distance or relative angle between the docking area and the ship from the measurement results obtained by the ranging device.

8. An information processing device comprising a control unit that executes the following process: acquiring an image including a docking area photographed from a ship; identifying the docking area in the acquired image using segmentation AI; measuring the distance to the docking area using a distance measuring device; and calculating at least one of the relative distance or relative angle between the docking area and the ship from the measurement results obtained by the distance measuring device.

Images