Information processing program, information processing method, and information processing device

Abstract

To solve the problem of difficulty of accurately grasping a distance or an angle between a vessel and a shore since a point to be measured cannot be specified since there are various irregularities such as a fender on the pier and a range finder cannot discriminate therebetween when an instructor for the vessel carrying out shore-arrival in a port instructs a direction and an action for having the vessel carry out shore-arrival while checking the distance and the angle to the shore by visual observation.SOLUTION: A computer acquires an image including a shore-arrival range photographed from a vessel, the shore-arrival range is specified among the acquired images by using segmentation AI, a distance from the shore-arrival range is measured by using a range finder, and at least one of a relative distance or a relative angle between the shore-arrival range and the vessel is calculated from the measured result by the range finder.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] In major ports around the world, including Japan, many ships, especially large ships, that dock within the port are docked by pushers. In this case, the person in charge of docking instructs the pusher to push, stop, and other actions during the docking operation. More specifically, for example, the person in charge of docking visually checks the distance and angle to the quay, and instructs the pusher on the direction and actions to dock the large ship.

[0003] As such, instructions to the pusher boat are given visually, and are therefore difficult to give, as they depend heavily on the intuition and experience of the person giving the instructions.For this reason, it is thought that instructions to the pusher can be supported by calculating the distance and angle between the ship and the quay using a rangefinder such as LiDAR (Light Detection and Ranging) or LRF (Laser Range Finder) and presenting this information to the person giving the instructions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-059403 [Patent Document 2] Japanese Patent Application Laid-Open No. 2005-180949 [Patent Document 3] Patent Publication No. 2021-136466 Summary of the Invention [Problem to be solved by the invention]

[0005] However, for example, a quay has various irregularities, including fenders, and the distance measuring device cannot distinguish between these, so it is not possible to identify the docking area to be measured, making it difficult to accurately grasp the distance and angle between the ship and the quay. The docking area may be a quay, a floating pier connected to the quay, etc.

[0006] In one aspect, an object of the present invention is to provide an information processing program, an information processing method, and an information processing device that can more accurately grasp the distance and angle between a ship and a quay. [Means for solving the problem]

[0007] In one aspect, the information processing program causes a computer to perform the following process: acquire an image including a docking area photographed from a ship; identify the docking area in the acquired image using segmentation AI (Artificial Intelligence); measure the distance to the docking area using a ranging device; and calculate, from the measurement results obtained by the ranging device, at least one of the relative distance between the docking area and the ship or the relative angle, which is the difference in azimuth angle between the quay and the parallel body of the ship. [Effects of the Invention]

[0008] On the one hand, it allows for a more accurate understanding of the relative distance and angle between the ship and the quay. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining an example of a method for measuring a relative distance and a relative angle to a quay wall. [Figure 2] FIG. 2 is a diagram showing an example of a camera image of a quay taken from a ship. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of the information processing system 1 according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of the information processing device 10 according to the first embodiment. [Figure 5]FIG. 5 is a diagram illustrating an example of cliff surface extraction by segmentation according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of distance measurement using the range finder according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of the registration process according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of calculation of a relative distance and a relative angle according to the first embodiment. [Figure 9] FIG. 9 is a diagram showing the distance measurement range when the docking area and the ship approach each other. [Figure 10] FIG. 10 is a diagram for explaining the calculation of the relative distance. [Figure 11] FIG. 11 is a flowchart showing the flow of the relative distance and relative angle calculation process according to the first embodiment. [Figure 12] FIG. 12 is a diagram for explaining rolling correction. [Figure 13] FIG. 13 is a diagram for explaining calculation of a relative distance by an information processing device according to the third embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of the hardware configuration of the information processing device 10. As shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] Examples of the information processing program, information processing method, and information processing device according to the present embodiment will be described in detail below with reference to the accompanying drawings. Note that the present embodiment is not limited to these examples. Furthermore, the examples can be combined as appropriate within a consistent range.

[0011] (First embodiment) First, a method for measuring the relative distance and relative angle from a large ship to a quay will be described. FIG. 1 is a diagram for explaining an example of a method for measuring the relative distance and relative angle to a quay. As shown in FIG. 1, for example, the distances from two points, one in front of and one behind the large ship, to the quay can be measured using range finders such as LiDAR and LRF installed on the ship, thereby determining the relative distance and relative angle from the large ship to the quay. Note that while FIG. 1 shows a large ship as an example, this embodiment is not limited to large ships and may include other small ships. Furthermore, in the description of each embodiment, the term "ship" may refer to both large ships and other small ships.

[0012] Although an example of a method for measuring the relative distance and relative angle from a ship to a quay has been described using Figure 1, there are challenges in actual measurement. Figure 2 shows an example of a camera image of a quay taken from a ship. As shown in Figure 2, various objects, such as fenders, exist around an actual quay. Therefore, when measuring using a range finder such as LiDAR, measuring the quay surface, which is the vertical surface of the quay that forms the docking area, leads to a more accurate understanding of the relative distance and relative angle between the ship and the quay. Here, the relative angle is the difference in azimuth angle between the quay and the parallel body of the ship, such as angle θ0 in Figure 1. Note that the azimuth angle of the parallel body is usually the same as the bow heading of the ship, but in this embodiment, the azimuth angle of the parallel body may be different from the bow heading of the ship.

[0013] However, measurements using a range finder such as LiDAR can measure, for example, the relative distance to each point in space using a set of data of vertices in a three-dimensional space called point cloud data or point cloud, but cannot determine what each point in space is. Therefore, it is not possible to determine whether the relative distance of each point in the point cloud data is, for example, the relative distance to the quay surface or the relative distance to an object other than the quay surface, such as a fender. Therefore, one of the objectives of this embodiment is to more accurately grasp the relative distance and relative angle between a ship and a quay.

[0014] Next, an information processing system according to this embodiment will be described. Fig. 3 is a diagram showing an example of the configuration of an information processing system 1 according to the first embodiment. The information processing system 1 is a system that calculates the relative distance and relative angle between a ship and its docking area. The docking area for a ship may be a quay as shown in Figs. 1 and 2, or a floating pier floating on the sea surface and connected to the quay as shown in Figs. 6 and 8 described below. As shown in Fig. 3, the information processing system 1 is a system in which an information processing device 10, a camera device 100, and distance measuring devices 200-1 and 200-2 (hereinafter collectively referred to as "distance measuring devices 200") are connected to each other via a network 50 so as to be able to communicate with each other.

[0015] The network 50 may be any of a variety of communication networks, whether wired or wireless, such as an intranet. The network 50 may not be a single network, but may instead be configured such that an intranet and the Internet are connected via a network device such as a gateway or other device (not shown).

[0016] The information processing device 10 is, for example, a server computer installed on a ship or the like, or an information processing device such as a desktop PC (Personal Computer) or a notebook PC. Although the information processing device 10 is shown as a single computer in FIG. 3, it may be a distributed computing system made up of multiple computers. Furthermore, the information processing device 10 may be a cloud computer device managed by a service provider that provides cloud computing services.

[0017] The information processing device 10 acquires an image including a docking area captured by the camera device 100 from a ship, for example, and identifies the docking area from the acquired image using a segmentation AI. The segmentation AI may be, for example, semantic segmentation or instance segmentation, and extracts the docking area from the image using a machine learning model. The machine learning model is, for example, a machine learning model generated by machine learning using images including a docking area tagged with information as training data.

[0018] In addition, the information processing device 10 measures the distance to the docking area, for example, using a distance measuring device 200, and calculates the relative distance and relative angle between the docking area extracted from the image captured by the camera device 100 and the ship from the measurement results by the distance measuring device 200.

[0019] The camera device 100 is, for example, a digital camera that is installed on a ship and captures images of the ship's surroundings, such as a quay, from the ship. Images captured by the camera device 100 are transmitted to the information processing device 10.

[0020] The ranging device 200 is a range finder such as LiDAR or LRF that is installed at multiple locations, for example, at the front and rear of the ship, and measures the distance to objects around the ship. Point cloud data, which is the measurement result measured by the ranging device 200, is transmitted to the information processing device 10. Note that the distance to an object measured by the ranging device 200 is, strictly speaking, the distance between the ranging device 200 and the object, but as described using FIG. 1, in this embodiment, the ranging device 200 is installed on the ship, and therefore, it is processed as the distance between the ship and the object.

[0021] (Functional configuration of information processing device 10) Next, the functional configuration of the information processing device 10 that executes this embodiment will be described. Fig. 4 is a diagram showing an example of the configuration of the information processing device 10 according to the first embodiment. As shown in Fig. 4, the information processing device 10 has a communication unit 20, a storage unit 30, and a control unit 40.

[0022] The communication unit 20 is a processing unit that controls communication with other devices such as the camera device 100, and is, for example, a communication interface such as a network interface card.

[0023] The storage unit 30 has a function of storing various data and programs executed by the control unit 40, and is realized by a storage device such as a memory or a hard disk. The storage unit 30 also stores various information such as image information 31, measurement information 32, and model information 33.

[0024] The image information 31 stores, for example, images captured by the camera device 100. Furthermore, the image information 31 may store, for example, information regarding docking areas identified from the images captured by the camera device 100 using segmentation AI.

[0025] The measurement information 32 stores, for example, point cloud data that is the measurement results measured by the distance measuring device 200. Furthermore, the measurement information 32 may store, for example, calculation result data such as the relative distance and relative angle between the docking area and the ship calculated from the point cloud data.

[0026] The model information 33 stores, for example, information about a machine learning model for extracting a docking area from an image captured by the camera device 100, and model parameters of the machine learning model. The machine learning model is generated by machine learning using training data in which the image captured by the camera device 100 is used as an explanatory variable and the docking area in the image is used as a target variable.

[0027] The above information stored in the storage unit 30 is merely an example, and the storage unit 30 can store various information other than the above information.

[0028] The control unit 40 is a processing unit, such as a processor, that controls the entire information processing device 10. The control unit 40 includes an acquisition unit 41, an identification unit 42, a measurement unit 43, and a calculation unit 44. Each processing unit is an example of an electronic circuit included in the processor or an example of a process executed by the processor.

[0029] The acquisition unit 41 acquires, for example, an image including a docking area photographed by the camera device 100 from a ship from the image information 31. The acquisition unit 41 also acquires position information of the docking area in each of the acquired image and the measurement results measured by the distance measuring device 200, for example, by performing calibration on the acquired image and the measurement results measured by the distance measuring device 200. Here, the measurement results are three-dimensional information (two-dimensional coordinates + distance) assuming LiDAR. The calibration process is a process of acquiring position information in each of the image photographed by the camera device 100 and the measurement results measured by the distance measuring device 200 in order to align the positions in different coordinate systems in the image photographed by the camera device 100 and the measurement results measured by the distance measuring device 200.

[0030] The identification unit 42 identifies the docking area in the image acquired by the acquisition unit 41, for example, using segmentation AI. Note that the identification of the docking area may use a machine learning model generated by machine learning using training data in which the image captured by the camera device 100 is used as an explanatory variable and the docking area in the image is used as a target variable. Information on the machine learning model is stored, for example, in the model information 33. Note that the identification of the docking area by the identification unit 42 may be performed based on markers installed on the quay surface, which is the docking area.

[0031] Furthermore, the process of identifying the docking area may include a process of identifying a quay wall surface or a floating pier as the docking area. FIG. 5 is a diagram showing an example of quay wall surface extraction by segmentation according to the first embodiment. FIG. 5 shows an example of identifying and extracting a quay wall surface as the docking area. The image shown on the left side of FIG. 5 is, for example, an image of a quay wall photographed from a ship as shown in FIG. 2, i.e., an image of the quay wall and its surroundings photographed by the camera device 100. As shown on the right side of FIG. 5, at least the quay wall surface is extracted from the image by using segmentation AI. Note that, for example, as shown in FIG. 5, fenders, holes, and other installations may exist on the quay wall surface, and trucks and port facilities may exist around the quay wall, but these objects are distinguished from the quay wall surface, and only the quay wall surface is extracted from the image.

[0032] Furthermore, the identification unit 42 identifies and aligns the position of the docking area in the image captured by the camera device 100 and the measurement results measured by the distance measuring device 200 by performing registration of the position information in the image and the measurement results. The registration process will be described later, but the identification unit 42 aligns the positions of different coordinate systems in the image captured by the camera device 100 and the measurement results measured by the distance measuring device 200 by the registration process.

[0033] The measurement unit 43 measures the distance to the docking area, for example, using the distance measuring device 200. As described above, the measurement using the distance measuring device 200 can measure the distance to each point in space, but cannot determine what each point in space is, so strictly speaking, the measurement unit 43 measures the distance to objects around the ship, including the docking area. Then, the measurement unit 43 extracts the distance to the docking area in the measurement results using the image aligned by the identification unit 42 and the position of the docking area in the measurement results.

[0034] The calculation unit 44 calculates the relative distance and relative angle between the docking area and the ship, for example, from the measurement results obtained by the distance measuring device 200. The calculated relative distance and relative angle may be used, for example, to provide maneuvering instructions to the pusher.

[0035] The calculation of the relative distance and the relative angle by the calculation unit 44 will be described in more detail. FIG. 6 is a diagram showing an example of distance measurement using a range finder according to the first embodiment. As shown in FIG. 6, the distance measuring device 200 installed on a ship is usually located higher than the quay, and therefore the ray from the distance measuring device 200 (denoted as "LiDAR irradiation" in FIG. 6) is oblique irradiation. Therefore, it is necessary to calculate the relative distance in the horizontal direction, denoted as "measured distance" in FIG. 6. Therefore, the calculation unit 44 calculates the relative distance and the relative angle from the distance to the quay area, which is the measurement result, on the assumption that the quay area is vertical, for example. The calculation process of the relative distance and the relative angle will be described later.

[0036] (Function details) Next, the registration process will be described in more detail. Fig. 7 is a diagram showing an example of the registration process according to the first embodiment. Fig. 7 shows an example in which a point P, which is a docking area, is photographed by the camera device 100 and measured by the distance measuring device 200. In Fig. 7, the position on the image photographed by the camera device 100 that corresponds to the point P is P C , the position on the point cloud data measured by the distance measuring device 200 is P L , respectively.

[0037] The distance from the ship to the docking area, point P, is calculated by L However, since it is not possible to determine what each point in space is based on point cloud data alone, P L Therefore, the information processing device 10 determines the positions in the coordinate systems different from those of the image captured by the camera device 100 and the point cloud data, and in the example of FIG. C and P L By combining this, P C From P L More specifically, if point P is an object that is represented as a single point on a map, such as an edge of a quay corner or a mooring post, the world coordinates (X W ,YW ,Z W ) can be specified in advance from map data, etc. Also, the position P of the docking area on the image captured by the camera device 100 C coordinates (X C ,Y C ) can also be identified by using segmentation AI to identify the docking area. Therefore, the world coordinates of point P (X W ,Y W ,Z W ) and the position of the docking area on the image P C coordinates (X C ,Y C ), but they can be aligned.

[0038] Then, the world coordinates of the point P and the position P on the point cloud data measured by the distance measuring device 200 are calculated. L The coordinates of point P can be converted into the world coordinates (X W ,Y W ,Z W ) to P L coordinates (X L ,Y L ) can be calculated.

[0039]

number

[0040] In formula (1), r 11 ~r 33 is the amount of rotation of the origin of the coordinate system by the distance measuring device 200 from the origin of the coordinate system by the camera device 100, and t1 to t3 are the amounts of movement of the origin of the coordinate system by the distance measuring device 200 from the origin of the coordinate system by the camera device 100. 11 ~r 33 and t1 to t3 are assumed to be specified in advance based on the installation positions of the two devices, and are stored in advance in the storage unit 30. By the registration process using the formula (1), the position P Land the point P, which is a world coordinate and an absolute coordinate, and further, the position P on the image captured by the camera device 100. C will be aligned with each other.

[0041] Next, the calculation of the relative distance and relative angle between the ship and the docking area will be described in more detail. Fig. 8 is a diagram showing an example of the calculation of the relative distance and relative angle according to the first embodiment. Fig. 8 shows an example in which the docking area is the vertical surface of a quay, but the docking area may also be the vertical surface of a floating pier or a fender installed on the vertical surface of a quay.

[0042] First, as shown in Fig. 8, segmentation AI is used to detect the vertical surface of the quay wall, which is the docking area, from an actual image captured by the camera device 100. Then, before, after, or in parallel with the image capture of the docking area by the camera device 100, the distance between the ship and objects around the docking area, including the docking area, is measured by the distance measuring device 200. Note that the distance measurement by the distance measuring device 200 may be performed by distance measuring devices 200 set at at least two locations, one in front of and one in back of the parallel body, which is the hull, as shown in Fig. 8.

[0043] Here, it is conceivable that the docking area and the ship are approaching each other, and the horizontal distance from the docking area to the camera device 100 is small or negative. FIG. 9 is a diagram showing the distance measurement range when the docking area and the ship are approaching each other. FIG. 9 shows the angle of view 101 captured by the distance measurement device 200. In FIG. 9, the distance measurement devices 200 are lined up overlapping each other in the depth direction of the paper (the fore-and-aft direction of the ship). Within the angle of view 101, part of a quay wall 111 and part of the sea 112 are captured.

[0044] The distance measuring device 200 cuts out a part of the range 102 of the angle of view 101 and uses it for distance measurement. As shown in FIG. 9, if the horizontal distance from the docking area to the camera device 100 is small or negative, it is conceivable that the range 102 used for distance measurement will hardly or not include an image of the vertical surface of the quay wall 111. In this case, it becomes difficult for the information processing device 10 to detect the vertical surface. Therefore, the information processing device 10 uses segmentation AI to detect the horizontal surface of the quay wall 111 and detect the quay wall line. Then, the distance measuring device 200 may measure the distance by regarding the position of the detected quay wall line as the position of the vertical surface.

[0045] 7, calibration and registration processes are performed to align the image captured by the camera device 100 with the measurement results measured by the distance measuring device 200. This allows data on the docking area to be extracted from the point cloud data, and the distance between the ship and the docking area to be obtained.

[0046] Next, as shown in FIG. 8, the information processing device 10 calculates the inclination between the vertical plane of the quay and the parallel body of the ship, i.e., the relative angle between the ship and the docking area, by geometric calculation processing. The information processing device 10 can use, for example, inverse trigonometric functions as the geometric calculation processing. The information processing device 10 then calculates the relative distance between the ship and the docking area, for example, from the relative angle calculated by performing the geometric calculation processing and the distance between the ranging device 200 and the docking area acquired from the point cloud data. FIG. 10 is a diagram for explaining the calculation of relative distance. When the angle when looking down on the quay from the ranging device 200, i.e., the incident angle from the ranging device 200 to the distance measurement target with respect to the horizontal plane, is Φ, and the distance between the ranging device 200 and the distance measurement target point in the docking area is L#1, the information processing device 10 calculates the relative distance as L#2 = L#1 × cos(Φ).

[0047] (Processing flow) Next, a flow of a process for calculating the relative distance and relative angle between the ship and the docking area, which is executed by the information processing device 10, will be described. Fig. 11 is a flowchart showing the flow of the relative distance / relative angle calculation process according to the first embodiment. The relative distance / relative angle calculation process shown in Fig. 11 may be executed, for example, when the information processing device 10 detects that the ship has entered within a predetermined range from the docking area.

[0048] 11 , the information processing device 10 acquires an image of the docking area captured by the camera device 100 and stored in the image information 31 (step S101). The camera device 100 may capture an image in response to an instruction sent from the information processing device 10 to the camera device 100, or may capture an image at any time by the camera device 100. The image captured by the camera device 100 is sent to the information processing device 10 and stored in the image information 31.

[0049] Next, the information processing device 10 identifies a docking area from the image acquired in step S101, for example, using a segmentation AI (step S102). The identified docking area may be, for example, a vertical surface of a quay or floating pier in the docking area, or a fender installed on the vertical surface of the quay, and is determined in advance for each docking area. Then, a machine learning model to be used by the segmentation AI is generated in advance by machine learning using images including the docking area to which information tags have been added as training data.

[0050] Next, the information processing device 10 measures the distance to the docking area using distance measuring devices 200 installed at multiple locations, for example, at the front and rear of the ship (step S103). As described above, the measurement by the distance measuring devices 200 cannot determine the identity of each point in space, so strictly speaking, the information processing device 10 measures the distance to objects around the ship, including the docking area. Point cloud data, which is the measurement result measured by the distance measuring devices 200, is transmitted to the information processing device 10 and stored in the measurement information 32. The measurement in step S103 may be performed before steps S101 and S102, or in parallel with steps S101 and S102.

[0051] Next, the information processing device 10 aligns the image acquired in step S101 with the measurement results measured in step S103 (step S104) by performing calibration processing and registration processing, for example, as described with reference to Fig. 7. This allows the information processing device 10 to extract data on the docking area from the measurement results measured in step S103, and to obtain the distance between the ship and the docking area.

[0052] Next, the information processing device 10 performs, for example, geometric calculation processing to calculate the relative distance between the ship and the docking area, and calculates the relative angle as described with reference to Fig. 8 (step S105). After step S105 is executed, the relative distance / relative angle calculation processing shown in Fig. 11 ends.

[0053] (effect) As described above, the information processing device 10 acquires an image including a docking area photographed from a ship, uses segmentation AI to identify the docking area in the acquired image, measures the distance to the docking area using a ranging device, and calculates the relative distance and relative angle between the docking area and the ship from the measurement results from the ranging device.

[0054] In this way, by using segmentation AI, the information processing device 10 can identify the quay surface from the captured image, calculate the relative distance and relative angle to the quay surface from the distance to the quay surface measured using a rangefinder, and more accurately grasp the distance and angle between the ship and the quay.

[0055] Furthermore, the process of specifying a docking area, which is executed by the information processing device 10, includes a process of specifying a quay surface or a floating pier as the docking area.

[0056] This allows the information processing device 10 to more accurately grasp not only the quay as the docking area, but also the distance and angle between the ship and the floating pier.

[0057] Furthermore, the process of calculating the relative distance and relative angle executed by the information processing device 10 includes a process of calculating the relative distance and relative angle from the distance to the docking area, which is the measurement result, on the assumption that the docking area is vertical.

[0058] This allows the information processing device 10 to more accurately grasp the distance and angle between the ship and the quay.

[0059] In addition, the information processing device 10 performs calibration on the image and the measurement results to obtain position information of the docking area in each of the image and the measurement results, and performs registration on the position information in each of the image and the measurement results to identify and align the position of the docking area in the image and the measurement results.

[0060] This allows the information processing device 10 to more accurately grasp the distance and angle between the ship and the quay.

[0061] (Second embodiment) Next, a second embodiment will be described. An information processing device 10 according to this embodiment is also shown in the block diagram of Fig. 4. In the following description, description of the operation of each unit similar to that of the first embodiment will be omitted.

[0062] A ship equipped with a camera device 100 and a distance measuring device 200 undergoes rolling, pitching, yawing, and heaving at sea. Rolling is rotation around the longitudinal axis of the ship. Pitching is rotation around the lateral axis of the ship. Yawing is rotation around the vertical axis. Heaving is vertical displacement of the ship. The angle of view of the camera device 100 fixed to the hull is affected by rolling, pitching, yawing, and heaving. Therefore, the information processing device 10 according to this embodiment performs distance measurement by correcting the angle of view in consideration of the effects of rolling, pitching, yawing, and heaving.

[0063] The ship according to this embodiment has rotation measuring instruments such as inclinometers that measure rolling, pitching, and yawing. The ship also has draft measuring instruments such as draft gauges that measure heaving. Furthermore, the accuracy of heaving can be improved by correcting with tide information, which is sea level height.

[0064] The calculation unit 44 calculates the relative angle with respect to the docking area, for example, assuming that the docking area is vertical. Next, the calculation unit 44 obtains the values ​​of rolling, pitching, and yawing from the rotation amount measuring instrument. The calculation unit 44 also obtains the value of heaving from the draft measuring instrument.

[0065] The calculation unit 44 corrects the angle of incidence from the distance measuring device 200 using information on rolling, pitching, yawing, and heaving. Then, the calculation unit 44 calculates the relative distance using the distance to the docking area, which is the measurement result, and the corrected relative angle and relative height.

[0066] FIG. 12 is a diagram for explaining rolling correction. Here, the rolling state of the ship is shown in accordance with the distance measuring device 200. Here, the quay line is the target point for distance measurement. Here, Ld is the distance detected by the distance measuring device 200. Also, θd is the inclination angle of the quay line from the distance measuring device 200 calculated from the angle of view of the camera device 100. Also, L10 is the horizontal relative distance between the distance measuring device 200 and the quay line.

[0067] State 301 shows a state in which the ship is not rolling. In this case, the inclination angle of the quay line from the distance measuring device 200 calculated from the angle of view of the camera device 100 matches with the horizontal direction of the distance measuring device 200 and θ, which is the angle connecting the distance measuring device 200 and the quay line. Therefore, the calculation unit 44 calculates the relative distance L10 as Ld × cos(θ).

[0068] In contrast, in state 302, the ship is rolling by an angle θs. In this case, the sum of the angle θd of the distance measuring device 200 and the rolling angle θs matches θ, which is the angle between the horizontal direction of the distance measuring device 200 and the quay line. In other words, θ = θd + θs. Therefore, the calculation unit 44 calculates the relative distance L10 = Ld × cos(θ) = Ld × cos(θd + θs).

[0069] Although the easy-to-understand rolling correction has been explained as an example here, the calculation unit 44 can also calculate other elements using geometric correction in the same way. The calculation unit 44 can also geometrically calculate the position of the ship's side, which is the subject of the relative distance between the position where the distance measuring device 200 is installed and the quay.

[0070] (effect) In this way, by calculating the relative distance by correcting the angle of incidence from the distance measuring device 200 and the mounting height of the distance measuring device 200 based on rolling, pitching, yawing, and heaving, it is possible to improve the accuracy of the calculated relative distance.

[0071] (Third embodiment) Next, a third embodiment will be described. An information processing device 10 according to this embodiment is also shown in the block diagram of Fig. 4. In the following description, description of the operation of each unit similar to that of the first embodiment will be omitted.

[0072] In the first embodiment, when the identification unit 42 identifies the docking area from the image acquired by the acquisition unit 41 using segmentation AI, it distinguishes fenders, holes, and other installations on the quay surface from the quay surface and extracts the quay surface from the image. In this case, if a fender is present at the position on the quay closest to the distance measurement origin, which serves as the ship's reference for relative distance, that position is excluded from the survey target. Therefore, the length from the ship to the closest position is not used as the relative distance, which may result in a decrease in accuracy.

[0073] Therefore, when the position of the quay where the fender is installed is the closest, the information processing device 10 according to this embodiment calculates the relative distance to the position of the fender using the relative distance to the position of the other quay, and sets this as the relative distance between the ship and the quay surface. The calculation process of the relative distance to the position of the fender by the information processing device 10 will be described below. Fig. 13 is a diagram for explaining the calculation of the relative distance by the information processing device according to the third example.

[0074] The calculation unit 44 obtains the distance to the docking area, which is the measurement result by the distance measuring device 200. In this case, the calculation unit 44 obtains the distances to multiple points on the docking surface. Then, using the image of the docking surface obtained by the camera device 100 and the distance to the docking area, which is the measurement result by the distance measuring device 200, the calculation unit 44 determines whether or not the distance to the position of the nearest quay wall surface is included. Hereinafter, the position of the nearest quay wall surface will be referred to as the "nearest point." If the distance to the nearest point is included, the calculation unit 44 uses that distance to calculate the relative distance.

[0075] On the other hand, if the distance to the nearest point is not included in the acquired distance because a fender 211 is present at the closest position, the calculation unit 44 uses the distance to an appropriate position 201 to calculate a relative distance L11 to the position 201. The calculation unit 44 also uses the distance to another appropriate position 202 to calculate a relative distance L12 to the position 202.

[0076] Next, the calculation unit 44 calculates the angle θ of the ship facing the quay using the relative distances L11 and L12. 11 and θ 12 Next, the calculation unit 44 calculates the relative distances L11 and L12 and the angle θ 11 and θ 12 Based on this, a geometric calculation process is performed to calculate the relative distance L1 from the distance measurement origin 210 to the nearest point 203.

[0077] Here, the thickness of the fender 211 is predetermined according to specifications. Therefore, the calculation unit 44 stores in advance a length L3 indicating the thickness of the fender 211. The calculation unit 44 calculates the relative distance L2 from the ship to the tip of the fender 211 by subtracting the length L3, which is the thickness of the fender 211, from the calculated relative distance L1.

[0078] Conversely, if the relative distance L2 from the distance measurement origin 210 to the tip of the fender 211 can be obtained by another method, the calculation unit 44 can also determine the deterioration of the fender 211 by the following method. For example, the calculation unit 44 adds the obtained relative distance L2 to the length L3, which is the specified thickness of the fender 211, and compares the result with the relative distance L1 to the nearest point 203, which is calculated based on the relative distances L11 and L12 and the angle of the quay with respect to the ship. Then, if the sum of the relative distance L2 and the length L3 is shorter than the relative distance L1, the calculation unit 44 can determine that the fender 211 has deteriorated by that amount.

[0079] (system) The information, including the processing procedures, control procedures, specific names, various data, and parameters shown in the above documents and drawings, may be changed as desired unless otherwise specified. Furthermore, the specific examples, distributions, and numerical values ​​described in the embodiments are merely examples and may be changed as desired.

[0080] Furthermore, the specific form of distribution or integration of the components of each device is not limited to that shown in the drawings. That is, all or part of the components may be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions of each device may be realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0081] (Hardware) Fig. 14 is a diagram illustrating an example of the hardware configuration of the information processing device 10. As shown in Fig. 14, the information processing device 10 includes a communication interface 10a, a hard disk drive (HDD) 10b, a memory 10c, and a processor 10d. The components shown in Fig. 14 are connected to each other via a bus or the like.

[0082] The communication interface 10a is a network interface card or the like, and performs communication with other information processing devices. The HDD 10b stores programs and data that operate the functions shown in FIG.

[0083] The processor 10d is a hardware circuit that operates a process that executes each function described in FIG. 4 and other figures by reading a program that executes the same processing as each processing unit shown in FIG. 4 from the HDD 10b or the like and expanding the program into the memory 10c. That is, this process executes the same functions as each processing unit of the information processing device 10. Specifically, the processor 10d reads a program having the same functions as the acquisition unit 41, the identification unit 42, and the like from the HDD 10b or the like. Then, the processor 10d executes a process that executes the same processing as the acquisition unit 41, the identification unit 42, and the like.

[0084] In this way, the information processing device 10 operates as an information processing device that executes operation control processing by reading and executing a program that executes processing similar to that of each processing unit shown in Fig. 4. The information processing device 10 can also realize functions similar to those of the above-described embodiment by reading a program from a recording medium using a medium reading device and executing the read program. Note that the program in these other embodiments is not limited to being executed by the information processing device 10. For example, this embodiment may also be applied in the same way to cases where another computer executes a program or where the information processing device 10 and another computer cooperate to execute a program.

[0085] A program that executes the same processes as those of the processing units shown in Fig. 4 can be distributed via a network such as the Internet. This program can be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disc (DVD), and can be executed by being read from the recording medium by a computer.

[0086] The following additional notes are provided regarding the embodiments including the above examples.

[0087] (Appendix 1) Obtain images including the docking area taken from the ship, Using segmentation AI (Artificial Intelligence), the docking area is identified from the acquired image, Using a distance measuring device, measure the distance to the docking area; At least one of the relative distance or the relative angle between the docking area and the ship is calculated from the measurement result by the distance measuring device. An information processing program that causes a computer to execute a process.

[0088] (Supplementary Note 2) The information processing program according to Supplementary Note 1, wherein the process of identifying the docking area includes a process of identifying a quay surface or a floating pier as the docking area.

[0089] (Supplementary Note 3) The process of calculating at least one of the relative distance and the relative angle includes: On the assumption that the docking area is vertical, at least one of the relative distance and the relative angle is calculated from the distance to the docking area, which is the measurement result. 2. The information processing program according to claim 1, comprising:

[0090] (Appendix 4) By performing calibration on the image and the measurement results, position information of the docking area in each of the image and the measurement results is obtained; By performing registration on the position information in the image and the measurement results, the position of the docking area in the image and the measurement results is identified and aligned. 2. The information processing program according to claim 1, wherein the information processing program causes the computer to execute the processing.

[0091] (Supplementary Note 5) The process of calculating at least one of the relative distance and the relative angle includes: Acquire information on the amount of rotation and draft of the vessel, correcting the angle of incidence based on the information on the amount of rotation and the information on the amount of draft; The relative distance is calculated based on the corrected incident angle. 2. The information processing program according to claim 1, further comprising:

[0092] (Appendix 6) The process of identifying the docking area is a process of identifying the docking area by excluding fenders from the acquired image, The process of calculating the relative distance includes: The method includes a process of calculating the distance between the ship and the fender based on the calculated relative distances for two points on the docking area. 2. The information processing program according to claim 1,

[0093] (Appendix 7) Obtain images including the docking area taken from the ship, Using segmentation AI, the docking area is identified from the acquired image; Using a distance measuring device, measure the distance to the docking area; At least one of the relative distance or the relative angle between the docking area and the ship is calculated from the measurement result by the distance measuring device. An information processing method characterized in that the processing is executed by a computer.

[0094] (Supplementary Note 8) The information processing method according to Supplementary Note 7, wherein the process of identifying the docking area includes a process of identifying a quay surface or a floating pier as the docking area.

[0095] (Supplementary Note 9) The process of calculating at least one of the relative distance and the relative angle includes: On the assumption that the docking area is vertical, at least one of the relative distance and the relative angle is calculated from the distance to the docking area, which is the measurement result. 8. The information processing method according to claim 7, further comprising:

[0096] (Appendix 10) By performing calibration on the image and the measurement results, position information of the docking area in each of the image and the measurement results is obtained; By performing registration on the position information in the image and the measurement results, the position of the docking area in the image and the measurement results is identified and aligned. 8. The information processing method according to claim 7, wherein the processing is executed by the computer.

[0097] (Supplementary Note 11) The process of calculating at least one of the relative distance and the relative angle includes: Acquire information on the amount of rotation and draft of the vessel, correcting the angle of incidence based on the information on the amount of rotation and the information on the amount of draft; The relative distance is calculated based on the corrected incident angle. 8. The information processing method according to claim 7, further comprising:

[0098] (Supplementary Note 12) The process of identifying the docking area includes: a process of identifying the docking area by excluding fenders from the acquired image, The process of calculating the relative distance includes: The method includes a process of calculating the distance between the ship and the fender based on the calculated relative distances for two points on the docking area. 8. The information processing method according to claim 7,

[0099] (Appendix 13) Obtain images including the docking area taken from the ship, Using segmentation AI, the docking area is identified from the acquired image; Using a distance measuring device, measure the distance to the docking area; At least one of the relative distance or the relative angle between the docking area and the ship is calculated from the measurement result by the distance measuring device. An information processing device comprising a control unit that executes processing.

[0100] (Supplementary Note 14) The information processing device according to Supplementary Note 13, wherein the process of specifying the docking area includes a process of specifying a quay surface or a floating pier as the docking area.

[0101] (Supplementary Note 15) The process of calculating at least one of the relative distance and the relative angle includes: On the assumption that the docking area is vertical, at least one of the relative distance and the relative angle is calculated from the distance to the docking area, which is the measurement result. 14. The information processing device according to claim 13, further comprising:

[0102] (Appendix 16) By performing calibration on the image and the measurement results, position information of the docking area in each of the image and the measurement results is obtained; By performing registration on the position information in the image and the measurement results, the position of the docking area in the image and the measurement results is identified and aligned. 14. The information processing device according to claim 13, wherein the control unit executes the processing.

[0103] (Supplementary Note 17) The process of calculating the relative angle is Acquire information on the amount of rotation and draft of the vessel, correcting the angle of incidence based on the information on the amount of rotation and the information on the amount of draft; The relative distance is calculated based on the corrected incident angle. 14. The information processing device according to claim 13, further comprising:

[0104] (Appendix 18) The process of identifying the docking area includes: a process of identifying the docking area by excluding fenders from the acquired image, The process of calculating the relative distance includes: The method includes a process of calculating the distance between the ship and the fender based on the calculated relative distances for two points on the docking area. 14. The information processing device according to claim 13,

[0105] (Appendix 19) A processor; a memory operatively connected to the processor; An information processing device comprising: Acquire an image including a docking area taken from a ship; Using segmentation AI, the docking area is identified from the acquired image; Using a distance measuring device, measure the distance to the docking area; At least one of the relative distance or the relative angle between the docking area and the ship is calculated from the measurement result by the distance measuring device. An information processing device that executes processing. [Explanation of symbols]

[0106] 1. Information Processing Systems 10. Information processing equipment 10a communication interface 10b HDD 10c memory 10d processor 20 Communications Department 30 Storage section 31 Image information 32 Measurement Information 33 Model Information 40 Control Unit 41 Acquisition Department 42 Specific part 43 Measuring part 44 Calculation section 50 Network 100 Camera Equipment 200 Rangefinder

Claims

1. Acquire an image including a docking area taken from a ship; Using segmentation AI (Artificial Intelligence), the docking area is identified from the acquired image; Using a distance measuring device, measure the distance to the docking area; Calculating at least one of the relative distance or the relative angle between the docking area and the ship from the measurement results of the distance measuring device An information processing program that causes a computer to execute a process.

2. 2. The information processing program according to claim 1, wherein the process of specifying the docking area includes a process of specifying a quay surface or a floating pier as the docking area.

3. The process of calculating at least one of the relative distance and the relative angle includes: On the assumption that the docking area is vertical, at least one of the relative distance and the relative angle is calculated from the distance to the docking area, which is the measurement result.

2. The information processing program according to claim 1, further comprising:

4. performing calibration on the image and the measurement results to obtain position information of the docking area in each of the image and the measurement results; By performing registration on the position information in the image and the measurement results, the position of the docking area in the image and the measurement results is identified and aligned.

2. The information processing program according to claim 1, wherein the information processing program causes the computer to execute processing.

5. The process of calculating at least one of the relative distance and the relative angle includes: Acquire information on the amount of rotation and draft of the vessel, correcting the angle of incidence based on the information on the amount of rotation and the information on the amount of draft; The relative distance is calculated based on the corrected incident angle.

2. The information processing program according to claim 1, further comprising:

6. The process of identifying the docking area includes: a process of identifying the docking area by excluding fenders from the acquired image, The process of calculating the relative distance includes: The method includes a process of calculating the distance between the ship and the fender based on the calculated relative distances for two points on the docking area.

2. The information processing program according to claim 1, wherein:

7. Acquire an image including a docking area taken from a ship; Using segmentation AI, identify the docking area in the acquired image; Using a distance measuring device, measure the distance to the docking area; Calculating at least one of the relative distance or the relative angle between the docking area and the ship from the measurement results of the distance measuring device An information processing method characterized in that the processing is executed by a computer.

8. Acquire an image including a docking area taken from a ship; Using segmentation AI, identify the docking area in the acquired image; Using a distance measuring device, measure the distance to the docking area; Calculating at least one of the relative distance or the relative angle between the docking area and the ship from the measurement results of the distance measuring device An information processing device comprising a control unit that executes processing.

Images