Ship information measuring program, ship information measuring method and ship information measuring device

Abstract

To provide a ship information measuring program, a ship information measuring method, and a ship information measuring device that can easily and accurately measure ship information such as a dimension, azimuth, and a position of a ship.SOLUTION: A ship information measuring device causes a computer to execute processes for capturing a predetermined ship with an image acquisition device to acquire a captured image of the predetermined ship; extracting a feature point of the predetermined ship in the captured image based on an output obtained by inputting the acquired captured image into a mechanical learning model extracting a feature point in a ship from an image; acquiring a distance from an image acquisition device to a horizontal plane where the predetermined ship is positioned; acquiring an inclination angle relative to the horizontal plane of the image acquisition device; and specifying ship information of the predetermined ship based on a position, a distance, and an inclination angle of the feature point in the captured image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a ship information measuring program, a ship information measuring method, and a ship information measuring device. [Background technology]

[0002] The position, dimensions, and heading of a ship are very important as key attributes that describe a ship, i.e., ship information. These information are used as basic values ​​in a wide range of applications, such as determining the type of ship, route selection in maritime traffic, and collision prevention.

[0003] Conventionally, one method for obtaining key ship attributes is to use an Automatic Identification System (AIS), which transmits these attributes from the ship itself. AIS automatically transmits and receives information on a ship's identification code, type, position, course, speed, navigation status, and other safety information over VHF radio waves, enabling information exchange between ship stations and between ship stations and land-based navigation aids. While AIS is a convenient and convenient means of obtaining ship information, because it is transmitted by the ship itself, it can be prone to issues such as transmitter failure, human error (e.g., incorrect or forgotten settings), intentional tampering, and intentional device disconnection, resulting in inaccurate information. In such cases, it is preferable to obtain ship information directly through measurements, as described below.

[0004] There are three technologies for non-contact measurement of the dimensions of large structures such as ships. One is a method that uses a device such as a 3D laser for measurement. For example, it is possible to measure the dimensions of a ship by scanning while moving a 3D laser scanner device along the ship and obtaining 3D position information of the hull surface shape. The other is a method that measures dimensions using images.

[0005] The other method is to use deep learning for segmentation. For example, an area that can be used for dimension measurement is surrounded and segmented in an image of the object to be measured, and deep learning is then performed using the segmentation, and the results of this deep learning can be used to determine the dimensions of the ship.

[0006] One technique for measuring the dimensions of an object is to identify the maximum vertical and horizontal widths in the captured image of a grain, and then calculate the maximum grain width. Another technique for detecting the shape of an object is to perform LRF (Local Regression Fitting) learning using the captured image of the object, and then detect the object's location points from the captured image. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-83775 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-256131 Summary of the Invention [Problem to be solved by the invention]

[0008] However, methods for measuring dimensions using devices such as 3D lasers require expensive equipment and various tasks, such as installing reflective targets on the object to be measured, making it difficult to achieve easy and accurate measurements. Furthermore, methods for measuring dimensions using images require manual input of the measurement target on the image using a mouse or other device, making it difficult to measure automatically, making it difficult to achieve easy and accurate measurements. Furthermore, methods using deep learning segmentation are based on the task of enclosing an area, which means that they only obtain rough dimensions of the exterior, making it difficult to obtain defined dimensions such as the ship's overall length and maximum width. Furthermore, methods using deep learning segmentation process the ship as two-dimensional information, making it difficult to accurately obtain attitude information such as the ship's orientation. Therefore, it is difficult to achieve easy and accurate measurements.

[0009] In addition, it is difficult to directly use the technology for calculating the size of grains and the technology for detecting the parts of objects using LRF learning to obtain information about ships at sea, and it is difficult to perform easy and accurate measurements of ship information using these technologies.

[0010] The disclosed technology has been developed in consideration of the above, and aims to provide a ship information measurement program, a ship information measurement method, and a ship information measurement device that can easily and accurately measure ship information. [Means for solving the problem]

[0011] In one aspect of the ship information measurement program, ship information measurement method, and ship information measurement device disclosed in the present application, a computer is caused to perform a process in which a specified ship is photographed with an image acquisition device to obtain a photographed image of the specified ship, the photographed image is input into a machine learning model that extracts feature points of the ship from the image, and based on the output obtained, feature points of the specified ship in the photographed image are extracted, the distance from the image acquisition device to the horizontal plane on which the specified ship is located is obtained, the inclination angle of the image acquisition device relative to the horizontal plane is obtained, and ship information of the specified ship is identified based on the position, the distance, and the inclination angle of the feature points in the photographed image. [Effects of the Invention]

[0012] In one aspect, the present invention provides an easy and accurate measurement of ship information. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of an operation method of a vessel information measuring device according to an embodiment. [Figure 2] FIG. 2 is a diagram showing the features and dimensions of the ship information. [Figure 3] FIG. 3 is a diagram showing an example of the ship length and ship width. [Figure 4] FIG. 4 is an image diagram showing the installation state and appearance of the ship information measuring device. [Figure 5] FIG. 5 is a block diagram of the ship information measuring device. [Figure 6] FIG. 6 is a diagram for explaining various coordinate systems. [Figure 7] FIG. 7 is a diagram showing the feature point extraction process performed by the image recognition unit. [Figure 8] FIG. 8 is a diagram showing how to obtain conversion coefficients for converting pixel positions on the image coordinate system into those on the screen coordinate system. [Figure 9] FIG. 9 is a diagram showing the transformation from the screen coordinate system to the camera coordinate system. [Figure 10]FIG. 10 is a diagram showing the relationship between points on the virtual projection plane and points in real space. [Figure 11] FIG. 11 is a diagram showing the relationship between points in real space and an image acquisition device. [Figure 12] FIG. 12 is a diagram showing the relationship between the angle obtained from the tilt measurement unit and the Euler angles. [Figure 13] FIG. 13 is a flowchart of a process of measuring ship information by the ship information measuring device according to the embodiment. [Figure 14] FIG. 14 is a diagram for explaining the calculation of the measurement error. [Figure 15] FIG. 15 is a diagram showing the accuracy measurement results. [Figure 16] FIG. 16 is a diagram showing the hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0014] The following describes in detail embodiments of the ship information measuring program, ship information measuring method, and ship information measuring device disclosed herein with reference to the accompanying drawings. Note that the ship information measuring program, ship information measuring method, and ship information measuring device disclosed herein are not limited to the following embodiments. [Example]

[0015] FIG. 1 is a diagram showing an example of an operation method of a ship information measuring device according to an embodiment. The ship information measuring device 100 according to this embodiment is mounted on an unmanned aerial vehicle 1, for example, as shown in FIG. 1. The unmanned aerial vehicle 1 is a drone, a radio-controlled aircraft, or the like. However, the ship information measuring device 100 only needs to be placed in a position where it can photograph ships at sea, and may be mounted on a manned airplane, a balloon, or the like, or on a tower at sea. In the unmanned aerial vehicle 1 flying over the sea, the ship information measuring device 100 measures ship information including the positions, dimensions, and orientations of ships 21 and 22 floating on the sea.

[0016] 2 is a diagram showing feature points and dimensions of ship information. Here, a ship 21 will be described as an example. For example, the ship information measuring device 100 acquires an image of the ship 21 and uses the acquired image to identify the positions of feature points 211, 212, 221, and 222 in three-dimensional space. The feature point 211 corresponds to the bow of the ship 21, and the feature point 212 corresponds to the stern of the ship 21. Furthermore, the feature point 212 corresponds to the starboard side of the ship 21, and the feature point 222 corresponds to the port side of the ship 21.

[0017] The vessel information measuring device 100 then calculates the length of the vessel, which is the distance from the characteristic point 211 at the bow to the characteristic point 212 at the stern. The vessel information measuring device 100 also calculates the line width, which is the distance from the characteristic point 221 at the port side to the characteristic point 222 at the starboard side.

[0018] 3 is a diagram showing an example of ship length and ship width. The length L1 of the ship 21 is called the overall length. The length L2 of the ship 21 is called the registered length. The length L3 of the ship 21 is called the length between perpendiculars. The ship information measuring device 100 can treat any of the lengths L1 to L3 as the ship length by changing the objects extracted as the feature points 211 and 212.

[0019] Furthermore, the width W1 of the ship 21 is called the maximum width. Furthermore, the width W2 of the ship 21 is called the mold width or registered width. The ship information measuring device 100 can treat either width W1 or W2 as the ship's width by changing the target to be extracted as the feature points 221 and 222.

[0020] Figure 4 is an image diagram of the installation state and appearance of the ship information measuring device. The ship information measuring device 100 is stored in a two-axis gimbal 10 installed on the underside of the body near the front of the unmanned vehicle 1. As shown in Figure 4, the two-axis gimbal 10 has openings 11 and 12. Furthermore, the two-axis gimbal 10 has an EL (Elevator) axis 14 and an AZ (Azimuth) axis 15 as rotation axes. That is, the two-axis gimbal 10 rotates in a direction perpendicular to the body of the unmanned vehicle 1 around the EL axis 14. The two-axis gimbal 10 also rotates in a direction parallel to the body of the unmanned vehicle 1 around the AZ axis 15. The two-axis gimbal 10 stores the ship information measuring device 100.

[0021] As shown in Fig. 4, the ship information measuring device 100 has a camera 101, a rangefinder 102, an inclinometer 103, and a computer 104. When the ship information measuring device 100 is stored in the biaxial gimbal 10, the imaging lens of the camera 101 is positioned so as to face the opening 12. In this case, the visual axis 13 of the camera 101 is positioned in a direction from the opening 12 of the biaxial gimbal 10 toward the outside, as shown in Fig. 4. Furthermore, if the rangefinder 102 is, for example, a laser rangefinder, it is positioned so that the laser emitting lens faces the opening 11 of the biaxial gimbal 10.

[0022] Fig. 5 is a block diagram of the ship information measuring device. Next, details of the ship information measuring device 100 will be described with reference to Fig. 5. As shown in Fig. 5, the ship information measuring device 100 has an image acquisition unit 111, a distance measurement unit 112, an inclination measurement unit 113, an image recognition unit 114, a calculation unit 115, and a communication unit 116.

[0023] FIG. 6 is a diagram for explaining various coordinate systems. In the following explanation, the camera coordinate system Σ CAM , camera horizontal coordinate system Σ HDR , the screen coordinate system Σ SCR and the image coordinate system Σ IMG Four coordinate systems are used:

[0024] Camera coordinate system Σ CAM is the coordinate system 301 in FIG. 6. The camera coordinate system Σ CAMis a coordinate system fixed with respect to the image acquisition unit 111, with the optical center of the image acquisition unit 111 as the origin, the visual axis 120 as the +x-axis, the downward direction of the image acquisition unit 111 with respect to the visual axis 120 as the +z-axis, and the axis perpendicular to the x-axis and z-axis as the y-axis. CAM The units of the x, y, and z components in are all meters (m).

[0025] Camera horizontal coordinate system Σ HOR is the coordinate system 302 in FIG. 6. The camera horizontal coordinate system Σ HOR is the camera coordinate system Σ CAM is rotated around the x-axis and the y-axis, the z-axis is directed from the optical center of the image acquisition unit 111 to the vertical direction relative to the ground surface, and the visual axis 120 is aligned with the horizontal coordinate system Σ HOR The horizontal coordinate system Σ is a coordinate system in which the x-axis is defined to exist on the xz plane of the camera. HOR The x-axis of the camera coordinate system Σ lies on a horizontal plane 122 with respect to the ground surface that passes through the optical center of the camera 101 of the image acquisition unit 111. CAM The units of the x, y, and z components are all m (meters).

[0026] Screen coordinate system Σ SCR is the coordinate system 303 in FIG. 6. The screen coordinate system Σ SCR is the camera coordinate system Σ CAM is translated along the visual axis 120 toward the ship 21 by the focal length, and is a coordinate system virtually arranged by a perspective projection model. Here, the virtual projection plane 123 is a coordinate system in which the camera coordinate system Σ CAM The screen coordinate system Σ is a plane whose normal is the visual axis 120 at a position where the screen is moved parallel to the visual axis 120 by the focal length toward the ship 21. SCR In the screen coordinate system Σ, the y-axis and z-axis are positioned on the virtual projection plane 123. The coordinate system 303 in FIG. 6 shows the y-axis and z-axis on the virtual projection plane 123. SCR The screen coordinate system Σ has its x-axis in the depth direction of FIG. SCR The units of the x, y, and z components in are all m (meters), and the x component is always 0.

[0027] Image coordinate system Σ IMG is the coordinate system 304 in FIG. IMG represents coordinates on the virtual projection plane 123 of the perspective projection model, and is a two-dimensional coordinate system corresponding to the pixels of the image. IMG has u and v axes. The u axis is in the screen coordinate system Σ SCR The v coordinate is in the screen coordinate system Σ SCR The image coordinate system Σ overlaps with the z coordinate in IMG The units of the u and v components in are both pixels.

[0028] For example, the ship information measuring device 100 extracts the feature point 200 in Fig. 6. The feature point 200 to be extracted may be located at a position deviated from the visual axis 120 as shown in Fig. 6.

[0029] The vessel information measuring device 100 is configured so that the x-axis of the camera 101 of the image acquiring unit 111, the x-axis of the distance measuring unit 112, and the x-axis of the tilt measuring unit 113 are aligned. Here, the x-axis of the camera 101 of the image acquiring unit 111 corresponds to the visual axis 120 of the camera 101 of the image acquiring unit 111. The x-axis of the distance measuring unit 112 corresponds to the direction in which the distance is measured.

[0030] 4. The image acquisition unit 111 photographs the ship 21, which is the target of information acquisition, with the camera 101 to acquire an image of the ship 21. Then, the image acquisition unit 111 outputs the acquired image data of the ship 21 to the image recognition unit 114.

[0031] Distance measurement unit 112 is realized by rangefinder 102 in Fig. 4. Distance measurement unit 112 measures the distance from the optical center of image capture by image acquisition unit 111 to the horizontal plane on which the ship is located at the end of visual axis 120. If visual axis 120 is at the sea surface, this distance is the distance from the optical center of image capture by image acquisition unit 111 to the sea surface. Distance measurement unit 112 then outputs the measured distance data to calculation unit 115.

[0032] The tilt measurement unit 113 is realized by the inclinometer 103 in Fig. 4. The tilt measurement unit 113 calculates the tilt angle relative to the horizontal plane on which the ship is located in the horizontal camera coordinate system Σ of the imaging lens of the image acquisition unit 111. HOR The tilt measurement unit 113 constantly acquires the roll angle and pitch angle relative to the tilt angle. The tilt measurement unit 113 outputs information on the measured roll angle and pitch angle to the calculation unit 115.

[0033] The image recognition unit 114 receives input of image data from the image acquisition unit 111. Then, the image recognition unit 114 extracts feature points 211, 212, 221, and 222 of the ship 21 in the image. Here, the extraction process of the feature point 211 from the image by the image recognition unit 114 will be described using the feature point 211 as an example.

[0034] 7 is a diagram showing the feature point extraction process performed by the image recognition unit 114. For example, the image recognition unit 114 has a neural network 140, which is a trained machine learning model used to extract feature points 211 from image data.

[0035] The neural network 140 may use, for example, HRNet, which can estimate the positions of feature points 211 from within an image with high accuracy. In the case of NRNet, the neural network 140 has a high-resolution network 141 that does not lose resolution in parallel with a network that narrows down the search to extract feature points. The neural network 140 estimates the feature points 211 using the high-resolution network 141 together with the network that narrows down the search to extract feature points. The neural network 140 performs supervised learning using image data that has the positions of each feature point as teacher labels. Note that by creating teacher labels in accordance with the various dimension definitions shown in FIG. 3, the ship information measuring device 100 can perform measurements in accordance with various dimension definitions.

[0036] The image recognition unit 114 inputs image data of the ship 21 to the neural network 140. Thereafter, the image recognition unit 114 acquires a probability distribution 142 of the positions of the feature points 211 as output from the neural network 140. The image recognition unit 114 then estimates a position 143 of the feature points 211 on the image using the probability distribution of the positions of the feature points 211. In this way, by determining the position 143 of the feature points 211 from the probability distribution 142 of the positions of the feature points 211, the image recognition unit 144 can recognize the position 143 of the feature points 211 from information about the surroundings, even if the feature points 211 are not directly visible in the image. In other words, the ship information measuring device 100 recognizes the feature points 211 from the entire surrounding image, and therefore can measure ship information even if there are areas that are partially obscured.

[0037] Thereafter, the image recognition unit 114 outputs the extracted feature points 211, 212, 221, and 222 of the ship 21 to the calculation unit 115.

[0038] The calculation unit 115 receives an input of distance data from the distance measurement unit 112. The calculation unit 115 also receives an input of information on the roll angle and pitch angle of the image acquisition unit 111 from the tilt measurement unit 113. Furthermore, the calculation unit 115 receives an input of position information of feature points 211, 212, 221, and 222 of the ship 21 on the image from the image recognition unit 114.

[0039] Then, the calculation unit 115 calculates the positions in real space of the feature points 211, 212, 221, and 222 using the distance data, information on the roll angle and pitch angle of the image acquisition unit 111, and position information of the feature points 211, 212, 221, and 222. Here, the calculation unit 115 converts each of the feature points 211, 212, 221, and 222 into a position in real space, but the process is the same for all of them. Therefore, the following description will be given taking the feature point 211 as an example.

[0040] The calculation unit 115 calculates the image coordinate system Σ IMG The pixel position on the screen is expressed as the screen coordinate system Σ using the following formula (1): SCRHereinafter, the feature point 211 shown on the image will be referred to as a feature point 211'.

[0041]

number

[0042] where a u and a v is the image coordinate system Σ IMG The pixel position on the screen coordinate system Σ SCR is the conversion factor for converting to a u and a v The unit of is "m / pixel". Also, u and v are in the image coordinate system Σ IMG It represents the position of the upper feature point 211' and is expressed in units of "pixels." SCR p is the screen coordinate system Σ SCR is the position vector of the feature point 211' on the virtual projection plane 123 as seen from above.

[0043] Screen coordinate system Σ SCR is an example of a “first coordinate system.” That is, the calculation unit 115 converts the positions of the feature points in the captured image into positions in the first coordinate system that uses the capturing direction of the camera 101 as a reference.

[0044] 8 is a diagram showing how to obtain conversion coefficients for converting pixel positions on the image coordinate system into the screen coordinate system. The coordinate system 301 in FIG. 8 is the camera coordinate system Σ CAM The point 310 is in the camera coordinate system Σ CAM Also, the coordinate system 303 corresponds to the origin of the screen coordinate system Σ SCR This corresponds to

[0045] Image coordinate system Σ IMG The pixel position on the screen coordinate system Σ SCR a is the conversion factor for converting u and a v is expressed as shown in the following equations (2) and (3) from the definition of the tangent of a right-angled triangle.

[0046]

number

number

[0047] Here, the pic H corresponds to the horizontal pixel 311 in FIG. V corresponds to the vertical pixel 312 in the image in FIG. H corresponds to the horizontal angle of view 313 in Figure 8, and the FOV V corresponds to the vertical angle of view 314 in FIG.

[0048] 9 is a diagram showing the transformation from the screen coordinate system to the camera coordinate system. The coordinate system 301 in FIG. 9 is the camera coordinate system Σ CAM The coordinate system 303 is the screen coordinate system Σ SCR The camera coordinate system Σ CAM and the screen coordinate system Σ SCR are separated by a focal length f in the x-axis direction.

[0049] The calculation unit 115 calculates the coordinate system Σ CAM and the screen coordinate system Σ SCR Using the following formula (4) obtained from the vector relation between and, the screen coordinate system Σ SCR The position vector of the feature point 211' in the camera coordinate system Σ CAM to a position vector in CAM p is the camera coordinate system Σ of the feature point 211′ CAM represents the position vector at

[0050]

number

[0051] That is, the calculation unit 115 calculates the screen coordinate system Σ as shown in FIG. SCR The position vector 315 of the feature point 211 in the camera coordinate system Σ CAMThe vector 316 is converted into a position vector 316 of the feature point 211' in the image.

[0052] Next, the calculation unit 115 calculates the coordinate system Σ CAM The position vector of the feature point 211' expressed by the camera horizontal coordinate system Σ HOR Convert it into a position vector in

[0053]

number

[0054] where: HOR P is the horizontal camera coordinate system Σ of the feature point 211′ HOR represents the position vector in HOR R CAM is the camera coordinate system Σ CAM The position vector of the camera horizontal coordinate system Σ HDR This is the coordinate transformation matrix for converting the coordinates into the position vector of HOR R CAM is the camera coordinate system Σ CAM the camera horizontal coordinate system Σ HDR Then, rotate the camera coordinate system Σ CAM By rotating θx about the X axis of the camera coordinate system Σ CAM the camera horizontal coordinate system Σ HDR is a matrix that matches the matrix S. Also, S stands for sin. For example, S θy = sin(θy). Also, C represents cos. For example, C θy =cos(θy).

[0055] Here, the calculation unit 115 calculates the following: HOR R CAM The calculation unit 115 generates a simple matrix I 3×3 , the camera horizontal coordinate system Σ HDR In order to rotate the coordinate system Σ by θy about the Y axis, the calculation unit 115 multiplies the coordinate system Σ by a basic rotation matrix about the Y axis expressed by the following formula (7) from the left. CAMTo rotate θx about the X axis, we multiply it from the right by the basic rotation matrix about the X axis shown in the following formula (6): HOR R CAM Generate.

[0056]

number

number

[0057] That is, the calculation unit 115 calculates the camera coordinate system Σ CAM The position vector of the camera horizontal coordinate system Σ HOR is the coordinate transformation matrix for converting the position vector HOR R CAM Generate.

[0058]

number

[0059] The calculation unit 115 can generate a coordinate transformation matrix even if the order of rotation is changed. When performing rotation about the Z axis, the calculation unit 115 can generate a coordinate transformation matrix using a basic rotation matrix about the Z axis expressed by the following mathematical formula (9).

[0060]

number

[0061] Here, the camera horizontal coordinate system Σ HOR is an example of the "second coordinate system." That is, the calculation unit 115 calculates the coordinate system Σ SCR The position of the feature point in the horizontal camera coordinate system Σ is a second coordinate system based on a plane parallel to the horizontal plane. HOR Convert to a position in

[0062] 10 is a diagram showing the relationship between points on the virtual projection plane and points in real space. A coordinate system 401 represents a coordinate space in real space 400. Here, feature point 211 is a point in real space 400. Furthermore, feature point 211' is a point on virtual projection plane 123.

[0063] Next, the calculation unit 115 calculates the position vector of the feature point 211 in the real space. CAM P 211 and the position vector of the feature point 211' on the virtual projection plane 123 CAM p and the camera coordinate system Σ CAM Since they are on the same line at CAM P 211 and CAM As shown in FIG. 10, the position vector P of the feature point 211 in the real space and the position vector p of the feature point 211′ on the virtual projection plane 123 are expressed in the camera coordinate system Σ CAM If the vectors are on the same line as viewed from the origin, the position vector P can be expressed as a scalar multiplication of the position vector p.

[0064] Therefore, the calculation unit 115 calculates the position vector as shown in the following formula (10). CAM P 211 of CAM It is denoted by p, where b is a scalar.

[0065]

number

[0066] 11 is a diagram showing the relationship between points in real space and the image acquisition device. Here, it is assumed that the feature point 211 is on the sea surface 125, and the sea surface 125 is horizontal. In this case, the horizontal plane 122 relative to the image acquisition unit 111 is parallel to the sea surface 125 on which the feature point 211 exists. Therefore, the calculation unit 115 calculates the camera horizontal coordinate system Σ of the feature point 211 in real space. HOR Position vector in HOR P is expressed by the following formula (11).

[0067]

number

[0068] where Px and Py are the horizontal coordinate system of the camera Σ HOR Position vector in HOR represents the x and y components of P. Also, h is the horizontal coordinate system of the camera Σ HOR θy represents the height of the optical center of the image acquisition unit 111, which is the origin of the camera horizontal coordinate system Σ HOR 11, θy is the angle of the visual axis 120 with respect to the horizontal plane 122, which corresponds to the xy plane of the image, i.e., the angle that the visual axis 120 moves when rotated around the y axis until it coincides with the horizontal plane 122. In FIG. 11, θy is a negative number.

[0069] Next, the calculation unit 115 solves the equations for the three variables b, Px, and Py using the formulas (5), (10), and (11). Specifically, the calculation unit 115 performs calculations as shown in the following formula (12) from the formulas (5), (10), and (11).

[0070]

number

[0071] Then, the calculation unit 115 solves the equation (12) to obtain b as the following equation (13).

[0072]

number

[0073] Furthermore, the calculation unit 115 obtains Px as the following formula (14).

[0074]

number

[0075] Furthermore, the calculation unit 115 obtains Py as the following formula (15).

[0076]

number

[0077] Here, the calculation unit 115 can calculate u and v from the position of the feature point 211 on the screen acquired from the image recognition unit 114. The calculation unit 115 can also set the distance acquired from the distance measurement unit 112 as r. Furthermore, the calculation unit 115 can calculate the camera coordinate system Σ CAM The camera horizontal coordinate system Σ HDR The Euler angles θx and θy that represent the attitude with respect to the object can be obtained.

[0078] 12 is a diagram showing the relationship between the angle obtained from the tilt measurement unit and the Euler angles. In FIG. 12, the camera coordinate system Σ shown in the coordinate system 301 CAM and the camera horizontal coordinate system Σ shown in the coordinate system 302. HDR and the horizontal plane 122. Also, the vector 305 is in the camera coordinate system Σ CAM This is the vertical direction toward the sea surface as seen from above, and is expressed by the matrix shown in Figure 12.

[0079] 12 from the roll angle and pitch angle of the image acquisition unit 111 acquired from the tilt measurement unit 113. Specifically, the calculation unit 115 calculates the Euler angles θx and θy as θy=Φ1 and θx=arcsin(sin(θ) / cos(θy)).

[0080] Furthermore, the calculation unit 115 converts the calculated Euler angles θx and θy into the camera coordinate system Σ CAM Using a vector 305 that corresponds to the vertical direction toward the sea surface as seen from above, this can be expressed as the following equations (16) and (17).

[0081]

number

number

[0082] In this way, the calculation unit 115 calculates Px and Py, which represent the position of the feature point 211, using formulas (14) and (15). In this way, the calculation unit 115 identifies the ship information of the specified ship based on the position, distance, and tilt angle of the feature point in the second coordinate system expressed by formula (5).

[0083] Here, for example, when u=0, Px and Py representing the position of the feature point 211 are expressed as the following equation (18).

[0084]

number

[0085] Furthermore, for example, when v=0, Px and Py representing the position of the feature point 211 are expressed as the following equation (19).

[0086]

number

[0087] Furthermore, for example, when θx=0, Px and Py representing the position of the feature point 211 are expressed as the following equation (20).

[0088]

number

[0089] Furthermore, for example, when θx=0 and u=0, Px and Py representing the position of the feature point 211 are expressed as the following equation (21).

[0090]

number

[0091] Furthermore, for example, when θx=0 and v=0, Px and Py representing the position of the feature point 211 are expressed as the following equation (22).

[0092]

number

[0093] In this way, the calculation unit 115 calculates the position in real space of the feature point 211. For example, when calculating the ship's length, the calculation unit 115 similarly calculates the position in real space of the feature point 212.

[0094] For example, the image coordinate system Σ IMG When the coordinate values ​​of the feature point 211 on the image plane are (u1, v1), the calculation unit 115 calculates the coordinate values ​​(P x1 ,P y1 ) is calculated as the following formula (23).

[0095]

number

[0096] Also, the image coordinate system Σ IMG When the coordinate values ​​of the feature point 212 on the image plane are (u2, v2), the calculation unit 115 calculates the coordinate values ​​(P x2 ,P y2 ) is calculated as the following formula (24).

[0097]

number

[0098] Here, the parameters in the formulas (23) and (24) will be summarized again. r is the coordinate system of the camera Σ on the visual axis 120. CAM is the distance from the origin to the horizontal plane that corresponds to the sea surface. Also, θx and θy are the distances in the camera horizontal coordinate system Σ HDR camera coordinate system Σ CAMIn addition, f is the focal length, which is the distance from the optical origin of the image acquisition unit 111 to the virtual projection plane 123. In addition, au and av are the coordinates of the image coordinate system Σ IMG The pixel position on the screen coordinate system Σ SCR is the conversion factor for converting

[0099] Then, the calculation unit 115 calculates the ship length L by the following equation (25) using the positions of the feature points 211 and 212 in the real space. 1-2 can be calculated.

number

[0100] Furthermore, the calculation unit 115 can calculate the position of the ship 21 using the following equation (26).

number

[0101] Furthermore, the calculation unit 115 can calculate the heading of the ship 21 using the following equation (27).

number

[0102] The calculation unit 115 can also calculate the ship width in the same way. Furthermore, the calculation unit 115 calculates the camera coordinate system Σ in real space from a GPS (Global Positioning System) (not shown) mounted on the ship information measuring device 100 or the unmanned vehicle 1. CAM The calculation unit 115 then calculates the position and orientation of the ship 21 from the obtained feature points 211, 212, 221, and 222. Thereafter, the calculation unit 115 outputs information on the calculated position, dimensions, and orientation of the ship 21 to the communication unit 116.

[0103] Continuing the explanation by returning to Figure 1, the communication unit 116 receives input of information on the position, dimensions, and orientation of the ship 21 from the calculation unit 115. The communication unit 116 then transmits the information on the position, dimensions, and orientation of the ship 21 to the terminal device 3 via an antenna (not shown) mounted on the unmanned vehicle 1.

[0104] 13 is a flowchart of the process of measuring ship information by the ship information measuring device according to the embodiment. Next, the flow of the process of measuring ship information by the ship information measuring device 100 according to the embodiment will be described with reference to FIG.

[0105] The image acquisition unit 111 photographs the ship 21 that is the target of information measurement and acquires an image of the ship 21 (step S1).

[0106] The distance measurement unit 112 measures the distance along the visual axis 120 from the optical origin of the image acquisition unit 111 to the sea surface (step S2).

[0107] The tilt measurement unit 113 measures the roll angle and pitch angle of the image recognition unit 114 (step S3).

[0108] The image recognition unit 114 inputs the image data of the ship 21 acquired from the image acquisition unit 111 into a neural network 140 such as HRNet, and acquires as output a probability distribution of the positions of the feature points 211, 212, 221, and 222. Then, the image recognition unit 114 uses the acquired probability distribution to calculate the position of each point on the image, i.e., the image coordinate system Σ IMG The position vectors of the points 211, 212, 221, and 222 are obtained and the feature points 211, 212, 221, and 222 are extracted (step S4).

[0109] The calculation unit 115 receives an input of the position vectors of the feature points 211, 212, 221, and 222 from the image recognition unit 114. Then, the calculation unit 115 selects one point from among the feature points 211, 212, 221, and 222 (step S5). Here, the selected point will be referred to as the "feature point 210" without distinguishing between the feature points 211, 212, 221, and 222.

[0110] The calculation unit 115 calculates the image coordinate system Σ of the feature point 210 using the formulas (1) and (4). IMG The position vector in the camera coordinate system Σ CAM (Step S6).

[0111] Next, the calculation unit 115 calculates the coordinate system Σ of the feature point 210 using the formula (5). CAM The position vector in the camera horizontal coordinate system Σ HDR (Step S7).

[0112] Next, the calculation unit 115 calculates the position vector of the feature point 210 in real space. CAM P and the position vector of the feature point 210 on the virtual projection plane 123 CAM p and the camera coordinate system Σ CAM Then, the mathematical formula (10) is generated, which indicates that the points are on the same line (step S8).

[0113] Next, the calculation unit 115 generates the mathematical formula (11) which indicates that the feature point 210 exists on a horizontal plane parallel to the horizontal plane 122 relative to the image acquisition unit 111 (step S9).

[0114] Next, the calculation unit 115 calculates the position of the feature point 210 in real space using the formulas (5), (10), and (11) (step S10).

[0115] Thereafter, the calculation unit 115 determines whether or not calculation of the positions has been completed for all of the feature points 211, 212, 221, and 222 (step S11). If there are any points among the feature points 211, 212, 221, and 222 for which calculation of the positions has not been completed (step S11: No), the calculation unit 115 returns to step S5.

[0116] On the other hand, if the calculation of the positions of all of the feature points 211, 212, 221, and 222 has been completed (step S11: Yes), the calculation unit 115 calculates the position of the feature points 211, 212, 221, and 222 from the GPS in the camera coordinate system Σ CAM Then, the calculation unit 115 acquires the position information of the origin of the camera coordinate system Σ CAMThe position, dimensions and direction of the ship are calculated from the position information of the origin and the position information of each of the characteristic points 211, 212, 221 and 222 (step S12).

[0117] The communication unit 116 outputs the information on the position, dimensions and direction of the ship acquired from the calculation unit 115 to the terminal device 3 (step S13).

[0118] As described above, the ship information measuring device 100 extracts characteristic points from image data of the ship 21, calculates the positions of the extracted characteristic points, and uses the calculated positions to generate and notify ship information including the position, dimensions, and heading of the ship 21. This enables the ship information measuring device 100 to easily and accurately measure ship information.

[0119] Specifically, the ship information measuring device 100 processes the ship image acquired by the image acquisition unit 111 in the image recognition unit 114, thereby making it possible to accurately extract feature points 211, 212, 221, and 222 relating to the position, dimensions, and orientation of the ship 21 displayed on the screen. The ship information measuring device 100 can automatically extract the feature points 211, 212, 221, and 222 without relying on human intervention, and can easily extract the feature points 211, 212, 221, and 222.

[0120] Furthermore, the ship information measuring device 100 calculates the position, dimensions, and orientation of the ship using the measured distance, roll angle, and pitch angle of the image acquisition unit 111, as well as position information of characteristic points and the condition that the target ship is on a horizontal plane. In this way, the ship information measuring device 100 can perform measurements with a simple configuration without having to separately measure the measurement position of the laser and the position of the ship 21, making it possible to easily obtain ship information.

[0121] Next, the evaluation results of the image recognition processing by the image recognition unit 114 of the ship information measuring device 100 will be explained. Figure 14 is a diagram for explaining the calculation of the measurement error. Here, the evaluation was performed based on the estimation accuracy of the ship's length using the estimation errors err1 and err2 shown in Figure 14. The estimation error err1 is the error between the correct bow position and the estimated bow position. Furthermore, the estimation error err2 is the error between the correct stern position and the estimated stern of the bow. Then, ((err1) 2 +(err2) 2 ) 1 / 2 We used the sum of squares estimation error expressed as:

[0122] FIG. 15 shows the results of accuracy measurements. FIG. 15 shows the learning results for classification into 3, 5, and 6 classes. As shown in FIG. 15, by performing machine learning a sufficient number of times using a sufficient number of training data, it is possible to measure the ship's length and width with an error of about 1 to 2 pixels in the top 90% of evaluation data. According to these measurement results, a sufficient number of training data pieces is 600 to 700 or more, and a sufficient number of learning epochs is about 200 to 400. Therefore, under similar conditions to those of these measurement results, a similar level of measurement accuracy can be expected. It is preferable that the test data be of the same type as the training data. Furthermore, the estimation accuracy tends to improve as the number of training data pieces increases.

[0123] For example, the position, dimensions, and orientation of the ship 21 measured by the ship information measuring device 100 mounted on the unmanned aerial vehicle 1 can be used for coastal security, port safety monitoring, and marine accident response. For example, the information on the position, dimensions, and orientation of the ship 21 obtained by the ship information measuring device 100 can be used to identify the ship 21 and estimate its movements, thereby predicting the ship 21's route and strengthening port safety monitoring. Furthermore, ships are typically equipped with an AIS (Automatic Identification System) to broadcast their own ship information. Therefore, the position, dimensions, and orientation of the ship 21 measured by the ship information measuring device 100 usually match the ship information from the AIS. In contrast, a suspicious ship may not broadcast ship information from the AIS or may output falsified information. In such cases, information that does not match the position, dimensions, and orientation of the ship 21 measured by the ship information measuring device 100 is broadcast. Therefore, the position, dimensions, and orientation of the ship 21 measured by the ship information measuring device 100 can be used to detect the suspicious ship.

[0124] (Hardware configuration) Fig. 16 is a diagram showing the hardware configuration of the computer. Next, an example of the hardware configuration for realizing each function of the computer 104 installed in the vessel information measuring device 100 will be described with reference to Fig. 16.

[0125] 16, the computer 104 includes, for example, a CPU (Central Processing Unit) 91, a memory 92, a hard disk 93, and a network interface 94. The CPU 91 is connected to the memory 92, the hard disk 93, and the network interface 94 via a bus.

[0126] The network interface 94 is an interface for communication between the computer 104 and an external device. The network interface 94 realizes the function of the communication unit 116 illustrated in Fig. 5. For example, the network interface 94 relays communication between the terminal device 3 and the CPU 91 via an antenna mounted on the unmanned aerial vehicle 1.

[0127] The hard disk 93 is an auxiliary storage device and stores various programs including a program for realizing the functions of the image recognition unit 114 and the calculation unit 115 illustrated in FIG.

[0128] The memory 92 is a main storage device and may be, for example, a dynamic random access memory (DRAM).

[0129] The CPU 91 reads various programs from the hard disk 93, expands them into the memory 92, and executes them. In this way, the CPU 91 realizes the functions of the image recognition unit 114 and the calculation unit 115 illustrated in Fig. 5. Note that a GPU or an embedded processor suitable for machine learning can also be added to realize the functions of the image recognition unit 114 and the calculation unit 115 and to create machine learning programs to be used therefor. [Explanation of symbols]

[0130] 1 drone 3 Terminal Devices 10 2-axis gimbal 11, 12 aperture 21, 22 ships 100 Ship information measurement device 101 Camera 102 Distance meter 103 Inclinometer 104 Computer 111 Image acquisition unit 112 Distance measurement unit 113 Inclination measurement section 114 Image Recognition Unit 115 Calculation Unit 116 Communications Department

Claims

1. Photographing a predetermined vessel with an image capture device to obtain a photographed image of the predetermined vessel; extracting feature points of the predetermined ship in the photographed image based on an output obtained by inputting the acquired photographed image into a machine learning model that extracts feature points of the ship from the image; Acquire the distance from the image acquisition device to the horizontal plane where the specified vessel is located; acquiring an inclination angle of the image capture device relative to the horizontal plane; Identifying ship information of the predetermined ship based on the position of the feature point in the photographed image, the distance, and the tilt angle. A ship information measurement program that causes a computer to execute processing.

2. The ship information measurement program described in claim 1, characterized in that the computer is further made to execute a process of training the machine learning model to extract points corresponding to the definition of ship dimensions as the feature points.

3. In the process of extracting the feature points, obtaining a probability distribution of the feature points in the captured image output from the machine learning model; Identifying the positions of the feature points based on the probability distribution 2. The ship information measuring program according to claim 1, wherein the program causes the computer to execute processing.

4. In the specific processing of the ship information, converting the positions of the feature points in the captured image into positions in a first coordinate system based on the capturing direction of the image acquisition device; transforming the position of the feature point in the first coordinate system into a position in a second coordinate system based on a plane parallel to the horizontal plane; Identifying ship information of the predetermined ship based on the position of the characteristic point in the second coordinate system, the distance, and the tilt angle.

2. The ship information measuring program according to claim 1, wherein the program causes the computer to execute processing.

5. The vessel information measuring device Photographing a predetermined vessel with an image capture device to obtain a photographed image of the predetermined vessel; extracting feature points of the predetermined ship in the photographed image based on an output obtained by inputting the acquired photographed image into a machine learning model that extracts feature points of the ship from the image; Acquire the distance from the image acquisition device to the horizontal plane where the specified vessel is located; acquiring an inclination angle of the image capture device relative to the horizontal plane; Identifying ship information of the predetermined ship based on the position of the feature point in the photographed image, the distance, and the tilt angle. A ship information measuring method characterized by performing processing.

6. an image acquisition unit that acquires an image of a predetermined ship by using an image acquisition device; an image recognition unit that extracts feature points of the specified ship from the photographed image based on an output obtained by inputting the acquired photographed image into a machine learning model that extracts feature points of the ship from the image; a distance measurement unit that acquires the distance from the image acquisition device to a horizontal plane on which the specified vessel is located; an inclination measurement unit that acquires an inclination angle of the image acquisition device with respect to the horizontal plane; a calculation unit that identifies ship information of the specified ship based on the positions of the feature points in the captured image extracted by the image recognition unit, the distance acquired by the distance measurement unit, and the tilt angle acquired by the tilt measurement unit; and A ship information measuring device comprising:

Images