Navigation assistance device, navigation assistance data generation method, and navigation assistance data generation program

Abstract

[Problem] To suppress any deviation of the position of a marker on video due to movement of a hull. [Solution] This navigation assistance device comprises a video data generation unit, a designation unit, a coordinate calculation unit, a position measurement unit, a movement vector setting unit, and a correction unit. The video data generation unit generates surrounding video data of the hull. The designation unit designates a designated position on video based on the surrounding video data. The coordinate calculation unit calculates position coordinates within a virtual space corresponding to the designated position on the video. The position measurement unit is installed on the hull and measures an actual movement vector of the hull in an Earth coordinate system. The movement vector setting unit sets a virtual movement vector within the virtual space from the actual movement vector. The correction unit corrects the designated coordinates within the virtual space corresponding to the designated position on the basis of the virtual movement vector.

Description

Navigation support device, navigation support data generation method, and navigation support data generation program

[0001] The present invention relates to AR technology for ships.

[0002] Patent Document 1 describes a system that displays boundary line display data of a travel route superimposed on photographed image data of the area ahead of a ship.

[0003] Japanese Patent Application Laid-Open No. 2021-144751

[0004] However, there is a problem in that if the ship moves after placing markers at predetermined positions on the image around the ship, the positions of the markers on the image will shift.

[0005] Therefore, an object of the present invention is to suppress deviation of the position of a marker on an image due to movement of the ship's hull.

[0006] A navigation assistance device according to one embodiment of the present invention comprises a video data generation unit, a designation unit, a coordinate calculation unit, a positioning unit, a movement vector setting unit, and a correction unit. The video data generation unit generates video data of the hull's surroundings. The designation unit designates a designated position on the video based on the surrounding video data. The coordinate calculation unit calculates position coordinates in virtual space corresponding to the designated position on the video. The positioning unit is installed on the hull and measures the hull's actual movement vector in the Earth coordinate system. The movement vector setting unit sets a virtual movement vector in virtual space from the actual movement vector. The correction unit corrects the designated coordinates in virtual space corresponding to the designated position based on the virtual movement vector.

[0007] With this configuration, the designated coordinates in the virtual space are corrected in accordance with the movement of the hull. As a result, even if a marker is placed at a designated position, the position of the marker on the image is prevented from shifting due to the movement of the hull.

[0008] In a navigation assistance device according to one embodiment of the present invention, the correction unit corrects the designated position by correcting the orientation vector from the reference position of the surrounding image data in the virtual space to the designated position with the virtual movement vector.

[0009] A navigation support device according to an embodiment of the present invention includes a reference point setting unit that sets a reference position in a virtual space based on the actual position of a virtual camera set in a global coordinate system, and a correction unit that corrects the coordinates of a designated position in the virtual space.

[0010] A navigation assistance device according to an embodiment of the present invention includes a plurality of cameras that capture images of different areas around a ship hull, and a reference position in virtual space is set based on the positions of the plurality of cameras.

[0011] A navigation assistance device according to one embodiment of the present invention includes an AR data generation unit that generates AR data including marker data that indicates a specified position in a coordinate system of a virtual space, and a synthesis unit that synthesizes the AR data with surrounding image data.

[0012] The navigation support device according to one embodiment of the present invention includes a distance calculation unit that calculates the actual distance between the designated position in the Earth coordinate system of the designated position and the hull, and an AR data generation unit that generates AR data in which text data of the actual distance is placed near the marker data.

[0013] In a navigation assistance device according to one embodiment of the present invention, the AR data generation unit deletes the designated position if the actual distance is equal to or greater than the display deletion threshold.

[0014] FIG. 1 is a functional block diagram showing an example of the configuration of a navigation support system including a navigation support device according to an embodiment of the present invention. FIGS. 2A and 2B are diagrams showing an example of the arrangement of cameras relative to a hull. FIG. 3 is a flowchart showing an example of a method for generating surrounding image data. FIGS. 4A and 4B are flowcharts showing a specific example of stitching together image data. FIG. 5 is a diagram showing an example of setting the outer shape of a hull used to stitch together image data. FIG. 6 is a diagram showing an example of setting a boundary for stitching together image data. FIG. 7A is a diagram showing an example of a surrounding image according to the present invention, and FIG. 7B is a diagram showing an example of a surrounding image (comparative surrounding image) according to a comparative example (conventional configuration). FIG. 8 is a flowchart showing an example of a method for calculating the shortest distance between a specified position and a hull and a method for displaying the shortest distance. FIG. 9 is a flowchart showing an example of a specific method for calculating the shortest distance between a specified position and a hull. FIG. 10 is a diagram for explaining the concept of calculating the shortest distance. FIGS. 11A and 11B are diagrams showing an example of an image based on synthesized image data including the shortest distance. FIG. 12 is a diagram showing an example of an image based on composite video data when multiple specified positions are specified. FIG. 13 is a flowchart showing an example of a method for correcting specified positions. FIG. 14 is a diagram for explaining the concept of correcting specified positions. FIG. 15(A) is a composite image before the hull moves, FIG. 15(B) is a composite image after the hull moves when the configuration of the present application is used, and FIG. 15(C) is a composite image after the hull moves when the configuration of the present application is not used (in the case of a conventional configuration). FIG. 16 is a flowchart showing an example of a method for generating a grid line image. FIG. 17 is a flowchart showing an example of a specific method for generating grid lines. FIG. 18 is a diagram for explaining the concept of setting grid lines. FIG. 19 is a diagram showing an example of a composite image including grid lines. FIG. 20 is a diagram showing an example of a composite image including grid lines. FIG. 21 is a diagram showing an example of a composite image.

[0015] A navigation support technology according to an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a functional block diagram showing an example of the configuration of a navigation support system including a navigation support device according to an embodiment of the present invention.

[0016] The navigation assistance system includes a navigation assistance device 10, a bow camera CAMF, a port camera CAML, a starboard camera CAMR, a stern camera CAMB, and a forward monitoring camera CAMS.

[0017] The navigation assistance device 10 includes a designation unit 21, a positioning unit 22, a target detection unit 23, a landmark classification unit 24, a direction measurement unit 25, an attitude measurement unit 26, a coordinate calculation unit 31, a distance calculation unit 32, an AR data generation unit 33, a movement vector setting unit 34, a correction unit 35, a video data generation unit 40, a synthesis unit 50, and a reference point setting unit 100. Each of these functional units constituting the navigation assistance device 10 is realized by processing by a control unit. The control unit may be configured as either hardware or software. In other words, a navigation assistance program is stored in the navigation assistance device 10, etc., and the control unit executes the navigation assistance program to realize the functions of each of the functional units described above.

[0018] The configuration shown in Figure 1 is the configuration required when the navigation assistance device 10 executes all of the multiple functions described below, and the minimum configuration when executing each function individually requires only a portion of the configuration shown in Figure 1 depending on the function.

[0019] 2A and 2B are diagrams showing an example of the arrangement of cameras on the hull, where Fig. 2A is a plan view and Fig. 2B is a side view seen from the port side.

[0020] The hull 90 includes a bow 911, a stern 912, a port side 921, a starboard side 922, a deck 900, and a bridge 93. The bridge 93 is located at a position closer to the stern 912 than to the bow 911 of the hull 90, for example, near the stern 912, at a predetermined height.

[0021] The bow camera CAMF, the port camera CAML, the starboard camera CAMR, the stern camera CAMB, and the forward monitoring camera CAMS are connected to the video data generation unit 40.

[0022] The bow camera CAMF, port camera CAML, starboard camera CAMR, and stern camera CAMB are all fish cameras. For example, the horizontal and vertical viewing angles of the bow camera CAMF, port camera CAML, starboard camera CAMR, and stern camera CAMB are 180° or more, specifically, approximately 240°. Note that the viewing angles are just an example, and any angle greater than 180° will suffice.

[0023] The bow camera CAMF is disposed on the deck 900, forward of the bridge 93, near the bow 911. The bow camera CAMF captures an image of the bow direction of the hull and generates bow video data PICF. The bow camera CAMF outputs the bow video data PICF to the video data generator 40.

[0024] The stern camera CAMB is disposed on the deck 900 behind the bridge 93 and near the stern 912. The stern camera CAMB captures an image of the stern direction of the hull and generates stern image data PICB. The stern camera CAMB outputs the stern image data PICB to the image data generator 40.

[0025] The port camera CAML is disposed at a high position on the bridge 93 (for example, at the same height as the wheelhouse). The port camera CAML captures images of the port side of the hull and generates port video data PICL. The port camera CAML outputs the port video data PICL to the video data generator 40.

[0026] The starboard camera CAMR is disposed at a high position on the bridge 93 (for example, at the same height as the wheelhouse). The starboard camera CAMR captures images of the starboard side of the hull and generates starboard video data PICR. The starboard camera CAMR outputs the starboard video data PICR to the video data generator 40.

[0027] The port camera CAML and starboard camera CAMR are positioned higher than the bow camera CAMF and the stern camera CAMB, and therefore the port camera CAML and starboard camera CAMR can capture images of the water surface near the bow 911 and the water surface near the stern 912. In particular, the bow 911 is farther away from the bridge 93 in the horizontal direction, and therefore the water surface near the bow 911 can be captured more reliably.

[0028] The forward monitoring camera CAMS captures high-definition images with a narrower viewing angle than the bow camera CAMF.

[0029] The forward monitoring camera CAMS is disposed at a high position in front of the bridge 93. The forward monitoring camera CAMS captures an image of the area ahead, including the bow 911, and generates forward monitoring video data. The forward monitoring camera CAMS outputs the forward monitoring video data to the video data generator 40.

[0030] A virtual space is set for this hull 90. A reference point of the virtual space is set by the reference point setting unit 100. The reference point of the virtual space is the position of a virtual camera, and is set, for example, within the bridge 93. More specifically, when the hull 90 is viewed from above (in still water), the reference point of the virtual space (the position of the virtual camera) is set between the bow camera CAMF and the stern camera CAMB in the bow-stern direction, and between the port camera CAML and the starboard camera CAMR in the port and starboard directions. Furthermore, it is preferable that the reference point of the virtual space (the position of the virtual camera) be set at the installation position of the forward monitoring camera CAMS.

[0031] The virtual space is set, for example, in accordance with the Earth coordinate system. The virtual space is set as a three-axis coordinate system, with the latitude direction, longitude direction, and vertical direction of the Earth coordinate system as one coordinate axis. This makes it easy to set the position coordinates, speed, and direction of other ships obtained by AIS or the like in the virtual space.

[0032] To simplify calculations, the virtual space is constructed using two horizontal axes representing a local tangent plane on the Earth (a large ellipsoid) and a vertical axis relative to the tangent plane. In this virtual space, the latitude and longitude difference between the ship's past and current positions used in ranging can be treated as a value on a virtual space plane, obtained, for example, by a linear approximation formula using Hubeny's formula. In other words, because the Earth is extremely large compared to the range observable by the navigation support device 10 (a range measurable with a radius of about 1 nm), even using this approximation, sufficient accuracy for practical use can be achieved in the virtual space.

[0033] Furthermore, the coordinate system of the virtual space and the hull coordinate system can be converted based on the orientation of the hull 90 measured by the orientation measurement unit 25 .

[0034] The video data generation unit 40 connects the bow video data PICF, port video data PICL, starboard video data PICR, and stern video data PICB together to generate surrounding video data showing the area around the hull 90. In this case, the surrounding video data preferably includes video of the bow 911, stern 912, port side 921, and starboard side 922 of the hull 90. The video data generation unit 40 outputs the surrounding video data to the synthesis unit 50.

[0035] The video data generator 40 generates the surrounding video image data using a reference point in the virtual space as the reference point of the video. Therefore, the positional relationship between the virtual space coordinate system and each pixel of the video based on the surrounding video data is uniquely set. In other words, the position of each pixel of the video can be set by its position coordinate in the virtual space coordinate system.

[0036] At this time, the video data generation unit 40 can match the bow direction in the virtual space with the bow direction of the surrounding video data based on the direction measured by the direction measurement unit 25. As a result, the coordinates of each pixel of the video from the surrounding video data are set with high precision in the coordinate system of the virtual space.

[0037] Furthermore, the video data generation unit 40 can switch between the surrounding video data and the forward monitoring video data and output them to the synthesis unit 50. Note that each pixel of the video of the forward monitoring video data can also be set using position coordinates in the virtual space coordinate system. As with the surrounding video data, the coordinates of each pixel of the video of the forward monitoring video data are also set with high precision in the virtual space coordinate system based on the direction measured by the direction measurement unit 25.

[0038] The designation unit 21 is composed of, for example, a pointer displayed on a display (not shown). The display displays the above-mentioned surrounding image data or forward monitoring image data, and the AR data (described later), along with the pointer. The designation unit 21 receives designation of a pixel when a specific position on the image displayed on the display is selected with the pointer through an external input operation using a mouse or the like, and sets the pixel as the designated position on the image. The designation unit 21 outputs the designated position (pixel) on the image to the coordinate calculation unit 31.

[0039] The coordinate calculation unit 31 calculates the coordinates of the designated position in virtual space. Since the relationship between the pixels of the image and the coordinate system of the virtual space is set in advance as described above, the coordinate calculation unit 31 can calculate the coordinates of the designated position in virtual space from the designated position (pixel) on the image. The coordinate calculation unit 31 outputs the position coordinates of the designated position in virtual space to the distance calculation unit 32.

[0040] The external shape of the hull 90 in virtual space is stored in the distance calculation unit 32. The distance calculation unit 32 calculates the shortest distance between the position coordinates of the specified position in virtual space and the external shape of the hull. The distance calculation unit 32 outputs the shortest distance to the AR data generation unit 33.

[0041] The AR data generation unit 33 generates AR data including text data of the shortest distance, marker data indicating a specified position, etc. The AR data generation unit 33 can include other auxiliary graphics, such as grid image data, as will be described later, in the AR data.

[0042] Furthermore, the AR data generating unit 33 can include various types of navigation support information acquired by the AIS or the like as described above in the AR data.

[0043] The AR data generating unit 33 outputs the AR data to the combining unit 50 .

[0044] When surrounding image data is input, the synthesis unit 50 synthesizes the surrounding image data with the AR data to generate AR-added image data. When forward monitoring image data is input, the synthesis unit 50 synthesizes the forward monitoring data with the AR data. In this case, the synthesis unit 50 synthesizes the surrounding image data with the AR data based on a reference point in the virtual space. The synthesis unit 50 outputs the synthesized image data to a display (not shown) or the like.

[0045] The positioning unit 22, movement vector setting unit 34, and correction unit 35 are used, for example, to update and correct the coordinates of the designated position when the hull 90 is moving. In addition, the position measured by the positioning unit 22 is used to calculate the position coordinates in virtual space of the target detected by the target detection unit 23.

[0046] The positioning unit 22 receives positioning signals from a positioning system such as GPS via an external positioning antenna and measures the position and velocity of the hull 90 in the Earth's coordinate system. The positioning unit 22 can also receive sensor signals from an external inertial sensor and measure the velocity, acceleration, and angular velocity of the hull 90 in the hull's coordinate system. Furthermore, the positioning unit 22 can measure the position, velocity, etc. by combining measurements using the positioning signals and measurements using the sensor signals.

[0047] Furthermore, the positioning unit 22 calculates an actual movement vector in the Earth coordinate system from the time when the designated position is specified to the time when the designated position is to be updated, based on the measured position and speed. Note that the positioning unit 22 may calculate the actual movement vector in the ship's coordinate system and convert it into an actual movement vector in the Earth coordinate system based on the direction.

[0048] The movement vector setting unit 34 calculates the inverse vector of the actual movement vector and sets the calculated inverse vector as the virtual movement vector. Because the Earth coordinate system and the coordinate system of the virtual space are the same, the virtual movement vector representing the movement of the designated position in the virtual space can be set by the inverse vector of the actual movement vector. The movement vector setting unit 34 outputs the virtual movement vector to the correction unit 35.

[0049] The correction unit 35 corrects the coordinates of the designated position in the virtual space using the virtual movement vector. The correction unit 35 outputs the virtual movement vector to the coordinate calculation unit 31, and the coordinate calculation unit 31 outputs the corrected coordinates of the designated position to the distance calculation unit 32.

[0050] The target detection unit 23 and the target identification unit 24 are used, for example, when the position of a detected target is set as a designated position, unlike when a user designates a designated position.

[0051] The target detection unit 23 is configured with radar, LiDAR, etc. The target detection unit 23 detects targets around the hull 90 and outputs the positions of the detected targets.

[0052] The object marker classifying unit 24 identifies the type of the detected object and outputs the identification result and the position of the detected object to the designation unit 21.

[0053] The designation unit 21 sets the position of the detected target to a designated position. At this time, the position coordinates of the designated position are set in the hull coordinate system. The coordinate calculation unit 31 calculates the position coordinates of the designated position (specific position) in virtual space using the position coordinates of the hull 90 in the Earth coordinate system measured by the positioning unit 22 and the designated position (corresponding to the specific position) in the hull coordinate system input from the designation unit 21. The coordinate calculation unit 31 outputs the calculated position coordinates of the designated position (specific position) in virtual space to the distance calculation unit 32.

[0054] The direction measurement unit 25 receives an output signal from an external direction sensor as input and measures the direction of the hull 90 (for example, the bow direction).

[0055] The attitude measurement unit 26 receives an output signal from an external attitude sensor and measures the attitude of the hull 90. When the hull 90 sways, the attitude measurement unit 26 is used for calculating position coordinates in the coordinate calculation unit 31 and for correcting sway when the compositing unit 50 combines the surrounding video data and AR data.

[0056] For example, the coordinate calculation unit 31 uses the orientation measured by the orientation measurement unit 36 ​​to correct the position coordinates in the virtual space of a specified position (pixel) on the image based on the surrounding image data.

[0057] Furthermore, the synthesis unit 50 corrects the AR-added video data using the attitude measured by the attitude measurement unit 36 ​​so that it appears as if the hull 90 is rocking.

[0058] By appropriately using the above configuration, the navigation support device 10 performs various processes, for example, as shown below.

[0059] [Generation of Surrounding Video Data] Fig. 3 is a flowchart showing an example of a method for generating surrounding video data. Fig. 4(A) and Fig. 4(B) are flowcharts showing a specific example of joining video data. Fig. 4(A) shows a case where video data for the bow, stern, port side, and starboard side are joined together, and Fig. 4(B) shows a case where video data for the bow, port side, and starboard side are joined together.

[0060] Fig. 5 is a diagram showing an example of setting the outer shape of the hull used to join the video data, and Fig. 6 is a diagram showing an example of setting the boundary for joining the video data.

[0061] 7A is a diagram showing an example of a surrounding image according to the configuration of the present invention, and FIG. 7B is a diagram showing an example of a surrounding image (comparative surrounding image) according to a comparative example (conventional configuration). Note that although the image of the stern side is not shown in FIG. 7A and FIG. 7B, the image of the stern side has the same configuration as the image of the bow side.

[0062] In the following, an example of joining together video data for the bow, stern, port side, and starboard side will be described with reference to FIGS. 3, 4(A), 5, and 6.

[0063] The video data generating unit 40 receives the bow video data PICF, the port video data PICL, the starboard video data PICR, and the stern video data PICB.

[0064] The bow image data PICF is image data captured by the bow camera CAMF, and represents an image including the water surface in the bow direction (bow 911 side) of the hull 90. The bow image data PICF includes images of the bow 911, port side 921, and starboard side 922 of the hull 90.

[0065] The port side image data PICL is image data captured by the port side camera CAML, and is data representing an image including the water surface in the port direction (port side 921 side) of the hull 90. The port side image data PICL includes images of the port side 921, bow 911, and stern 912 of the hull 90.

[0066] The starboard image data PICR is image data captured by the starboard camera CAMR, and represents an image including the water surface in the starboard direction (starboard side 922 side) of the hull 90. The starboard image data PICR includes images of the starboard side 922, bow 911, and stern 912 of the hull 90.

[0067] The stern image data PICB is image data captured by the stern camera CAMB, and represents an image including the water surface in the stern direction (stern 912 side) of the hull 90. The stern image data PICB includes images of the stern 912, port side 921, and starboard side 922 of the hull 90.

[0068] The reference point setting unit 100 sets a reference point PSar (see FIG. 5) that joins the bow image data PICF, the port image data PICL, the starboard image data PICR, and the stern image data PICB together. The reference point setting unit 100 sets the position coordinates of the virtual camera, i.e., the reference position coordinates (position coordinates of the origin) in the virtual space, to the reference point PSar (S11).

[0069] The image data generation unit 40 uses the external shape 80 of the hull 90 (see Figure 5) and the reference point PSar to connect the bow image data PICF, port image data PICL, starboard image data PICR, and stern image data PICB to generate surrounding image data (S12).

[0070] More specifically, the video data is spliced ​​together as follows.

[0071] The video data generating unit 40 stores the external shape 80 of the hull 90 using position coordinates in virtual space.

[0072] Specifically, the outer shape 80 of the hull 90 is composed of a port line 821, a starboard line 822, a bow line 811, and a stern line 812. The port line 821 is a straight line (line segment) that runs along and overlaps the port side 921 of the hull 90 in a plan view of the hull 90. The starboard line 822 is a straight line (line segment) that runs along and overlaps the starboard side 922 of the hull 90 in a plan view of the hull 90. The bow line 811 is a straight line (line segment) that passes through the bow 911 and is perpendicular to the port line 821 and starboard line 822 in a plan view of the hull 90. The stern line 812 is a straight line (line segment) that passes through the stern 912 and is perpendicular to the port line 821 and the starboard line 822 when the hull 90 is viewed from above. Therefore, the outer shape 80 of the hull 90 is a rectangle in which the hull 90 is inscribed when the hull 90 is viewed from above.

[0073] The external shape 80 is stored by at least the position coordinates of the intersection P11 (see FIGS. 5 and 6) between the port line 821 and the bow line 811, the position coordinates of the intersection P12 (see FIGS. 5 and 6) between the starboard line 822 and the bow line 811, the position coordinates of the intersection P21 (see FIGS. 5 and 6) between the port line 821 and the stern line 812, and the position coordinates of the intersection P22 (see FIGS. 5 and 6) between the starboard line 822 and the stern line 812. In other words, the external shape 80 is stored by at least the position coordinates of the corners of a rectangle.

[0074] The video data generation unit 40 sets a plurality of connection lines connecting the reference point PSar to each of the corners of the rectangle (intersection points P11, P12, P21, and P22). The video data generation unit 40 sets the plurality of connection lines along a plurality of boundary lines of the video data to be joined together (S121).

[0075] More specifically, the video data generation unit 40 sets a connecting line (straight line) connecting the reference point PSar and the intersection point P11 to the boundary line BL11 (see FIG. 6). The video data generation unit 40 sets a connecting line (straight line) connecting the reference point PSar and the intersection point P12 to the boundary line BL12 (see FIG. 6). The video data generation unit 40 sets a connecting line (straight line) connecting the reference point PSar and the intersection point P21 to the boundary line BL21 (see FIG. 6). The video data generation unit 40 sets a connecting line (straight line) connecting the reference point PSar and the intersection point P22 to the boundary line BL22 (see FIG. 6).

[0076] The video data generating unit 40 sets a boundary line in the bow direction based on the bow-side inscribed circle of the outer shape (S122).

[0077] More specifically, the video data generation unit 40 calculates a circle (bow-side inscribed circle) that is centered at the reference point PSar and tangent to the bow line 811. The video data generation unit 40 detects the intersections of the boundary lines BL11 and BL12 with the bow-side inscribed circle. The video data generation unit 40 extracts an arc that connects the boundary lines BL11 and BL12 on the bow 911 side of the bow-side inscribed circle, and sets this arc as the boundary line BL31 in the bow direction.

[0078] The video data generating unit 40 sets a boundary line in the stern direction based on the stern-side inscribed circle of the outer shape (S123).

[0079] More specifically, the video data generating unit 40 calculates a circle (stern-side inscribed circle) that is centered at the reference point PSar and tangent to the stern line 812. The video data generating unit 40 detects the intersections of the boundary lines BL21 and BL22 with the stern-side inscribed circle. The video data generating unit 40 extracts an arc that connects the boundary lines BL21 and BL22 on the stern 912 side of the stern-side inscribed circle, and sets this arc as the boundary line BL32 in the stern direction.

[0080] The video data generating unit 40 extracts portions of each video data based on the boundary lines (S124).

[0081] More specifically, the video data generation unit 40 partially extracts the area between boundary lines BL11 and BL21 from the port-side video data PICL. The video data generation unit 40 partially extracts the area between boundary lines BL12 and BL22 from the starboard-side video data PICR.

[0082] The video data generation unit 40 partially extracts the area outside the boundary line BL31 (the far side from the hull 90 with the bow 911 as the reference) between the boundary lines BL11 and BL12 from the bow video data PICF. The video data generation unit 40 partially extracts the area outside the boundary line BL32 (the far side from the hull 90 with the stern 912 as the reference) between the boundary lines BL21 and BL22 from the stern video data PICB.

[0083] The video data generating unit 40 connects the extracted video data based on the boundary lines to generate surrounding video data (S125).

[0084] More specifically, the video data generation unit 40 splices together the bow video data PICF and the port video data PICL based on boundary line BL11. The video data generation unit 40 splices together the bow video data PICF and the starboard video data PICR based on boundary line BL12. The video data generation unit 40 splices together the stern video data PICB and the port video data PICL based on boundary line BL21. The video data generation unit 40 splices together the stern video data PICB and the starboard video data PICR based on boundary line BL22.

[0085] Through this processing, the video data generator 40 can generate surrounding video data that shows the water surface all around the hull 90, as shown in Figure 7(A). Therefore, the navigation assistance device 10 can prevent loss of surrounding video data.

[0086] On the other hand, in the comparative example (conventional configuration), as shown in FIG. 7B, an image gap occurs at the bow 911 of the hull 90.

[0087] For example, in a vessel with a high draft, if the bow camera CAMF is placed on the deck 900, it is difficult to capture an image of the water surface near the bow 911.

[0088] However, the navigation assistance device 10 uses images including the bow 911 captured by the port camera CAML and the starboard camera CAMR and stitches them together as described above. This allows the navigation assistance device 10 to generate surrounding video data that shows the water surface near the bow 911. In this case, as shown in FIG. 7A , the surrounding video data includes images of the bow 911, the port side 921, and the starboard side 922. As a result, the surrounding video data includes images of the water surface, the bow 911, the port side 921, and the starboard side 922. The user can clearly understand that the video generated by the surrounding video data is a surrounding video near the hull 90. For example, the user can easily recognize the situation near the hull 90 (e.g., other nearby ships) as shown in FIG. 7A .

[0089] Furthermore, in the case of a large cargo ship or the like with a long hull 90 and a bridge 93 located close to the stern, the portion of the hull 90 between the bow 911 and the bridge 93 does not exist in the image. However, the image data generator 40 performs stitching using the boundary line BL31 described above. This allows the navigation assistance device 10 to generate surrounding image data that includes an image of the portion of the hull 90 between the bow 911 and the bridge 93. Therefore, the navigation assistance device 10 can provide the user with a natural surrounding image.

[0090] Furthermore, the navigation assistance device 10 sets the positions of the virtual cameras (reference points in the virtual space and reference points for stitching together video data) based on the positions of the bow camera CAMF, the port camera CAML, the starboard camera CAMR, and the stern camera CAMB in a plan view of the hull 90. This allows the synthesis unit 50 to suppress deviations in the synthesis positions of the surrounding video data and the AR data in the synthesized video data.

[0091] Furthermore, the port camera CAML and starboard camera CAMR are positioned vertically higher than the bow camera CAMF and stern camera CAMB. This allows the port camera CAML and starboard camera CAMR to more reliably capture images of the water surface near the bow 911 and the water surface near the stern 912. Therefore, the video data generator 40 can generate surrounding video data that more reliably includes the water surface near the bow 911 and the water surface near the stern 912.

[0092] The port camera CAML, starboard camera CAMR, bow camera CAMF, and stern camera CAMB are all fisheye cameras, which allows the horizontal field of view of each camera to be widened, thereby more reliably eliminating blind spots around the hull 90.

[0093] Furthermore, the video data generation unit 40 performs vibration correction on the surrounding video data based on the attitude measured by the attitude measurement unit 26. As a result, even if the hull 90 vibrates, the video data generation unit 40 can correct the surrounding video so that it does not vibrate. In other words, in reality, it is the hull 90 that vibrates, and not the Earth (the area around the hull 90). Because each camera is installed on the hull 90, the surrounding video data vibrates when the hull 90 vibrates. For this reason, the video data generation unit 40 can generate surrounding video data that does not vibrate by correcting the surrounding video data using the attitude of the hull 90.

[0094] [Calculation of the shortest distance between a designated position (specific position) and the hull] Figure 8 is a flowchart showing an example of a method for calculating the shortest distance between a designated position and the hull and a method for displaying the shortest distance. Figure 9 is a flowchart showing an example of a specific method for calculating the shortest distance between a designated position and the hull. Figure 10 is a diagram for explaining the concept of calculating the shortest distance. Figures 11(A) and 11(B) are diagrams showing an example of an image based on composite video data including the shortest distance. The designated positions differ between Figures 11(A) and 11(B). Figure 12 is a diagram showing an example of an image based on composite video data when multiple designated positions are designated.

[0095] As described above, the video data generation unit 40 generates surrounding video data showing the surroundings of the hull 90 (S21). The synthesis unit 50 synthesizes the surrounding video data with the AR data to generate synthesized video data.

[0096] The designation unit 21 designates a designated position PM (see FIGS. 11A and 11B) on the image based on the surrounding image data (S22). For example, the user operates a pointer placed on the image to click or the like at the designated position PM. The designation unit 21 detects this click position and designates the designated position PM on the image in pixel units. The designation unit 21 outputs the designated position PM to the coordinate calculation unit 31.

[0097] The coordinate calculation unit 31 calculates the coordinates of the designated position PM in the virtual space corresponding to the pixel of the designated position PM on the image (S23). The coordinate calculation unit 31 outputs the coordinates of the designated position PM in the virtual space to the distance calculation unit 32.

[0098] The distance calculation unit 32 calculates the shortest distance DISmin between the designated position PM in the virtual space and the hull 90 (S24). The distance calculation unit 32 outputs the shortest distance DISmin to the AR data generation unit 33.

[0099] More specifically, the distance calculation unit 32 calculates the shortest distance DISmin as follows: At this time, the distance calculation unit 32 calculates the shortest distance DISmin in the coordinate system of the virtual space.

[0100] The distance calculation unit 32 acquires the coordinates (corresponding to hull region coordinates) of multiple corners (intersection points P11, P12, P21, P22) of the external shape of the hull 90 (S241).

[0101] The distance calculation unit 32 calculates the distances between the designated position PM and the multiple corners (intersections P11, P12, P21, and P22) (S242). At this time, the distance calculation unit 32 calculates the distances by projecting the designated position PM and the multiple corners onto a horizontal plane (a two-dimensional plane whose axes are the bow-stern direction and the starboard and port directions).

[0102] The distance calculation unit 32 compares the distances and selects the closest corner (intersection) (S243). For example, in the case of Figure 10, the designated position PM is located on the port side 921 of the hull 90, closer to the bow 911 than to the stern 912. In this case, the distance calculation unit 32 selects the intersection P11.

[0103] The distance calculation unit 32 selects the port line 821 that passes through the selected corner (intersection point) (S244). For example, in the case of Fig. 10, the distance calculation unit 32 selects the port line 821 that passes through the intersection point P11.

[0104] The distance calculation unit 32 calculates the angle θ between the straight line connecting the designated position PM and the selected corner (intersection) and the selected outline line (S245). For example, in the case of Figure 10, the distance calculation unit 32 calculates the angle θ between the straight line connecting the designated position PM and the intersection P11 and the port line 821.

[0105] The distance calculation unit 32 calculates the shortest distance DISmin using known geometric calculations from the distance SLM between the designated position PM and the selected corner (intersection) and the angle θ between them (S246). The shortest point PI on the hull 90, which is the shortest distance DISmin, corresponds to the foot of a perpendicular line drawn from the designated position PM to the hull 90 (see FIG. 10). Therefore, the distance calculation unit 32 can calculate the shortest distance DISmin using known geometric calculations using trigonometric functions, etc.

[0106] If the function of the outline of the ship's ship is known, the distance calculation unit 32 can also calculate the shortest distance DISmin using the formula for the distance between a point and a straight line.

[0107] The shortest distance DISmin may be calculated in the Earth coordinate system. However, this may increase the number of coordinate conversions and other factors that may cause calculation errors. Therefore, it is preferable to calculate the shortest distance DISmin in the virtual space coordinate system.

[0108] The AR data generating unit 33 generates AR data including text data of the shortest distance DISmin (S25).

[0109] More specifically, the AR data generation unit 33 generates text data DT (see FIG. 11) of the shortest distance DISmin. The AR data generation unit 33 places marker data MKR (see FIG. 11) indicating the designated position PM at the position of the designated position PM. The AR data generation unit 33 places the text data DT of the shortest distance DISmin near the marker data MKR.

[0110] By arranging the marker data MKR, the user can easily grasp the designated position PM. Furthermore, by arranging the text data DT near the marker data MKR, the user can easily grasp the shortest distance from the hull 90 to the designated position PM. In particular, since the shortest distance to the port side or starboard side can be easily grasped, the shortest distance between the quay and the hull 90 when docking or departing can be easily grasped.

[0111] The AR data generator 33 outputs the AR data to the synthesizer 50. The synthesizer 50 synthesizes the AR data and the surrounding image data to generate synthesized image data. As a result, as shown in Figures 11(A) and 11(B), the navigation assistance device 10 can provide the user with an image that displays the shortest distance DISmin between the designated position PM and the hull 90. The navigation assistance device 10 can then measure the distance between any position on the image and the hull through intuitive operations and provide the result to the user.

[0112] 11(A) and 11(B), an auxiliary line LNA may be added to the AR data to visually indicate the shortest distance DISmin, allowing the user to easily determine the position on the hull 90 that is closest to the designated position PM.

[0113] 11A and 11B, the designated position PM can be moved. The designated position PM can be moved, for example, by clipping and moving the designated position PM (marker data MKR) on the image. The distance calculation unit 32 recalculates and updates the shortest distance DISmin in accordance with the movement of the designated position PM.

[0114] The navigation assistance device 10 can also use this click to cancel the designated position PM. For example, the navigation assistance device 10 can cancel the designated position PM by detecting another click on the designated position PM.

[0115] 12, the navigation assistance device 10 can specify multiple designated positions PM1, PM2, and PM3 and calculate the shortest distances DISmin1, DISmin2, and DISmin3 for each of them. In this case, the designation unit 21 specifies the multiple designated positions PM1, PM2, and PM3. For example, the multiple designated positions PM1, PM2, and PM3 can be specified by clicking multiple locations on the image.

[0116] The distance calculation unit 32 calculates the shortest distances DISmin1, DISmin2, and DISmin3 for each of the multiple designated positions PM1, PM2, and PM3 using the method described above.

[0117] The AR data generating unit 33 generates text data DT1, DT2, and DT3 of the shortest distance for each of the plurality of designated positions PM1, PM2, and PM3.

[0118] The AR data generation unit 33 places marker data MKR1 indicating the designated position PM1 at the position of the designated position PM1, places text data DT1 of the shortest distance DISmin1 near the marker data MKR1, and places an auxiliary line LNA1. The AR data generation unit 33 places marker data MKR2 indicating the designated position PM2 at the position of the designated position PM2, places text data DT2 of the shortest distance DISmin2 near the marker data MKR2, and places an auxiliary line LNA2. The AR data generation unit 33 places marker data MKR3 indicating the designated position PM3 at the position of the designated position PM3, places text data DT3 of the shortest distance DISmin3 near the marker data MKR3, and places an auxiliary line LNA3.

[0119] With this configuration, the navigation assistance device 10 can simultaneously provide the user with the shortest distances to multiple designated positions in a single composite image.

[0120] The navigation assistance device 10 may be configured to designate the designated position PM only when the distance from the hull 90 to the designated position PM is equal to or less than a threshold distance. In the surrounding image, the length corresponding to one pixel increases as the distance to the hull 90 increases. Therefore, by limiting the range of the designated position PM to a range that is not too far from the hull 90, the navigation assistance device 10 can reduce designation errors of the designated position PM and calculation errors of the shortest distance DISmin.

[0121] In the above case, the user specifies the designated position PM, but the designated position can also be specified by the position of a detected target.

[0122] In this case, the target detection unit 23 detects targets around the hull 90 and their actual positions (positions in the Earth coordinate system). The designation unit 21 sets the actual position of the target as a designated position PM. The distance calculation unit 32 calculates the shortest distance DISmin using this designated position PM.

[0123] The navigation assistance device 10 can designate or cancel the designated position PM based on the distance from the hull 90 to the target.

[0124] In this case, the distance calculation unit 32 calculates the target distance from the hull 90 to the target based on the actual position of the hull 90 and the actual position of the target acquired from the positioning unit 22. When the target distance becomes equal to or less than a predetermined target distance, the designation unit 21 designates the target as a designated position PM based on the actual position of the target. On the other hand, when the target distance becomes greater than the predetermined target viewing distance, the designation unit 21 cancels the designation of the designated position PM based on the actual position of the target.

[0125] The navigation support device 10 can also specify the designated position PM depending on the type of target.

[0126] In this case, the object identification unit 24 identifies the type of target detected by the target detection unit 23. The designation unit 21 sets a designated position PM if the type of target is a quay, another ship, or a buoy. The type of target that can be set as the designated position PM can be set by the user. This allows the navigation assistance device 10 to provide the shortest distance DISmin to the target desired by the user.

[0127] The navigation assistance device 10 updates the calculation of the shortest distance DISmin in accordance with the movement of the hull 90.

[0128] In this case, the coordinate calculation unit 31 acquires the movement result of the hull 90 and corrects the designated position PM. This correction method will be described later in "Correction of Designated Position." The distance calculation unit 32 updates the calculation of the shortest distance DISmin based on the designated position PM corrected in accordance with the movement of the hull 90.

[0129] In this case, the updated shortest distance DISmin is input to the designation unit 21. When the shortest distance DISmin becomes equal to or greater than a predetermined actual distance, the designation unit 21 deletes the designated position, and the distance calculation unit 32 stops calculating the shortest distance DISmin. At this time, when the shortest distance DISmin becomes equal to or greater than the predetermined actual distance, the AR data generation unit 33 stops generating the text data DT for the shortest distance DISmin.

[0130] This allows the navigation assistance device 10 to avoid providing the user with the shortest distance DISmin that has a large error.

[0131] [Correction of Designated Position] Fig. 13 is a flowchart showing an example of a method for correcting a designated position. Fig. 14 is a diagram for explaining the concept of correcting a designated position. In Fig. 14, Xar and Yar indicate coordinate axes of the virtual space coordinate system, and Xb and Yb indicate coordinate axes of the hull coordinate system.

[0132] Figure 15(A) is a composite image before the hull moves, Figure 15(B) is a composite image after the hull moves when the configuration of the present application is used, and Figure 15(C) is a composite image after the hull moves when the configuration of the present application is not used (in the case of a conventional configuration).

[0133] The video data generation unit 40 uses the above-described method to generate video data of the surroundings of the hull 90 (S31). The designation unit 21 uses the above-described method to designate a designated position PM on the video based on the surrounding video data (S32). The coordinate calculation unit 31 uses the above-described method to calculate position coordinates in virtual space corresponding to the designated position on the video (S33).

[0134] The positioning unit 22 measures the actual movement vector Vm (see FIG. 14) of the hull 90 in the Earth coordinate system using the above-mentioned method (S34). Note that the positioning unit 22 may also measure the actual movement vector Vm in the hull coordinate system.

[0135] The movement vector setting unit 34 sets a virtual movement vector in virtual space from the actual movement vector (S35). More specifically, if the actual movement vector is measured in the Earth coordinate system, the movement vector setting unit 34 calculates an inverse vector of the actual movement vector and sets the calculated inverse vector as the virtual movement vector Vc (see FIG. 14). If the actual movement vector is measured in the hull coordinate system, the movement vector setting unit 34 converts the actual movement vector in the hull coordinate system into an actual movement vector in the Earth coordinate system and calculates an inverse vector of the converted actual movement vector to set the virtual movement vector Vc.

[0136] The correction unit 35 corrects the position coordinates (designated coordinates) of the designated position PM in the virtual space based on the virtual movement vector Vc (S36). In other words, the correction unit 35 corrects the orientation vector from the reference point PSar of the surrounding video data in the virtual space to the designated position PM by using the virtual movement vector VC, thereby correcting the designated position PM.

[0137] Since the designated position PM is a position where a pixel on the image is designated, if no correction is made, the position relative to the reference point PSar in the virtual space will not change even if the hull 90 moves. For this reason, movement of the hull 90 results in the false designated position PMfi shown in Figure 14.

[0138] However, the navigation assistance device 10 can correct the false designated position PMfi with the virtual movement vector Vc, as shown in FIG. 14, and continue to set the designated position PM according to the movement of the hull 90.

[0139] With this configuration, the navigation assistance device 10 can change the composite image shown in Figure 15(A) to the composite image shown in Figure 15(B) in accordance with the movement of the hull 90. In other words, when the hull 90 moves, the navigation assistance device 10 can move the position of the designated position PM on the composite image in accordance with this movement. Therefore, the navigation assistance device 10 can suppress deviation of the position of the marker on the image due to movement of the hull 90.

[0140] At this time, the navigation assistance device 10 recalculates the shortest distance DISmin between the designated position PM and the hull 90 due to the movement of the hull 90, and updates and displays the shortest distance DISmin.

[0141] 15C, without the configuration of the present invention, even if the hull 90 moves, the designated position PMfi on the composite image will be the same as the designated position PM before the hull 90 moved (when the designated position PM was specified). In addition, the shortest distance DISmin will not be updated.

[0142] In this way, the navigation assistance device 10 can provide the shortest distance DISmin at each point in time with high accuracy while suppressing undesirable changes in the designated position PM on the image due to movement of the hull 90.

[0143] [Generation of Grid Line Image] Fig. 16 is a flowchart showing an example of a method for generating a grid line image. Fig. 17 is a flowchart showing an example of a specific method for generating grid lines. Fig. 18 is a diagram for explaining the concept of setting grid lines. Figs. 19 and 20 are diagrams showing examples of a composite image including grid lines. Fig. 19 shows a case where the image is composed only of grid lines parallel to the ship's side, and Fig. 20 shows a case where the image is composed of grid lines parallel to the ship's side (first grid lines) and grid lines perpendicular to the ship's side (second grid lines).

[0144] The video data generating unit 40 uses the above-described method to generate video data of the surroundings of the hull 90, including the port side (port side 921 and starboard side 922) of the hull 90 and the water surface beyond the port side (S41).

[0145] The AR data generating unit 33 generates AR data including grid image data having a plurality of first grid lines that are parallel to the port side (port side 921 and starboard side 922) and spaced apart from each other (S42).

[0146] More specifically, for example, in the case of the port side 921, a plurality of first grid lines are set as shown below. Note that the setting method for the starboard side 922 is the same as that for the port side 921, and only the port side 921 will be described.

[0147] 18, the AR data generation unit 33 sets a straight first reference grid line GLS that overlaps the port side 921 of the hull 90 in the virtual space (S421). In other words, the AR data generation unit 33 sets a straight line that passes through the intersection point P11 and the intersection point P21 of the rectangle that represents the outer shape of the hull 90 as the first reference grid line GLS.

[0148] The AR data generator 33 sets a plurality of first grid lines GL1 at equal distances PCH1 on the opposite side to the hull 90 in a direction perpendicular to the first reference grid line GLS (S422). The plurality of first grid lines GL1 are straight lines parallel to the first reference grid line GLS.

[0149] The synthesizing unit 50 synthesizes the AR data and the surrounding image data so as to arrange a plurality of first grid lines GL1 in the area corresponding to the water surface with the port side (port side 921 or starboard side 922) as the reference (S43).

[0150] 19 , the navigation assistance device 10 can provide a composite image in which a plurality of first grid lines GL1L are arranged on the port side 921 of the hull 90 at the same height as the water surface and parallel to the port side 921, and a plurality of first grid lines GL1R are arranged on the starboard side 922 of the hull 90 at the same height as the water surface and parallel to the starboard side 922. This allows the navigation assistance device 10 to generate an image in which the distance to targets such as other ships on the water surface and to quays, etc. can be easily recognized. This allows the user to intuitively grasp the distance to targets such as other ships on the water surface and to quays, etc. on the port side 921 and starboard side 922 of the hull 90.

[0151] The navigation assistance device 10 may also display a first reference grid line GLS1L that overlaps the port side 921 and a first reference grid line GLS1R that overlaps the starboard side 922. This allows the user to easily grasp the positions of the port side 921 and the starboard side 922. In this case, for example, the first reference grid line GLS1L and the first reference grid line GLS1R may be displayed in a manner different from that of the multiple first grid lines GL1L and the multiple first grid lines GL1R. This allows the user to more clearly grasp the positions of the port side 921 and the starboard side 922.

[0152] The spacing PCH1 between the multiple first grid lines GL1 (GL1L, GL1R) can be set by the user. In this case, the spacing between the first grid lines GL1L on the port side 921 and the spacing between the first grid lines GL1R on the starboard side 922 can also be made different.

[0153] The interval PCH1 between the multiple first grid lines GL1 can be set based on the ship speed measured by the positioning unit 22. For example, the interval PCH1 can be narrowed as the ship speed increases, or the interval PCH1 can be narrowed as the ship speed decreases.

[0154] The AR data generator 33 can set the spacing PCH1 between the multiple first reference grid lines GLS based on the actual distance in the Earth coordinate system between a specific position (designated position PM or the position of a detected target) and the ship. In this case, the navigation assistance device 10 is composed of a designation unit or target detection unit 23 and includes an identification unit that identifies a specific position outside the hull 90. The navigation assistance device 10 calculates the actual distance between the specific position in the Earth coordinate system and the ship. Specifically, if the specific position is the position of a target, the target detection unit 23 measures the distance to the target and calculates the actual distance. On the other hand, if the specific position is the designated position PM, the shortest distance DISmin calculated by the above-mentioned method can be used as the actual distance.

[0155] The AR data generator 33 can further set a plurality of second grid lines GLF2 as shown in Fig. 20. Although Fig. 20 shows only the bow 911 side, the second grid lines GLS2 can be set in the same manner on the stern 912 side. Specifically, the plurality of second grid lines GLS2 are set as follows.

[0156] The AR data generation unit 33 sets a second reference grid line GLS2F that is a straight line in the virtual space that is perpendicular to the ship's side and tangent to the bow 911. In other words, the AR data generation unit 33 sets a straight line that passes through the intersection points P11 and P12 of the rectangle that indicates the external shape of the hull 90 as the second reference grid line GLS2F.

[0157] The AR data generator 33 sets a plurality of second grid lines GL2F in a direction perpendicular to the second reference grid line GLS2F and equidistant from the hull 90. The plurality of second grid lines GL2F are straight lines parallel to the second reference grid line GLS2F.

[0158] The synthesis unit 50 synthesizes the grid image data and the surrounding video data so that a plurality of second grid lines GL2F are arranged forward of the bow 911 of the hull 90 based on the bow 911 in the area corresponding to the water surface.

[0159] With this configuration, the navigation assistance device 10 can place grid lines not only on the port side 921 and starboard side 922, but also on the bow 911 and stern 912. This allows the user to intuitively grasp the distances to targets such as other ships on the water surface and to quays and the like around the entire periphery of the hull 90.

[0160] The AR data generator 33 can set the spacing between the first grid lines GL1 and the spacing between the second grid lines GL2 to be the same or different, allowing the navigation assistance device 10 to set the grid lines according to the situation around the hull 90.

[0161] [Example of a composite image obtained by combining the above-described processes] Fig. 21 is a diagram showing an example of a composite image. The composite image shown in Fig. 21 is displayed based on composite image data obtained by combining the surrounding image data generated by the above-described method with AR data including information on the shortest distance DISmin and a plurality of grid lines.

[0162] In this way, by combining the above-mentioned methods, the navigation assistance device 10 can provide an AR composite image according to the information desired by the user. In this case, the navigation assistance device 10 can select the information to be displayed as AR data, generate AR composite image data according to the selection, and provide it to the user. term

[0163] Not necessarily all objects or advantages may be achieved in accordance with any particular embodiment described herein. Thus, for example, one skilled in the art will appreciate that a particular embodiment may be configured to operate to achieve or optimize one or more advantages as taught herein without necessarily achieving other objects or advantages as taught or suggested herein.

[0164] All processes described herein may be embodied and fully automated by software code modules executed by a computing system including one or more computers or processors. The code modules may be stored on any type of non-transitory computer-readable medium or other computer storage device. Some or all of the methods may be embodied in dedicated computer hardware.

[0165] Many other variations beyond those described herein will be apparent from this disclosure. For example, depending on the embodiment, certain operations, events, or functions of any of the algorithms described herein may be performed in a different sequence, added, merged, or omitted entirely (e.g., not all described acts or events are necessary to execute an algorithm). Furthermore, in certain embodiments, operations or events may be performed in parallel rather than sequentially, e.g., via multithreading, interrupt processing, or multiple processors or processor cores, or on other parallel architectures. Furthermore, different tasks or processes may be performed by different machines and / or computing systems that may function together.

[0166] The various illustrative logical blocks and modules described in connection with the embodiments disclosed herein may be implemented or executed by a machine such as a processor. The processor may be a microprocessor, but alternatively, the processor may be a controller, microcontroller, or state machine, or a combination thereof. The processor may include electrical circuitry configured to process computer-executable instructions. In another embodiment, the processor includes an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable device that performs logical operations without processing computer-executable instructions. A processor may also be implemented as a combination of computing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Although described herein primarily with reference to digital technology, a processor may also include primarily analog elements. For example, some or all of the signal processing algorithms described herein may be implemented by analog circuitry or mixed analog and digital circuitry. The computing environment can include any type of computer system, including, but not limited to, a microprocessor, mainframe computer, digital signal processor, portable computing device, device controller, or computer system based on a computational engine within an appliance.

[0167] Unless otherwise specified, conditional language such as "can," "could," "would," or "potential" is understood within the context in which it is generally used to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language does not generally imply that features, elements, and / or steps are required in any manner in one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included in or performed in any particular embodiment.

[0168] Disjunctive language such as "at least one of X, Y, Z," unless specifically stated otherwise, is understood in its general context to indicate that an item, term, etc. can be either X, Y, Z, or any combination thereof (e.g., X, Y, Z). Thus, such disjunctive language does not generally imply that a particular embodiment requires at least one of X, at least one of Y, or at least one of Z, respectively, to be present.

[0169] Any process descriptions, elements, or blocks in the flow diagrams described herein and / or illustrated in the accompanying drawings should be understood as potentially representing modules, segments, or portions of code, comprising one or more executable instructions for implementing a particular logical function or element in the process. Alternative embodiments are included within the scope of the embodiments described herein, in which elements or functions may be performed out of order, substantially simultaneously, or in reverse order from that shown or described, depending on the functionality involved, as will be understood by those skilled in the art.

[0170] Unless otherwise expressly stated, numeral terms such as "one" should generally be construed to include one or more described items. Thus, phrases such as "one device configured to" are intended to include one or more listed devices. Such one or more listed devices may also be collectively configured to perform the recited reference. For example, "a processor configured to perform the following A, B, and C" may include a first processor configured to perform A and a second processor configured to perform B and C. Additionally, even if a specific number of enumerations of the introduced embodiments are explicitly recited, those skilled in the art should construe such enumerations to typically mean at least the recited number (e.g., the mere enumeration of "two enumerations" without other modifiers typically means at least two enumerations, or two or more enumerations).

[0171] In general, it will be appreciated by those skilled in the art that the terms used herein generally intend "non-limiting" terms (e.g., the term "including" should be interpreted as "including but not limited to at least," the term "having" should be interpreted as "having at least," the term "including" should be interpreted as "including, but not limited to," etc.).

[0172] For purposes of description, the term "horizontal" as used herein is defined as a plane parallel to the plane or surface of the floor of the area in which the described system is used or the plane in which the described method is performed, regardless of its orientation. The term "floor" can be interchanged with the terms "ground" or "water surface." The term "vertical / plumb" refers to a direction perpendicular / vertical to a defined horizontal line. Terms such as "upper," "lower," "below," "top," "side," "higher," "lower," "above," "over," "below," etc. are defined relative to the horizontal plane.

[0173] As used herein, the terms "attach," "connect," "mate," and other related terms, unless otherwise noted, should be interpreted to include detachable, movable, fixed, adjustable, and / or removable connections or couplings. Connections / couplings include direct connections and / or connections with intermediate structures between the two components described.

[0174] Unless otherwise expressly stated, as used herein, numbers preceded by terms such as "approximately," "about," and "substantially" are inclusive of the recited number and also refer to an amount close to the recited amount that performs the desired function or achieves the desired result. For example, "approximately," "about," and "substantially" refer to values ​​less than 10% of the recited numerical value, unless otherwise expressly stated. As used herein, features of the disclosed embodiments preceded by terms such as "approximately," "about," and "substantially" refer to features that have some variability that also perform the desired function or achieve the desired result for that feature.

[0175] Many variations and modifications may be made to the above-described embodiments, and these elements should be understood to be among other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure and are protected by the following claims.

[0176] 10: Navigation support device 21: Designation unit 22: Positioning unit 23: Target detection unit 24: Target identification unit 25: Direction measurement unit 26: Attitude measurement unit 31: Coordinate calculation unit 32: Distance calculation unit 33: AR data generation unit 34: Movement vector setting unit 35: Correction unit 40: Video data generation unit 50: Synthesis unit 100: Reference point setting unit CAMF: Bow camera CAML: Port camera CAMR: Starboard camera CAMB: Stern camera CAMS: Forward monitoring camera PICF: Bow video data PICL: Port video data PICR: Starboard video data PICB: Stern video data

Claims

1. A navigation support device comprising: an image data generation unit that generates surrounding image data of a hull; a designation unit that designates a designated position on the image based on the surrounding image data; a coordinate calculation unit that calculates position coordinates in a virtual space corresponding to the designated position on the image; a positioning unit installed on the hull that measures an actual movement vector of the hull in an Earth coordinate system; a movement vector setting unit that sets a virtual movement vector in virtual space from the actual movement vector; and a correction unit that corrects the designated coordinates in the virtual space that correspond to the designated position based on the virtual movement vector.

2. A navigation support device as described in claim 1, wherein the correction unit corrects the designated position by correcting an orientation vector from a reference position of the surrounding image data in the virtual space to the designated position with the virtual movement vector.

3. A navigation support device as described in claim 1, comprising a reference point setting unit that sets a reference position in the virtual space based on the actual position of a virtual camera set in a global coordinate system, and the correction unit corrects the coordinates of the designated position in the virtual space.

4. A navigation support device as claimed in claim 3, comprising a plurality of cameras each capturing an image of a different area around the hull, and the reference position of the virtual space is set based on the positions of the plurality of cameras.

5. A navigation support device as described in claim 1, comprising: an AR data generation unit that generates AR data including marker data indicating the specified position in the coordinate system of the virtual space; and a synthesis unit that synthesizes the AR data with the surrounding image data.

6. A navigation support device as described in claim 5, further comprising a distance calculation unit that calculates an actual distance between the actual designated position in the Earth coordinate system of the designated position and the hull, and the AR data generation unit generates the AR data in which text data of the actual distance is placed in the vicinity of the marker data.

7. A marine vessel instrument according to claim 6, wherein the AR data generating unit deletes the specified position if the actual distance is equal to or greater than a display deletion threshold.

8. A navigation support data generating method, comprising: generating surrounding image data of a hull; specifying a specified position on the image based on the surrounding image data; calculating position coordinates in a virtual space corresponding to the specified position on the image; measuring an actual movement vector of the hull in an Earth coordinate system; setting a virtual movement vector in virtual space from the actual movement vector; and correcting the specified coordinates in the virtual space corresponding to the specified position based on the virtual movement vector.

9. A navigation support data generation program that causes a control unit to execute the following processes: generate surrounding image data of the hull; specify a specified position on the image based on the surrounding image data; calculate position coordinates in a virtual space corresponding to the specified position on the image; measure an actual movement vector of the hull in an Earth coordinate system; set a virtual movement vector in virtual space from the actual movement vector; and correct the specified coordinates in the virtual space corresponding to the specified position based on the virtual movement vector.

Images