Trend display device based on augmented reality

Abstract

The present application provides a kind of based on augmented reality (AR) fashion display device of tide information superimposed on the image photographed.The based on augmented reality (AR) fashion display device (1) of tide information superimposed on the image displayed in display screen (17), wherein the tide display device (1) has: image sensor information input unit (12), receives image sensor information;Tide information input unit (14), receives tide information;Tide selection unit (151), based on tide information, selects more than one tide (204,205) located in the field of view of image sensor (10);Position calculation unit (152), based on tide information and image sensor information, calculates the corresponding display position on display screen (17) for each tide;And tide information superimposition unit, for each selected tide, generates symbol indicating display position on display screen (17), and each symbol is output to display screen (17).

Description

Technical Field

[0001] This invention mainly relates to a current display device based on augmented reality (AR), and more specifically, to a device for overlaying current information onto images used for ship navigation using augmented reality. Background Technology

[0002] It is known that tidal currents affect the navigation of ships. Captains, crew, and other personnel aboard ships are sometimes unaware of when and how various tidal currents will affect the ship's course. For example, in waters where tidal currents may encounter other ships or where tidal conditions may differ, the position, direction, and / or speed of the tidal currents may be unknown to the ship's navigators. As a result, the ship's navigators may choose, or be forced to choose, a path that is not necessarily the most suitable for the ship in that water.

[0003] These currents are not only related to the time required for a ship's voyage, but also have a significant impact on fuel costs. From an environmental protection perspective, it is also crucial for crew members to accurately understand the state of the currents. Because maintenance work is often demanding, or ships are subject to the long-term influence of currents, it can sometimes adversely affect the ship's structure or navigation performance, or both. Therefore, from a safe navigation perspective, accurately understanding the currents is also essential.

[0004] Several systems have been developed in the past to assist ship navigation. Generally, nautical charts of a defined area (referred to as so-called nautical charts, sometimes including land portions) and charts showing the currents at various locations within that area are provided from an external server via a communication unit.

[0005] Information indicating tidal currents typically includes the measurement location and the direction and speed of the current at that location. Ships navigating in this area can obtain tidal current information using communication units such as satellite communications. The obtained information is displayed on the ship's navigation monitor, helping to determine the ship's route. However, most of these systems rely primarily on onboard equipment to sense various parameters related to the ship's surroundings. Nevertheless, using such onboard equipment to detect or measure conditions such as tidal currents is complex, difficult, and sometimes impossible. Furthermore, detection is limited to the vicinity of the ship; generally, the accuracy decreases as the detected object moves further away from the ship.

[0006] In navigation systems that utilize conventional augmented reality (AR), such as the system disclosed in Japanese Patent Application Publication No. 2019-121876, it is possible to overlay and display image information captured by an image sensor with information about moving vehicles on land obtained from information captured by sensors such as radar.

[0007] Furthermore, in previous AR navigation systems, tidal information was obtained from external sources via communication devices rather than from the vessel's own sensors, making it difficult to display tidal information. There is currently no AR-based navigation system capable of displaying tidal information from external devices on an AR display screen.

[0008] Patent document 1: Japanese Patent Application Publication No. 2019-121876. Summary of the Invention

[0009] Based on the above reasons, there is a need for systems and methods that consider tidal currents and their potential impact on ships to assist in navigation. The problem to be solved by this invention relates to an augmented reality-based tidal current display device, and more specifically, to an apparatus, method, and procedure for overlaying tidal current information onto images used for ship navigation.

[0010] To address the aforementioned issues, the present invention provides an AR-based tidal current display device that overlays tidal current information onto an image displayed on a screen. This tidal current display device comprises: an image sensor information input unit that receives image sensor information including the position and orientation of an image sensor mounted on a moving body located on the water surface, and outputs image data of the captured image onto the screen displaying the image; a tidal current information input unit that receives tidal current information including the position of the tidal current from a communication device located outside the moving body; a tidal current selection unit that selects one or more tidal currents within the field of view of the image sensor; a position calculation unit that, for each selected tidal current, calculates a display position corresponding to the position on the image displayed on the screen based on the tidal current information and the image sensor information; and a tidal current information overlay unit that, for each selected tidal current, generates a symbol representing the display position and outputs it to the screen display.

[0011] Furthermore, it can be configured to include: an image information input unit that receives images captured by an image sensor and outputs image data displayed on a display screen; and a trend information overlay unit that overlays corresponding trend symbols at various display positions on the image. The symbols are shapes representing the display positions of corresponding trends on the display screen, enabling the display to realistically represent the trends.

[0012] The AR-based tidal current display device of the present invention can be configured to further include: a chart information input unit that receives chart information representing a region containing a moving body, and a tidal current selection unit that sets a sub-region having a reference axis based on the orientation of the image sensor on the chart based on image sensor information, and selects the tidal current located within the sub-region based on the tidal current information.

[0013] In the aforementioned AR-based tidal current display device, the position calculation unit may include a local coordinate transformation unit that, based on the position and orientation of the image sensor, transforms the position of the tidal current coordinates on the nautical chart into the position of the tidal current in a local coordinate within a sub-region.

[0014] The position calculation unit can be configured to further include: a display coordinate transformation unit that converts the position of the tidal current in local coordinates into the position in coordinates of the display screen based on image sensor information. Furthermore, the local coordinate transformation unit inputs the direction and velocity of each selected tidal current, determines the direction and velocity of each selected tidal current in the sub-region based on the position and orientation of the image sensor, and the display coordinate transformation unit inputs the direction and velocity of each selected tidal current in the sub-region, calculates the direction of each selected tidal current displayed on the display screen based on the image sensor's height h above the water surface W, the depression angle α, and the field of view angle β.

[0015] The tidal information overlay unit also generates a symbol representing at least one of the tidal direction and speed corresponding to the display position of each selected tidal current on the display screen, and outputs each symbol to the display screen.

[0016] The tidal flow information overlay unit can also display symbols representing the direction and velocity of each selected tidal flow on at least one of a sub-screen on the display screen or an external display screen outside the display screen. Furthermore, the tidal flow information overlay unit can be configured such that the characteristics of the length, thickness, and color of each symbol on the image containing the symbols displayed on the display screen change according to the velocity of the corresponding tidal flow.

[0017] The tidal flow information overlay unit can also generate the symbol according to the depth of each selected tidal flow, and display the symbol. The tidal flow information overlay unit can also generate the symbol corresponding to a predicted position representing the future position of the tidal flow displayed on the screen, and overlay the symbol onto the image on the screen.

[0018] The above structure can also be configured as an AR navigation system that overlays tidal current information onto an image displayed on a screen. Specifically, the AR navigation system of the present invention includes: an image sensor mounted on a mobile body located on the water surface, which captures images within its field of view and outputs image data containing dynamic or static images; a display unit that displays the images on a screen based on the image data; and a tidal current information overlay unit that acquires tidal current information, including the location of the tidal current, from a communication device located outside the mobile body and overlays it onto the image. The tidal current information overlay unit includes: an image sensor information input unit that receives image sensor information including the position and orientation of the image sensor; a tidal current information input unit that receives tidal current information, including the location of the tidal current, from a communication device located outside the mobile body; a tidal current selection unit that selects one or more tidal currents within the field of view of the image sensor; a position calculation unit that, for each selected tidal current, calculates a display position corresponding to the position displayed on the image on the screen based on the tidal current information and the image sensor information; and a tidal current information overlay unit that, for each selected tidal current, generates a symbol representing the display position and outputs it to the display unit.

[0019] In an AR navigation system, there may also be: a chart information input unit that receives chart information representing a region containing a moving object, and a current selection unit that, based on the image sensor information, sets a sub-region with a reference axis at the orientation of the image sensor on the chart, and selects the current located within the sub-region based on the current information.

[0020] Additionally, the position calculation unit may also include a local coordinate transformation unit, which, based on the position and orientation of the image sensor, converts the position on the tidal chart into a position in local coordinates within a sub-region. The position calculation unit may also include a display coordinate transformation unit, which, based on image sensor information, converts the position in local coordinates of the tidal current into a position in coordinates displayed on the screen.

[0021] The local coordinate transformation unit can also input the direction and velocity of each selected current, and determine the direction and velocity of each selected current in the sub-region based on the position and orientation of the image sensor. The display coordinate transformation unit can also input the direction and velocity of each selected current in the sub-region, and calculate the direction of each selected current displayed on the display screen based on the height h of the image sensor above the water surface W, the depression angle α, and the field of view angle β.

[0022] The flow information overlay unit can also generate symbols representing at least one of the direction and speed of each selected flow at its display position on the display screen, and output these symbols to the display screen. Alternatively, it can display the symbols representing the direction and speed of each selected flow on a sub-screen of the display screen or on at least one of an external display screen outside the display screen. The flow information overlay unit can also cause the characteristics of the length, thickness, and color of each symbol on the image containing the symbols displayed on the display screen to change according to the speed of the corresponding flow.

[0023] Furthermore, the tidal information overlay unit can generate and display symbols based on the depth of each selected tidal current. The tidal information overlay unit can also generate symbols corresponding to predicted positions representing the future positions of the tidal currents displayed on the screen, and overlay these symbols onto the image displayed on the screen.

[0024] Furthermore, the AR-based tidal current display method according to the present invention provides the following method: receiving an image captured by an image sensor mounted on a mobile body located on the water surface; outputting image data displayed on a display screen; receiving image sensor information including the position and orientation of the image sensor; receiving tidal current information including the position of the tidal current from a communication device located outside the mobile body; selecting one or more tidal currents within the field of view of the image sensor; for each selected tidal current, calculating a display position corresponding to the position displayed on the image screen based on the tidal current information and the image sensor information; and generating a symbol representing the display position for each selected tidal current and superimposing it on the image. Alternatively, nautical chart information representing a region containing the mobile body may be received, and a sub-region with a reference axis may be set based on the orientation of the image sensor on the nautical chart according to the image sensor information; and a tidal current located within the sub-region may be selected based on the tidal current information.

[0025] According to the AR-based tidal current display program of the present invention, a program stored in a non-transitory computer-readable medium storing computer-executable commands is provided. When the program is executed by a computer, the computer performs the following steps: receiving and storing image data based on images captured by an image sensor; receiving and storing image sensor information including the position and orientation of an image sensor containing a moving body mounted on the water surface; receiving and storing a nautical chart of the area containing the moving body; receiving and storing tidal current information related to one or more positions on the nautical chart; determining sub-regions within the area of ​​the nautical chart based on the image sensor information; selecting one or more tidal currents located within the sub-regions; calculating the display position of each selected tidal current on the display screen based on the tidal current information and the image sensor information; generating a symbol for each selected tidal current corresponding to each display position; and overlapping each symbol with each display position on the image displayed on the display screen based on the image data.

[0026] According to the display device, display method, and display program of the present invention, the state of the tidal current on the sea surface can be visually captured by an image sensor, so that the position and direction of the tidal current are consistent with the orientation and position of the sea surface on the image mapped by the image sensor, and the tidal current information provided by a device other than the ship can be displayed in augmented reality (AR). Attached Figure Description

[0027] The embodiments illustrated in this work can be best understood by referring to the accompanying drawings. Here, the same parts are generally indicated by the same numbers. The following description is intended only as an example, illustrating specific selections of embodiments of devices, systems, methods, and processes consistent with the subject matter claimed in this specification.

[0028] Figure 1 This is a block diagram showing the overall structure of an augmented reality (AR) type trend display device with an image sensor mounted on a moving body according to one embodiment of the present invention.

[0029] Figure 2 It is a diagram showing the region containing the moving body and the sub-regions within that region.

[0030] Figure 3 This is a diagram showing the overlay of trend information with images captured by an image sensor.

[0031] Figure 4 It is a side view showing the height of the image sensor above the water surface, the downward angle of the image sensor, and the field of view of the image sensor.

[0032] Figure 5 It is shown Figure 1 Block diagram of the various parts of the position calculation unit of an AR-based trend display device.

[0033] Figure 6 It is a diagram showing an image captured by an image sensor and a symbol superimposed on the image, as well as a superimposed image generated by superimposing the symbol on the image.

[0034] Figure 7 This shows the generated symbols, transformed in a three-dimensional manner, used for overlaying with images generated by an image sensor.

[0035] Figure 8 It is a display screen that shows an overlay image representing the position of more than one current and sub-screens representing the direction of each current.

[0036] Figure 9 The display shows an overlay image indicating the location of the current and an external display indicating the direction of the current.

[0037] Figure 10 This is a flowchart illustrating a method for overlaying trend information onto an image displayed on a display screen, according to one embodiment of the present disclosure.

[0038] Explanation of reference numerals in the attached figures:

[0039] 1: Trend display device

[0040] 10: Image sensor (camera, image sensor)

[0041] 11: Ships (mobile vehicles)

[0042] 12: Image sensor information input unit

[0043] 13: Chart Information Input Department

[0044] 14: Trend Information Input Department

[0045] 15: Processing Circuit

[0046] 16: Image Information Input Unit

[0047] 17: Display screen

[0048] W: Water surface

[0049] 151: Trend Selection Department

[0050] 152: Location Calculation Department

[0051] 153: Trend Information Overlay Department

[0052] 200: Nautical Chart

[0053] 201: Region containing moving objects

[0054] 202: Reference axis of sub-region

[0055] 203: Subregion

[0056] 204: Trend

[0057] 205: Other Trends

[0058] 204a: A symbol indicating the position of a current.

[0059] 205a: Symbol indicating the location of other currents.

[0060] 206, 502: Images captured by the image sensor

[0061] 402: Local coordinate transformation section

[0062] 404: Display coordinate transformation section

[0063] 504: Overlay image

[0064] 505b: Symbol indicating the direction of the current.

[0065] 505c: A two-dimensional symbol representing the direction of a current.

[0066] 506: Additional Information

[0067] 507b: Symbol indicating the direction of other currents.

[0068] 507c: A two-dimensional symbol indicating the direction of other currents.

[0069] 508: Display screen

[0070] 510: Overlay Images

[0071] 512: Sub-screen

[0072] 514a: Symbol indicating the position of the current.

[0073] 514c: Symbol indicating the direction of the current.

[0074] 516a: Symbol indicating the location of other currents.

[0075] 516c: A symbol indicating the direction of the current.

[0076] 518: External display screen Detailed Implementation

[0077] Here, examples of the apparatus are described. Further modifications may be made using other exemplary embodiments or features without departing from the spirit or scope of the subject matter set forth in this specification. In the following detailed description, reference will be made to the accompanying drawings, which form a part of the description.

[0078] The exemplary embodiments described in this specification are not limited thereto. The modes of this disclosure, as generally described in this specification and illustrated in the accompanying drawings, as clearly indicated in this specification, are readily understood to be capable of being arranged, replaced, combined, separated, and designed in various different structures.

[0079] First, refer to Figure 1 The structure of a power flow display device according to one embodiment will be described. Figure 1 This is a block diagram illustrating the overall structure of an augmented reality (AR) type trend display device 1 with an image sensor 10 mounted on a mobile body 11 according to one embodiment of the present invention. Figure 2 Chart 200 showing region 201 containing moving body 11 and sub-region 203 in chart 200 having a reference axis relative to the direction of travel of moving body 11. Figure 3 This indicates a situation where current information is superimposed on an image captured by image sensor 10.

[0080] Next, the main references are... Figure 1 The image sensor 10 can be mounted on a mobile body 11 (hereinafter, an embodiment will be described using a ship as an example, and therefore referred to as ship 11) and electrically connected to the display device 1.

[0081] As will be described later, the current display device 1 of this embodiment is electrically connected to the image sensor 10 mounted on the ship 11 and used as a ship instrument.

[0082] Image sensor 10 may, for example, be a limited field-of-view or wide-angle camera image sensor that captures images of the water surface W surrounding or at least a portion of the vessel 11. This image sensor 10 may also have a live output function, thereby generating image data (picture data) as the result of the capture in real time and outputting it to display screen 17. For example... Figure 1 As shown, the image sensor 10 only needs to be installed inside the vessel 11 with the shooting direction generally facing the water surface W in front of the hull.

[0083] The image sensor 10 can be mounted on the vessel 11 via a rotating mechanism (not shown). Therefore, by inputting signals indicating the orientation and depression angle from the display device 1, the shooting direction can be changed within a specified angular range with reference to the hull of the vessel 11. In addition, since the draft and attitude of the vessel 11 sometimes change due to waves, the height h of the image sensor 10 relative to the water surface W may sometimes change.

[0084] In particular, the AR-based trend display device 1 can also be connected to the display screen 17 in a communicable state to generate an image representing the surrounding conditions of the ship 11 using AR based on the position and orientation of the image sensor 10.

[0085] Display screen 17 can be configured, for example, as part of a navigation aid device referenced by a ship operator of the vessel 11. However, display screen 17 is not limited to the above-described structure, and may also be, for example, a display screen of a portable computer carried by a ship operator to monitor the surrounding situation from the vessel 11, a display screen for passengers to observe from inside the cabin of the vessel 11, or a display unit of a head-mounted display such as wearable glasses worn by passengers.

[0086] In embodiments of this disclosure, the image sensor 10 and the display screen 17 are located outside the display device 1. In another embodiment of this disclosure, the image sensor 10 and the display screen 17 are essential components of the display device 1. Alternatively, the image sensor 10, the display screen 17, and the AR-based trend display device 1 can also together constitute an AR-based navigation system to facilitate user navigation of the vessel 11.

[0087] The AR-based trend display device 1 can also be connected to various peripheral devices, including but not limited to keyboards and mice that users can operate to perform various functions consistent with this disclosure. For example, users can give various instructions for generating images to the AR-based trend display device 1 and the image sensor 10 by operating the keyboard and / or mouse and touch panel. These instructions include operations related to the orientation and tilt angle of the image sensor 10, settings for displaying / not displaying various information, settings for the shooting viewpoint, etc.

[0088] Next, we will refer to Figure 1 The structure of the AR-based tidal current display device 1 will be described in detail. The AR-based tidal current display device 1 includes: an image sensor information input unit 12, a chart information input unit 13, a tidal current information input unit 14, a processing circuit 15, and an image information input unit 16. The image information input unit 16 is configured to receive an image captured by the image sensor 10 and output the image data to the display screen 17.

[0089] Furthermore, the image sensor information input unit 12 is configured to receive and store image sensor information including the position and orientation of the image sensor 10 relative to the reference axis of the global topographic map, such as... Figure 4 As shown, the image sensor information may also include the height h of the image sensor 10 above the water surface W, the depression angle (α) of the image sensor 10, and the field of view (β) of the image sensor 10. The image sensor 10 is fixedly mounted on the vessel 11. Therefore, assuming the position of the image sensor 10 on the nautical chart 200 is the same as the position of the vessel 11 on the nautical chart 200, the bearing direction of the image sensor 10 is determined using the bearing direction of the vessel 11. Conversely, the bearing direction is determined by the same orientation.

[0090] Return to Figure 1 The chart information input unit 13 is configured to receive and store area 201 of the chart 200 containing the ship 11 based on pre-stored electronic chart information.

[0091] The tidal current information input unit 14 is configured, for example, to receive and store tidal current information, including the position (W) of the tidal current on the water surface, from external communication devices such as land stations, Global Navigation Satellite System (GNSS) receivers, Electronic Chart Display and Information Systems (ECDIS), Automatic Identification System (AIS) receivers, and radar devices. The tidal current information may also include the direction and speed of the tidal current. Typically, the tidal current information input unit 14 is configured to store tidal current information for the area 201 containing the vessel 11. The tidal current information input unit 14 is configured to store tidal current information relative to the reference axis of the chart 200, and to store the chart and global coordinates of the tidal current in area 201.

[0092] Reference Figure 1 , Figure 2 as well as Figure 3 The various diagrams will be explained. Figure 1 The processing circuit 15 shown includes: a power flow selection unit 151, which selects one or more power flows 204, 205 (see reference) located within the field of view of the image sensor 10 based on power flow information. Figure 2 The position calculation unit 152 calculates the corresponding display position of each selected current 204, 205 on the display screen 17 based on current information and image sensor information; and the current information overlay unit 153 generates symbols 204a, 205a representing the position of each selected current 204, 205 on the image 206 displayed on the display screen 17 for each selected current 204, 205.

[0093] The flow selection unit 151 sets the processing circuit 15 based on the area of ​​the image that the image sensor 10 can capture. For example... Figure 3 As shown, each symbol 204a, 205a is configured as a circular mark, which indicates the position of the corresponding current 204, 205 on image 206. Although the case where 204a and 205a are circular marks is shown, symbols 204a and 205a may also include other structures such as elliptical marks or square marks indicating the position of each current on image 206 displayed on display screen 17.

[0094] The tidal information overlay unit 153 outputs 204a and 205a to the display screen 17, overlaying 204a and 205a onto the image 206 captured by the image sensor 10, thereby displaying the position of the tidal currents 204 and 205 relative to the sea surface in a visually easy-to-understand manner.

[0095] In the context of this disclosure, processing circuitry 15 includes other circuitry that controls the operation of various components such as processors, computers, microcontrollers, or operation panels and memory. Processing circuitry 15 is capable of executing software, firmware, and / or other commands, which may be stored in volatile or non-volatile memory or otherwise provided to processing circuitry 15.

[0096] Next, refer to Figure 1 , Figure 2 as well as Figure 3 The various charts illustrate the selection of tidal currents. The tidal current selection unit 151 receives image sensor information, chart information, and tidal current information from the image sensor information input unit 12, the chart information input unit 13, and the tidal current information input unit 14, respectively. The tidal current selection unit 151 selects one or more tidal currents 204 and 205 located in sub-region 203, that is, located at the field of view (β) of the image sensor 10. Specifically, the tidal current selection unit 151 receives the chart 200 of the region 201 surrounding the ship 11, determines the sub-region 203 based on the position and orientation of the image sensor 10, and selects the first tidal current 204 and the second tidal current 205 within the sub-region 203 based on the tidal current information.

[0097] Two selected currents are shown here, but the current selection unit 151 can also select more than two or fewer currents.

[0098] Next, refer to Figure 4 The position calculation is explained below. The position calculation unit 152 calculates the position of each selected current 204, 205 to be displayed on the display screen 17 based on image sensor information and current flow information. Specifically, the position calculation unit 152 is configured to calculate the display coordinates of the first current 204 and the second current 205 to be displayed on the display screen 17.

[0099] by Figure 5 The structure of the position calculation unit 152 will be described in detail with reference to the central reference. The position calculation unit 152 includes a local coordinate transformation unit 402, which receives the chart coordinates of the selected first and second currents 204, 205 and transforms them into corresponding local coordinates based on the position and orientation of the image sensor 10. In this disclosure, the chart coordinates are the position on the chart 200, and the local coordinates are the position relative to the reference axis 202 in the sub-region 203, and also the position relative to the direction of travel of the vessel 11.

[0100] Additionally, the position calculation unit 152 includes: a display coordinate transformation unit 404, based on... Figure 4The image sensor 10 is shown at its height (h) above the water surface W, its depression angle (α), and its field of view (β). The selected local coordinates of the first and second tidal currents 204 and 205 are then converted into corresponding display coordinates. In the context of this disclosure, the display coordinates correspond to the display position on the display screen 17. Therefore, the position calculation unit 152 receives the chart coordinates of the tidal currents 204 and 205 from the chart 200 and generates the corresponding display coordinates for display on the display screen 17.

[0101] Furthermore, as an option, the local coordinate transformation unit 402 can also receive the direction and velocity of the first and second currents 204 and 205 selected from the current selection unit 151, and determine the direction and velocity associated with the first and second currents 204 and 205 within the sub-region 203 based on the position and orientation of the image sensor 10.

[0102] For reference Figure 2 As illustrated, the directions of the first and second currents 204 and 205 in chart 200 are relative to the global reference axis, and the directions of currents 204 and 205 in sub-region 203 are relative to the direction of travel of ship 11 and also relative to the reference axis 202.

[0103] The display coordinate transformation unit 404 calculates the direction and speed of each selected first and second current 204, 205 in the display screen 17 based on the direction and speed of the selected first and second currents 204, 205 in the sub-region 203, for example using mathematical and / or graphical transformations, and based on the height h of the image sensor 10 above the water surface W, the depression angle α, and the field of view angle β of the image sensor 10. Thus, the position calculation unit 152 inputs the orientation of the currents 204, 205 relative to the chart 200, generates the orientation information of the first and second currents 204, 205, and displays it on the display screen 17.

[0104] Next, refer to Figure 1 as well as Figure 6 The generation and overlay of symbols indicating the direction of current flows will be explained. The current flow information overlay unit 153 generates symbols 505b and 507b representing the position and direction of each current flow displayed on the image 502 of the display screen 17, based on information generated by the position calculation unit 152. The current flow information overlay unit 153 further overlays 505b and 507b onto the image 502, thereby generating an overlay image 504 representing the position and direction of each current flow in the overlay image 504. Furthermore, 505b and 507b may, for example, be the shape of an arrow representing the direction of the corresponding current flow.

[0105] Furthermore, the length, thickness, and color of the arrows in the overlay image 504 can be varied based on the corresponding current velocity. For example, shorter or thinner arrows can be used to represent weaker (slower-moving) currents in the image. Green or yellow can also be used to depict such arrows overlaid on image 502. Alternatively, longer or thicker arrows can be overlaid on image 502 to display stronger (higher-speed) currents. Optionally, for example, red can be used to graphically display longer or thicker arrows on image 502. The current information overlay unit 153 can also generate symbols such as 505b and 507b based on the depth of each current.

[0106] Furthermore, in the descriptions of the various symbols above, the characteristics of symbols 505b and 507b are not limited to specific shapes, sizes, or colors. These characteristics, namely shape, size, and color, can vary to suit more than one maritime code requirement and / or other application-specific requirements.

[0107] Furthermore, when symbols 505b and 507b are superimposed on image 505, the limited display area of ​​display screen 17 can be effectively utilized by simultaneously displaying additional information 506, which includes the total direction and speed. The positions of symbols 505b and 507b can be changed as needed, or removed from the superimposed image 504, so that graphics related to the superimposed image 504 are hidden by symbols 505b and 507b as much as possible, or not hidden at all.

[0108] Furthermore, the tidal current information overlay unit 153 generates symbols 505b and 507b, representing the direction and speed of the tidal current, respectively, on an image parallel to the water surface W, based on the downward angle α of the image sensor 10. For example... Figure 7 As shown, symbols 505b and 507b can be generated on an image parallel to the water surface W by converting two-dimensional symbols 505c and 507c into their corresponding three-dimensional symbols 505b and 507b, respectively.

[0109] Symbols 505b and 507b can also be displayed tilted depending on the tilt angle α of the image sensor 10. In this way, graphics related to the water surface W can be obtained from the image sensor 10. For example, even when the hull and the image sensor 10 are tilted relative to the water surface W, the user can visually grasp the surrounding situation, or symbols 505b and 507b can be displayed overlapping as needed.

[0110] Figure 8A display screen 508 shows an overlay image 510 indicating the position of more than one current, and a sub-screen 512 indicating the direction of the currents shown in the overlay image 510. The sub-screen 512 has a reference axis relative to the direction of travel of the vessel 11.

[0111] The overlay image 510 includes symbols 514a and 516a (similar to symbols 204a and 205a) representing the positions of more than one current on the image captured by the image sensor 10. In the sub-screen 512, two-dimensional symbols 514c and 516c (similar to symbols 505c and 507c) representing the direction of the current corresponding to symbols 514a and 516a in the overlay image 510 are displayed. Symbols 514a, 516a, 514c, and 516c are generated by the current information overlay unit 153. The overlay image 510 and the sub-screen 512 are synchronized with each other in time. Changes in the overlay image 510 are also reflected in the sub-screen 512.

[0112] Figure 9 A display screen 508 shows an overlay image 510 indicating the position of more than one current, and an external display screen 518 indicating the direction of the currents shown in the overlay image 510. The external display screen 518 has a reference axis, for example, relative to the direction of travel of a ship 11 such as a vessel.

[0113] The overlay image 510 includes symbols 514a and 516a (similar to symbols 204a and 205a) indicating the position of one or more currents on the image captured by the image sensor 10. In the external display screen 518, two-dimensional symbols 514c and 516c (similar to symbols 204c and 205c) indicating the direction of the currents corresponding to the symbols 514a and 516a of the overlay image 510 are displayed. Symbols 514a, 516a, 514c, and 516c are generated by the current information overlay unit 153. The contents displayed on the display screen 508 and the external display screen 518 are synchronized with each other.

[0114] Alternatively, the position calculation unit 152 can also predict the future position of more than one current displayed on the image captured by the image sensor. Therefore, in this case, the current information overlay unit 153 can also generate a symbol corresponding to the predicted future position, and overlay the generated symbol onto the image and display it on the display screen 17.

[0115] The ship's instruments (information sources for additional display information) connected to display device 1 are not limited to Figure 1 The instruments recorded may also include other ship instruments.

[0116] Furthermore, this disclosure can be applied not only to ships sailing at sea, but also to any moving body on water that can navigate in the sea, lakes, rivers, etc.

[0117] Next, one implementation of the trend display method will be described. Figure 10 This is a flowchart illustrating a method for overlaying trend information onto an image displayed on display screen 17, according to one embodiment of the present disclosure.

[0118] In step 602, the image information input unit 16 receives and stores image data based on the image captured by the image sensor 10.

[0119] In step 604, image sensor information including the position and orientation of the image sensor 10 installed on the ship 11 on the water surface (W) is received and stored in the image sensor information input unit 12.

[0120] In step 606, chart information of a chart 200 representing an area 201 containing the vessel 11 is received and stored in the chart information input unit 13.

[0121] In step 608, tidal current information related to one or more locations on the nautical chart 200 is received and stored in the tidal current information input unit 14.

[0122] In step 610, the current selection unit 151 determines a sub-region 203 within region 201 of the nautical chart 200 based on image sensor information.

[0123] In step 612, the power flow selection unit 151 selects one or more power flows 204, 205 located in sub-region 203 based on the power flow information.

[0124] In step 614, the position calculation unit 152 calculates the display position in the display screen 17 for each selected current based on the current information and the image sensor information.

[0125] In step 616, the power flow information overlay unit 153 generates a symbol for each selected power flow corresponding to each display position.

[0126] In step 618, the trend information overlay unit 153 overlays each symbol at each display position on the image displayed on the display screen 17 based on the image data.

[0127] The above are examples of embodiments of the present invention, and it is not possible to achieve all the objectives or effects / advantages based on any specific embodiment described in this specification. Therefore, those skilled in the art will, for example, realize that the specific embodiments taught in this specification may not achieve other objectives or effects / advantages as taught or suggested in this specification. Configuration is required to achieve or optimize one or more effects / advantages.

[0128] All the processes described in this specification are implemented through software code modules executed by a computing system comprising one or more computers or processors, thereby achieving full automation. The code modules can be stored on any type of non-transitory computer-readable medium or other computer storage device. Some or all of the methods can be implemented using dedicated computer hardware.

[0129] As will be clear from this disclosure, there are many other variations beyond those described in this specification. For example, according to embodiments, any particular action, event, or function in the algorithm described in this specification can be executed in different sequences, and can be added, combined, or completely excluded (e.g., not all described actions or events need to execute the algorithm).

[0130] Furthermore, in certain implementations, actions or events can be processed via multithreading, interrupt handling, or executed in parallel rather than sequentially via multiple processors or processor cores or other parallel architectures. Additionally, different tasks or processes can be executed through different machines and / or computing systems that can function together.

[0131] The various illustrative logic blocks and modules described in association with the embodiments disclosed in this specification can be implemented or executed by a machine such as a processor. The processor may also be a microprocessor, or a controller, microcontroller, state machine, or a combination thereof. The processor may include circuitry configured to process computer-executable commands. In another embodiment, the processor includes an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable devices that perform logical operations without processing computer-executable commands. The processor can also be implemented as a combination of computing devices, such as a combination of a digital signal processor (digital signal processing device) and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. While this specification primarily describes digital technologies, the processor may also primarily include analog elements. For example, some or all of the signal processing algorithms described in this specification can be implemented using analog circuitry or mixed analog and digital circuitry. Computing environments include microprocessors, mainframe computers, digital signal processors, portable computing devices, device controllers, or computer systems based on computing engines within a device, but may also include any type of computer system not limited to these.

[0132] Any process description, element, or block in the flowcharts described in this specification and / or the accompanying drawings should be understood as a potential module, segment, or code containing one or more executable commands for implementing a specific logical function or element in the process. Alternative embodiments are included within the scope of the embodiments described in this specification, and hereby, as those skilled in the art will understand, elements or functions may be substantially performed according to the associated functionality.

[0133] Unless otherwise explicitly stated, the numeral "a" should generally be interpreted as including more than one described item. Therefore, phrases such as "a device as described above" are intended to include more than one listed device. Such one or more listed devices can also be configured collectively in a manner that executes the described references. For example, "a processor configured to execute A, B, and C" could include a first processor configured to execute A and a second processor configured to execute B and C. Furthermore, even when a specific number of the imported embodiments is explicitly listed, such a listing typically means at least the number listed. The simple listing of "two listings" should be interpreted as referring to at least two listings or more listings.

[0134] Numerous modifications and alterations can be added to the above embodiments, and it should be understood that these elements are among other permissible examples. All such changes and modifications are intended to be included within the scope of this disclosure and protected by the scope of the following claims.

Claims

1. A trend display device based on AR, which overlays trend information onto an image displayed on a screen, wherein, have: The image sensor information input unit receives image sensor information including the position and orientation of the image sensor, wherein the image sensor is mounted on a moving body located on the water surface and outputs the image data of the captured image on the display screen; The chart information input unit receives chart information representing a region containing the moving body; The power flow information input unit receives power flow information, including the location of the power flow, from a communication device located outside the mobile body; The flow selection unit selects one or more flow rates located within the field of view of the image sensor; as well as The position calculation unit, for each selected current flow, calculates the display position corresponding to the position displayed on the image screen based on the current flow information and the image sensor information. The current selection unit, based on the image sensor information, defines a sub-region with a reference axis at the orientation of the image sensor on the nautical chart, and selects the current located within the sub-region based on the current information. The position calculation unit has: The local coordinate transformation unit converts the position of the current on the nautical chart into the position on the local coordinates within the sub-region based on the position and orientation of the image sensor. as well as The display coordinate transformation unit converts the position of the current in the local coordinates into the position in the coordinates of the display screen based on the image sensor information.

2. The AR-based trend display device according to claim 1, wherein, It also has: The image information input unit receives images captured by the image sensor and outputs image data displayed on the display screen. The trend information overlay unit generates a trend symbol corresponding to each of the display positions on the image, and overlays the symbol onto the image data.

3. The AR-based trend display device according to claim 2, wherein, The symbol is the shape of a marker indicating the display position of the corresponding trend in the display screen.

4. The AR-based trend display device according to claim 1, wherein, The local coordinate transformation unit inputs the direction and velocity of each selected current flow, and determines the direction and velocity of each selected current flow in the sub-region based on the position and orientation of the image sensor. The display coordinate transformation unit inputs the direction and velocity of each selected current in the sub-region, and calculates the direction of each selected current displayed on the display screen based on the height (h), depression angle (α), and field of view angle (β) of the image sensor above the water surface (W).

5. The AR-based trend display device according to claim 4, wherein, The current information overlay unit also generates a symbol representing at least one of the direction and speed of the current corresponding to the display position of each selected current on the display screen, and outputs each symbol to the display screen.

6. The AR-based trend display device according to claim 5, wherein, The tidal information overlay will also display symbols representing the direction and velocity of each selected tidal current on a sub-screen of the display screen or on an external display screen outside the display screen.

7. The AR-based trend display device according to claim 6, wherein, The current information overlay also causes the characteristics of each symbol, including the length, thickness, and color of the symbols displayed on the screen, to change according to the speed of the corresponding current.

8. The AR-based trend display device according to any one of claims 5 to 7, wherein, The current flow information overlay unit also generates and displays the symbol according to the depth of each selected current flow.

9. The AR-based trend display device according to claim 3, wherein, The current information overlay unit also generates a symbol corresponding to a predicted position representing the future position of the current displayed on the screen, and overlays the symbol onto the image displayed on the screen.

10. An AR navigation system that overlays trend information onto an image displayed on a screen, wherein, have: An image sensor is mounted on a moving body on the water surface to capture images within its field of view and output image data containing moving or still images. The chart information input unit receives chart information representing a region containing the moving body; The display unit displays the image on a display screen based on the image data; as well as The trend information display unit overlays trend information onto the image. The trend information display unit includes: The image sensor information input unit receives image sensor information including the position and orientation of the image sensor. The power flow information input unit acquires power flow information, including the location of the power flow, from a communication device located outside the mobile body; The flow selection unit selects one or more flow rates located within the field of view of the image sensor; The position calculation unit calculates, for each selected current, a display position corresponding to the position displayed on the image screen based on the current information and the image sensor information; The trend information overlay unit generates a symbol representing the display position for each selected trend and outputs it to the display unit; and The sub-region tidal current information display unit, based on the image sensor information, sets a sub-region with a reference axis at the orientation of the image sensor on the nautical chart, and selects the tidal current located within the sub-region based on the tidal current information to indicate the corresponding position on the nautical chart. The position calculation unit has: A local coordinate transformation unit, based on the position and orientation of the image sensor, converts the coordinates of the current on the nautical chart into local coordinates within the sub-region; and The display coordinate transformation unit converts the position of the current in the local coordinates into the position in the coordinates of the display screen based on the image sensor information.

11. The AR navigation system according to claim 10, wherein, The local coordinate transformation unit inputs the direction and velocity of each selected current flow, and determines the direction and velocity of each selected current flow in the sub-region based on the position and orientation of the image sensor. The display coordinate transformation unit inputs the direction and velocity of each selected current in the sub-region, and calculates the direction of each selected current displayed on the display screen based on the height (h), depression angle (α), and field of view angle (β) of the image sensor above the water surface (W).

12. The AR navigation system according to claim 11, wherein, The current information overlay unit also generates a symbol representing at least one of the direction and speed of the current corresponding to the display position of each selected current on the display screen, and outputs each symbol to the display screen.

13. The AR navigation system according to claim 12, wherein, The tidal information overlay will also display symbols representing the direction and velocity of each selected tidal current on a sub-screen of the display screen or on an external display screen outside the display screen.

14. The AR navigation system according to claim 13, wherein, The current information overlay also causes the characteristics of each symbol on the image, including the length, thickness, and color of the symbols displayed on the screen, to change according to the speed of the corresponding current.

15. The AR navigation system according to any one of claims 12 to 14, wherein, The current flow information overlay unit also generates and displays the symbol according to the depth of each selected current flow.

16. The AR navigation system according to claim 15, wherein, The current information overlay unit also generates a symbol corresponding to a predicted position representing the future position of the current displayed on the screen, and overlays the symbol onto the image displayed on the screen.

17. A trend display method based on AR, which overlays trend information onto a display screen, wherein, The AR-based trend display method includes the following steps: It receives images captured by an image sensor mounted on a moving body located on the water surface. Receive chart information representing a nautical chart containing the moving body. And output the image data displayed on the display screen. Receive image sensor information including the position and orientation of the image sensor. Receive power flow information, including the location of the power flow, from a communication device located outside the moving body. Select one or more currents located within the field of view of the image sensor. For each selected current, based on the current information and the image sensor information, a display position corresponding to the position displayed on the image screen is calculated. For each selected current, a symbol representing the display location is generated and overlaid on the image. Based on the image sensor information, a sub-region with a reference axis is defined for the orientation of the image sensor on the nautical chart, and the current within the sub-region is selected based on the current information to indicate the corresponding position on the nautical chart. The calculation of the display position corresponding to the position displayed on the image on the display screen includes the following steps: Based on the position and orientation of the image sensor, the coordinates of the tidal current on the nautical chart are converted into local coordinates within the sub-region; and Based on the image sensor information, the position of the current in the local coordinates is converted into the position in the coordinates of the displayed screen.

18. The AR-based trend display method according to claim 17, wherein, Based on the tidal flow information and the position and orientation of the image sensor, the direction and velocity of each selected tidal flow in the sub-region are determined. For each current flow located within the sub-region, the direction and speed of that current flow are displayed.

19. A computer program product comprising an AR-based trend display program that overlays and displays trend information on a display screen, wherein, When the computer executes this program, it causes the computer to perform the following steps: Receive and store image data based on images captured by an image sensor. Receives and stores image sensor information including the position and orientation of an image sensor on a moving object mounted on the water surface. Receive and store nautical charts of the area containing the moving body, and receive and store tidal current information related to more than one location on the nautical charts. Based on the image sensor information, sub-regions within the area of ​​the nautical chart are determined, and one or more currents located within those sub-regions are selected. Based on the current information and the image sensor information, the display position of each selected current on the display screen is calculated. Generate a symbol for the selected current flow corresponding to the display position. Based on the image data, each symbol is superimposed on the image displayed on the display screen at each of the display positions; The calculation of the display position of each selected tidal current on the display screen includes the following steps: based on the position and orientation of the image sensor, converting the coordinates of the tidal current on the nautical chart into local coordinates within the sub-region, and Based on the image sensor information, the position of the current in the local coordinates is converted into the position in the coordinates of the displayed screen.

Images