Navigation aid identification device, autonomous navigation system, navigation aid identification method, and program

Abstract

[Problem] To provide a sea mark identification device capable of distinguishing a port mark and a starboard mark regardless of the location of a ship. [Solution] A sea mark identification device comprises: an acquisition unit that acquires an image generated by a camera installed on a ship; a mode identification unit that identifies whether the mode of a lateral buoy contained in the image is a first mode or a second mode; a country determination unit that determines a country to which a location detected by a location detection unit for detecting the location of the ship belongs; and a port / starboard determination unit that, on the basis of predetermined corresponding relationships indicating whether the first mode and the second mode correspond to a port sign or a starboard sign in each country, determines whether the mark content of the lateral buoy is a port mark or a starboard mark, from the mode of the lateral buoy and the determined country.

Description

Technical Field

[0001] The present invention relates to a route marker identification device, an autonomous navigation system, a route marker identification method, and a program. It relates to.

Background Art

[0002] Patent Document 1 discloses an automatic visual recognition device that automatically identifies route markers.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, among the buoys floating on the sea surface, side buoys represent either a left-hand mark or a right-hand mark, but in some countries, the interpretations of the left-hand mark and the right-hand mark may be opposite.

[0005] The present invention has been made in view of the above problems, and its main object is to provide a route marker identification device, an autonomous navigation system, a route marker identification method, and a program capable of discriminating between a left-hand mark and a right-hand mark regardless of the position of a ship.

Means for Solving the Problems

[0006] To solve the above problems, a navigation aid identification device according to one aspect of the present invention includes: an acquisition unit that acquires an image generated by a camera installed on a ship; an aspect identification unit that identifies whether the aspect of a side buoy included in the image is a first aspect or a second aspect; a country determination unit that determines the country to which the position detected by a position detection unit that detects the position of the ship belongs; and a port / starboard determination unit that determines whether the marker content of the side buoy is a port aspect or a starboard aspect based on the aspect of the side buoy and the determined country, based on a predetermined correspondence relationship that indicates whether the first aspect and the second aspect correspond to port aspects or starboard aspects in each country.

[0007] In the above embodiment, the embodiment identification unit may identify whether the color of the side buoy is green or red.

[0008] In the above embodiment, the embodiment identification unit may identify whether the top mark of the side buoy is cylindrical or conical.

[0009] In the above embodiment, the system further includes a type identification unit for identifying the type of buoy in the image, The mode identification unit may identify the mode of the side buoy when the type of buoy is a side buoy.

[0010] In the above embodiment, the display control unit may further include a display control unit that displays a symbol representing the marking content of the side buoy on the first image, an electronic chart, or a radar image based on the marking content of the side buoy, the position of the side buoy in the first image, and the imaging direction of the camera.

[0011] In the above embodiment, the system may further include a consistency determination unit that determines the consistency between the marker content of the side buoy and the marker content represented by the navigation aid data recorded in the electronic chart, based on the marker content of the side buoy, the position of the side buoy in the first image, the imaging direction of the camera, and the position of the vessel.

[0012] In the above embodiment, the system may further include a display control unit that displays the consistency determination result on the first image, the electronic chart, or the radar image.

[0013] Furthermore, an autonomous navigation system in another aspect of the present invention may include the above-described navigation aid identification device and a navigation path calculation unit that calculates the ship's navigation path or navigation path width based on the position of the side buoys in the first image and the imaging direction of the camera, when the markings of the plurality of side buoys include at least two of the port side buoys, starboard side buoys, and safe waters buoys.

[0014] Furthermore, an autonomous navigation system in another aspect of the present invention may include the above-mentioned navigation aid identification device, a virtual marker acquisition unit that acquires data representing the position and content of a virtual marker, and a route calculation unit that calculates the ship's route or route width based on the content of the side buoy, the position of the virtual marker, and the content of the virtual marker.

[0015] Furthermore, an autonomous navigation system in another aspect of the present invention may include the above-mentioned navigation aid identification device, a position detection unit for detecting the position of the vessel, and a route calculation unit for setting a course turning point that the vessel should pass through based on the marking content of the side buoy, the position of the side buoy in the first image, the imaging direction of the camera, and the position of the vessel.

[0016] Furthermore, an autonomous navigation system in another aspect of the present invention may include the above-mentioned navigation aid identification device, a direction detection unit for detecting the heading of the vessel, and a route calculation unit for setting the direction in which the vessel should navigate based on the markings of the side buoys, the imaging direction of the camera, and the heading of the vessel.

[0017] In the above embodiment, an automatic steering system that performs autonomous navigation control based on the markings on the side buoys may be further provided.

[0018] In addition, the seaway marker identification method according to another aspect of the present invention acquires an image generated by a camera installed on a ship, identifies whether the aspect of the lateral buoy included in the image is either the first aspect or the second aspect, determines the country to which the position detected by the position detection unit that detects the position of the ship belongs, and based on a predetermined correspondence relationship indicating which of the left - hand side marker and the right - hand side marker the first aspect and the second aspect correspond to in each country, determines whether the marker content of the lateral buoy is the left - hand side marker or the right - hand side marker from the aspect of the lateral buoy and the determined country.

[0019] In addition, a program according to another aspect of the present invention causes a computer to execute acquiring an image generated by a camera installed on a ship, identifying whether the aspect of the lateral buoy included in the image is either the first aspect or the second aspect, determining the country to which the position detected by the position detection unit that detects the position of the ship belongs, and based on a predetermined correspondence relationship indicating which of the left - hand side marker and the right - hand side marker the first aspect and the second aspect correspond to in each country, determining whether the marker content of the lateral buoy is the left - hand side marker or the right - hand side marker from the aspect of the lateral buoy and the determined country.

Advantages of the Invention

[0020] According to the present invention, it is possible to distinguish between the left - hand side marker and the right - hand side marker regardless of the position of the ship.

Brief Description of the Drawings

[0021] [Figure 1] It is a block diagram showing a configuration example of an on - ship system. [Figure 2] It is a diagram for explaining the marker content of a buoy, etc. [Figure 3] It is a block diagram showing a functional configuration example of a seaway marker identification device. [Figure 4] It is a diagram showing an example of a first image. [Figure 5] It is a diagram showing an example of identification by a first identification unit. [Figure 6] It is a diagram showing an example of a second image. [Figure 7] It is a flowchart showing an example of the procedure of a sea route sign identification method. [Figure 8] It is a flowchart showing an example of the procedure of sign content identification processing. [Figure 9] It is a diagram showing an example of a buoy management database. [Figure 10] It is a diagram showing an example of display by a display unit. [Figure 11] It is a diagram showing another example of display by a display unit. [Figure 12] It is a block diagram showing another configuration example of a sea route sign identification device. [Figure 13] It is a block diagram showing yet another configuration example of a sea route sign identification device. [Figure 14] It is a block diagram showing a configuration example of a second identification unit according to a first modification example. [Figure 15] It is a flowchart showing an example of the procedure of left / right side sign identification processing. [Figure 16] It is a diagram showing an example of a country-specific left / right side mode table. [Figure 17] It is a block diagram showing a configuration example of a second identification unit according to a second modification example. [Figure 18] It is a flowchart showing an example of the procedure of sign content identification processing. [Figure 19] It is a diagram for explaining the flashing manner of a buoy, etc.

Embodiments for Carrying Out the Invention

[0022] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0023] FIG. 1 is a block diagram showing a configuration example of an autonomous navigation system 100. The autonomous navigation system 100 is an ICT system mounted on a ship. Hereinafter, a ship on which the autonomous navigation system 100 is mounted is referred to as a "self-ship".

[0024] The autonomous navigation system 100 includes a navigation aid identification device 1, a camera 2, a radar 3, an AIS 4, a radio communication unit 5, a display unit 6, a GNSS receiver 7, a gyrocompass 8, an ECDIS 9, and an automatic steering system 10. These devices are connected to a network N, such as a LAN, and are capable of network communication with each other.

[0025] The navigation aid identification device 1 is a computer that includes a CPU, RAM, ROM, non-volatile memory, and input / output interfaces. The CPU of the navigation aid identification device 1 performs information processing according to a program loaded from ROM or non-volatile memory into RAM.

[0026] The program may be supplied via an information storage medium such as an optical disc or memory card, or via a communication network such as the Internet or LAN.

[0027] Camera 2 is a digital camera that captures images of the exterior of the vessel and generates image data. Camera 2 is installed, for example, on the bridge of the vessel, facing the bow. Camera 2 is a visible light camera capable of capturing images in at least the visible range. It may also be capable of capturing images in the infrared range.

[0028] In this embodiment, camera 2 is a camera having pan-tilt and optical zoom functions, a so-called PTZ camera. Camera 2 pans, tilts, or zooms in response to commands from the navigation aid identification device 1.

[0029] Radar 3 emits radio waves around the vessel and receives the reflected waves, generating echo data based on the received signals. Radar 3 also identifies targets from the echo data and generates target tracking data (TT data) that represents the position and speed of the targets.

[0030] The Automatic Identification System (AIS) 4 receives AIS data from other vessels or shore-based control systems in the vicinity of the vessel. While AIS is not the only option, a VHF Data Exchange System (VDES) may also be used. The AIS data includes information such as the position and speed of other vessels.

[0031] Furthermore, AIS4 may acquire AIS data representing the position and content of virtual markers. AIS4 is an example of a virtual marker acquisition unit. Virtual markers using AIS are so-called virtual AIS navigational aids.

[0032] The radio communication unit 5 includes various radio equipment for enabling communication with other ships or land-based air traffic control, such as radio equipment for the very high frequency band, medium high frequency band, and short frequency band.

[0033] The display unit 6 is, for example, a display device with a touch sensor, a so-called touch panel. The display unit device may include, for example, a liquid crystal display device or an organic EL display device. It is not limited to a touch sensor; other pointing devices such as a trackball or mouse may also be used.

[0034] The display unit 6 displays images captured by the camera 2, radar images generated by the radar 3, electronic charts, or composite images created by combining radar images and electronic charts.

[0035] The GNSS receiver 7 detects the ship's position based on radio waves received from the GNSS (Global Navigation Satellite System). The GNSS receiver 7 is an example of a position detection unit that detects the ship's position.

[0036] The gyrocompass 8 detects the ship's heading. The gyrocompass 8 is an example of a heading detection unit that detects the ship's heading. Other types of compasses, such as GPS compasses, may be used instead of a gyrocompass.

[0037] The ECDIS (Electronic Chart Display and Information System) 9 obtains the ship's position from the GNSS receiver 7 and displays the ship's position on the electronic chart. The ECDIS 9 also displays the ship's planned route on the electronic chart. A GNSS plotter may be used instead of the ECDIS.

[0038] The automatic steering system 10 calculates a target rudder angle to point the bow toward the target course based on the target course obtained from the navigation aid identification device 1 and the heading obtained from the gyrocompass 8, and drives the steering gear so that the rudder angle approaches the target rudder angle. The automatic steering system 10 may also control the engine.

[0039] In this embodiment, the navigation aid identification device 1 is an independent device, but it is not limited to this and may be integrated with other devices such as ECDIS9. In other words, the functions of the navigation aid identification device 1 may be realized by other devices such as ECDIS10.

[0040] Furthermore, although the display unit 2 is an independent device in this embodiment, it is not limited to this, and a display unit provided by another device such as ECDIS9 may be used as the display unit 2 that displays the image generated by the navigation aid identification device 1.

[0041] Figure 2 is a diagram illustrating the markings on buoys. Buoys are navigational aids floating on the sea surface, also known as buoys. The types and markings of buoys are classified by their color and the shape of their top mark.

[0042] Buoys are categorized into side markers, azimuth markers, isolated obstacle markers, safe water markers, and special markers. Side markers include port markers and starboard markers. Port and starboard refer to left and right when facing the source of the water. Azimuth markers include north azimuth markers, east azimuth markers, south azimuth markers, and west azimuth markers.

[0043] For isolated obstacle markers, safe water area markers, and special markers, the marker content is not further subdivided. In other words, the type of buoy itself can be said to represent the marker content.

[0044] Incidentally, buoys floating on the sea surface are small compared to ships, making it difficult to identify the markings on buoys that are far away from ships. Therefore, in this embodiment, the accuracy of identifying the markings is improved by acquiring images in stages, as described below.

[0045] Figure 3 is a block diagram showing an example of the functional configuration of a navigation aid identification device 1 according to an embodiment. The navigation aid identification device 1 includes a first acquisition unit 11, a first identification unit 12, a second acquisition unit 13, a second identification unit 14, a display control unit 15, a route calculation unit 16, and a consistency determination unit 18.

[0046] These functional units are realized by the CPU of the navigation aid identification device 1 performing information processing according to a program. Note that some functional units, such as the display control unit 15 or the route calculation unit 16, may be realized by a separate computer, such as the ECDIS 9 or the automatic steering system 10, which is separate from the navigation aid identification device 1.

[0047] Furthermore, the navigation aid identification device 1 includes a model storage unit 17 for storing learned models. This storage unit is located in the non-volatile memory of the navigation aid identification device 1. However, the model storage unit 17 may be located outside the navigation aid identification device 1.

[0048] The first acquisition unit 11 acquires the first image generated by the camera 2. Specifically, the first acquisition unit 11 sequentially acquires multiple time-series first images generated by the camera 2 and provides them sequentially to the first identification unit 12.

[0049] The first image is an image captured when camera 2 is in its standard state. The standard state is, for example, when the optical zoom magnification is at its minimum and the imaging direction is facing the bow of the ship. Except for the period controlled by the second acquisition unit 13, camera 2 repeatedly generates the first image in the standard state.

[0050] The multiple first images in the time series may be, for example, multiple still images (frames) included in a video, or multiple still images individually generated by capturing images at predetermined time intervals.

[0051] Figure 4 shows an example of a first image P1 acquired by the first acquisition unit 11. This figure shows an example in which the first image P1 includes the ship's hull SP along with the port side marker LL and starboard side marker LR floating on the sea surface in front of the ship.

[0052] The first identification unit 12 identifies the position of the buoy in the first image P1. Specifically, the first identification unit 12 identifies the position of the buoy in the first image P1 using the first trained model stored in the model storage unit 17. In addition, the first identification unit 12 may further identify the type of buoy along with its position in the first image P1.

[0053] The first pre-trained model is generated by machine learning using training images as input data and the labels (or labels indicating the type of buoy) and locations of buoys within the training images as training data. The first pre-trained model thus generated estimates the labels (or labels indicating the type of buoy), locations, and accuracy of buoys within the first image P1. The location of a buoy is represented, for example, by the coordinates of the bounding box surrounding the buoy.

[0054] The first pre-trained model may be an object detection model such as SSD (Single Shot MultiBox Detector), YOLO (You Only Look Once), or Mask R-CNN. However, the first pre-trained model may also be a region segmentation model such as Semantic Segmentation or Instance Segmentation, or a feature point detection model such as Keypoint Detection.

[0055] Figure 5 shows an example of the identification of the first image P1 by the first identification unit 12. The figure shows an example in which the port side marker LL and the starboard side marker LR are identified as buoys (or side buoys) and enclosed by a boundary box BB.

[0056] The second acquisition unit 13 acquires a second image with higher resolution than the first image P1, corresponding to a subregion of the first image P1 that includes the position of the buoy. The subregion is, for example, a bounding box BB (see Figure 5) identified by the first identification unit 12.

[0057] Camera 2 is equipped with a lens unit 21 that provides an optical zoom function and a pan-tilt mechanism 22 that provides a pan-tilt function, and the second acquisition unit 13 acquires a second image by controlling the lens unit 21 and the pan-tilt mechanism 22 of Camera 2.

[0058] Specifically, the second acquisition unit 13 controls the lens unit 21 to cause the camera 2 to expand the real-space range corresponding to a sub-region of the first image P1 and capture the image, thereby acquiring the second image. By utilizing this optical zoom function, a second image with higher resolution than the first image P1 is acquired.

[0059] Furthermore, the second acquisition unit 13 controls the pan-tilt mechanism 22 to orient the imaging direction of the camera 2 to a real-space range corresponding to a sub-region of the first image P1. The second acquisition unit 13 sets a target value for the imaging direction of the camera 2 according to the position of the buoy in the first image P1 identified by the first identification unit 12.

[0060] Figure 6 shows an example of a second image P2 acquired by the second acquisition unit 13. This figure shows an example where the second image P2 includes a port side marker LL. In the second image P2, the color of the port side marker LL and the shape of the top mark TM are easier to identify than in the first image P1 (see Figure 4).

[0061] As shown in Figure 5 above, if multiple buoys (in the illustrated example, port marker LL and starboard marker LR) are identified in the first image P1, the second acquisition unit 13 will have the camera 2 sequentially image each of the multiple buoys, thereby acquiring a second image P2 for each of the multiple buoys.

[0062] The second identification unit 14 identifies the markings of the buoy from the second image P2. Specifically, the second identification unit 14 uses the second trained model stored in the model storage unit 17 to identify the markings of the buoy from the second image P2.

[0063] The second pre-trained model is generated using machine learning, with the training images as input data and the labels of the buoy markings within the training images as training data. This generated second pre-trained model estimates the labels and accuracy of the buoy markings within the second image P2.

[0064] For the second pre-trained model, for example, a model of the same type as the first pre-trained model described above may be used. In this case, the first and second pre-trained models will each have different first and second pre-trained parameters incorporated into a common inference program.

[0065] This is not the only example; the second pre-trained model may be an object recognition model that only identifies objects and does not detect their positions.

[0066] Furthermore, the second pre-trained model may include a pre-trained model for side signs that specializes in identifying the content of side signs, and a pre-trained model for directional signs that specializes in identifying the content of directional signs.

[0067] Figures 7 and 8 are flowcharts illustrating an example of the procedure for identifying navigational aids implemented by the navigational aid identification device 1. These figures primarily show the processes performed by the navigational aid identification device 1, specifically the processes related to image acquisition and identification of the aid's content.

[0068] The CPU of the navigation aid identification device 1 functions as a first acquisition unit 11, a first identification unit 12, a second acquisition unit 13, and a second identification unit 14 by executing the information processing shown in these figures according to the program.

[0069] As shown in Figure 7, first, the navigation aid identification device 1 acquires a first image P1 (see Figure 4) from the camera 2 (S11: Processing as the first acquisition unit 11).

[0070] Next, the navigation aid identification device 1 uses the first trained model to identify the position and type of buoy in the first image P1 (S12: Processing as the first identification unit 12).

[0071] Next, the navigation aid identification device 1 determines whether the type of buoy identified in the first image P1 is a side buoy or a bearing buoy (S13).

[0072] If the type of buoy is a lateral buoy or a bearing buoy (S13 → YES), the navigation aid identification device 1 controls the camera 2 to acquire a second image P2 (see Figure 6) which is an enlarged image of the buoy (S14: Processing as the second acquisition unit 13).

[0073] Next, the navigation aid identification device 1 performs an indicator content identification process to identify the indicator content of the buoy from the second image P2 (S15: processing as the second identification unit 14).

[0074] As shown in Figure 8, in the marker content identification process S15, if the type of buoy is a side buoy (S21 → side buoy), the navigational aid identification device 1 uses a second learned model for side buoys to identify whether the marker content is a port buoy or a starboard buoy (S22).

[0075] On the other hand, if the type of buoy is a azimuth buoy (S21 → azimuth buoy), the navigation aid identification device 1 uses a trained model for azimuth buoys as a second trained model to identify whether the buoy is a north azimuth buoy, east azimuth buoy, south azimuth buoy, or west azimuth buoy (S23).

[0076] Furthermore, if the buoy is not a side buoy or a bearing buoy (S13→NO), that is, if the buoy is an isolated obstacle marker, a safe water area marker, or a special marker, the navigation aid identification device 1 will not acquire the second image P2. This is because, in the case of those buoys, the type itself represents the marker content.

[0077] If multiple buoys are identified in the first image P1 in S12, the navigation aid identification device 1 performs S13 to S15 for all identified buoys (S16). That is, for all buoys that are either lateral buoys or azimuth buoys, the device acquires the second image P2 and identifies the marker content.

[0078] According to the embodiment described above, the mark content is identified from a second image P2, which has a higher resolution than the first image P1 and is imaged after being magnified based on the position of the buoy identified in the first image P1. This makes it possible to improve the accuracy of identifying the mark content.

[0079] Furthermore, according to this embodiment, the type of buoy is identified in the first image P1 and then the marking content is identified from the second image P2. This allows the marking content to be narrowed down according to the type of buoy, thereby further improving the accuracy of marking content identification.

[0080] The method is not limited to this; the buoy and its location may be identified from the first image P1, and the type of buoy and the markings on it may be identified from the second image P2.

[0081] Figure 9 shows an example of a buoy management database. The buoy management database is a database for managing information on identified or acquired buoys and is stored in the non-volatile memory of the navigation aid identification device 1. The buoy management database includes not only information on buoys identified from images from camera 2, but also information on virtual markers acquired by AIS4.

[0082] The buoy management database includes fields such as "Identifier," "Type," "Marking Content," "Location in Image," "Actual Location," and "Virtual Buoy." The "Identifier" is an identifier used to identify a buoy. "Virtual Buoy" indicates whether or not it is a virtual buoy.

[0083] "Type" indicates the type of buoy. "Marking Content" indicates the marking content of the buoy. If "Type" is a side marker or azimuth marker, then "Marking Content" will be entered as a port marker or north azimuth marker, etc. On the other hand, if "Type" is an isolated obstacle marker, safe water marker, or special marker, no data will be entered in "Marking Content".

[0084] "Image Position" represents the buoy's position within the first image P1 (see Figure 4). Note that for virtual buoys, no data is entered in "Image Position". "Actual Position" represents the actual position of the buoy. The actual position of a buoy identified from the image of camera 2 is calculated based on the buoy's image position and the imaging direction of camera 2.

[0085] Returning to the explanation of Figure 3, the display control unit 15 generates display data related to the buoy and outputs it to the display unit 6. Specifically, the display control unit 15 displays a symbol representing the buoy's markings on the first image P1, an electronic chart, or a radar image, etc., based on the markings of the identified buoy, the position of the buoy in the first image P1, and the imaging direction of the camera 2.

[0086] For example, as shown in Figure 10, the display control unit 15 displays an image on the display unit 6 in which symbols ML and MR, representing the contents of the port side marker LL and starboard side marker LR, are attached in association with their positions in the first image P1. The symbols ML and MR include, for example, strings of characters representing the contents of the markers.

[0087] Furthermore, as shown in Figure 11, the display control unit 15 displays an image on the display unit 6 in which symbols TL and TR, representing the contents of the port side marker LL and starboard side marker LR, are attached to positions corresponding to the actual positions of the port side marker LL and starboard side marker LR within the composite image CP, which is a composite image obtained by combining the electronic chart and the radar image. The symbols TL and TR have, for example, shapes that represent the contents of the markers.

[0088] The composite image CP displays the ship's symbol SF, its planned route RT, the point of change of course DF on the planned route RT, and the symbols EL of other ships, etc.

[0089] Furthermore, symbols VL and VR representing the content of the virtual marker may be displayed in the composite image CP. Symbols VL and VR have the same shape as symbols TL and TR. It is preferable that symbols VL and VR are displayed in a way that makes them distinguishable from symbols TL and TR, for example, by changing their transparency.

[0090] The route calculation unit 16 calculates a target course, i.e., bearing, turning point, and course, for autonomous navigation control based on the identified buoy markings. The calculated target course is provided to the automatic steering system 10, which performs autonomous navigation control. Here, misrecognition of navigation markings can cause serious accidents when performing autonomous navigation control. Therefore, by performing autonomous navigation control by identifying navigation markings with improved accuracy according to the present invention, an autonomous navigation system that can withstand operation in real environments can be realized.

[0091] As shown in Figure 5, the route calculation unit 16 calculates the planned route or route width of the vessel based on the positions of the port side marker LL and the starboard side marker LR in the first image P1 and the imaging direction of camera 2, when the buoys identified in the first image P1 include the port side marker LL and the starboard side marker LR. Specifically, the route calculation unit 16 sets the planned route RT of the vessel so that it passes between the port side marker LL and the starboard side marker LR from the vessel's position, based on the actual positions of the port side marker LL and the starboard side marker LR calculated from the positions of the port side marker LL and the starboard side marker LR in the first image P1 and the imaging direction of camera 2 (see Figure 11). In addition, if the buoy identified in the first image P1 includes a port marker LL or starboard marker LR and a safe waters marker, the route calculation unit 16 may set the vessel's planned route RT between the port marker LL or starboard marker LR and the safe waters marker.

[0092] Furthermore, the route calculation unit 16 may calculate the distance between the port side marker LL and the starboard side marker LR as the route width W, based on the actual positions of the port side marker LL and the starboard side marker LR calculated from the positions of the port side marker LL and the starboard side marker LR in the first image P1 and the imaging direction of the camera 2. The calculated route width W may be displayed in the first image P1 displayed on the display unit 6, or it may be displayed in the composite image CP which is a composite of the electronic chart and the radar image (see Figure 11).

[0093] The route calculation unit 16 may set turning points that the vessel should pass through based on the marking content of the identified buoy, the position of the buoy in the first image P1, the imaging direction of the camera 2, and the vessel's own position. Specifically, the route calculation unit 16 sets one or more turning points DF for setting the planned route RT for the vessel entering or leaving a port, based on the marking content of the identified side markers and bearing buoys, the actual position of those buoys calculated from the position of those buoys in the first image P1 and the imaging direction of the camera 2, and the vessel's own position (see Figure 11). Not limited to this, the route calculation unit 16 may also set one or more turning points for setting an avoidance route to avoid obstacles or special areas, based on the marking content of the identified isolated obstacle markers or special markers, the actual position of those buoys calculated from the position of those buoys in the first image P1 and the imaging direction of the camera 2, and the vessel's own position.

[0094] The route calculation unit 16 may set the direction the ship should navigate based on the markings of the identified buoys, the imaging direction of the camera 2, and the ship's heading. For example, the route calculation unit 16 maintains or adjusts the direction the ship should navigate so that buoys such as side buoys continue to be included in multiple time-series first images P1. The route calculation unit 16 may also use the positions of the buoys in the first images P1 to set the direction the ship should navigate so that it heads towards the space between the port and starboard markers, or in a direction along multiple port or starboard markers.

[0095] The route calculation unit 16 may calculate a target course, i.e., bearing, turning points, and route, for autonomous navigation control based on the position and content of virtual markers, in addition to the marking content of the identified buoys. Specifically, when data for virtual port marker VL and virtual starboard marker VR is acquired, the route calculation unit 16 may set the planned route RT of its own vessel so that it passes not only between port marker LL and starboard marker LR identified in the first image P1, but also between virtual port marker VL and virtual starboard marker VR.

[0096] The consistency determination unit 18 determines the consistency between the buoy's marker content and the marker content represented by the navigational aid data recorded in the electronic chart, based on the identified buoy's marker content, the buoy's position in the first image P1, the imaging direction of the camera 2, and the ship's position. Specifically, the consistency determination unit 18 calculates the actual position of the buoy from the buoy's position in the first image P1, the imaging direction of the camera 2, and the ship's position, and extracts navigational aid data corresponding to the buoy's actual position from the navigational aid data recorded in the electronic chart. It then determines whether the identified buoy's marker content matches the marker content represented by the extracted navigational aid data.

[0097] The display control unit 15 displays the determination result from the alignment determination unit 18 on the first image P1, electronic chart, or radar image, etc. For example, the display control unit 15 displays a symbol indicating whether or not alignment is achieved, associated with the buoy, on the first image P1 (see Figure 10) or composite image CP (see Figure 11), etc., displayed on the display unit 6. Alternatively, the display control unit 15 may display a symbol representing the marking content (symbols ML, MR in Figure 10 or symbols TL, TR in Figure 11, etc.) only on buoys that have been aligned.

[0098] The configuration of the navigation aid identification device 1 is not limited to the example shown in Figure 3. For example, as shown in Figure 12, the second acquisition unit 13A may be an image processing unit that acquires a second image by increasing the resolution of a portion of the first image. By performing this resolution increase, a second image with higher resolution than the first image is acquired.

[0099] The system is not limited to this, however, the first acquisition unit 11 may acquire a first image by performing a decimation or averaging process on the original image generated by the camera 2, and the second acquisition unit 13 may acquire a second image by cutting out a region corresponding to a sub-region of the first image from the original image. In this case, a second image with higher resolution than the first image will also be acquired.

[0100] Furthermore, as shown in Figure 13, the second acquisition unit 13B may be a camera control unit that acquires a second image by having an auxiliary camera 3, which has a higher resolution than camera 2, capture an area of ​​real space corresponding to a sub-region of the first image. By using the auxiliary camera 3 in this way, a second image with a higher resolution than the first image is acquired.

[0101] The auxiliary camera 3, like camera 2 shown in Figure 3 above, is equipped with a lens unit 31 that provides optical zoom functionality and a pan-tilt mechanism 32 that provides pan-tilt functionality. The lens unit 31 of the auxiliary camera 3 has a higher magnification than the lens unit 21 of camera 2.

[0102] [First variation] The first modified example is described below. Configurations or processes that overlap with the above embodiment may be denoted by the same reference numerals, and detailed explanations may be omitted.

[0103] The interpretation of port and starboard buoys can differ from country to country. Therefore, in this modified example, port and starboard buoys are distinguished regardless of the ship's position, as explained below.

[0104] Figure 14 is a block diagram showing an example configuration of the second identification unit 14A according to the first modified example. The figure mainly shows the functional unit for identifying the marking content of the side buoy, which is one of the functional units realized in the second identification unit 14A.

[0105] The second identification unit 14A includes a color identification unit 31, a shape identification unit 32, a country determination unit 33, and a port / starboard determination unit 34. The color identification unit 31 and the shape identification unit 32 are examples of the form identification unit.

[0106] When the type of buoy identified by the first identification unit 12 (type identification unit) shown in Figure 3 is a lateral buoy, the functional unit of the second identification unit 14A identifies the markings of the lateral buoy included in the second image P2 (see Figure 6).

[0107] Figure 15 is a flowchart showing an example of the procedure for the port and starboard marker identification process S22 according to the first modified example, which is implemented by the second identification unit 14A. The navigation marker identification device 1 performs the information processing shown in the figure according to the program.

[0108] The port and starboard buoy identification process S22 corresponds to S22 shown in Figure 8 above. That is, the port and starboard buoy identification device 1 executes the port and starboard buoy identification process S22 when the type of buoy identified in S12 shown in Figure 7 above is a side buoy.

[0109] First, the navigation aid identification device 1 identifies whether the side buoy included in the second image P2 is green or red (S31: processing as color identification unit 31). Green and red are examples of the first and second embodiments.

[0110] Next, the navigation aid identification device 1 identifies whether the top mark of the side buoy included in the second image P2 is cylindrical or conical (S32: processing as the shape identification unit 32). The cylindrical and conical shapes are examples of the first and second embodiments.

[0111] Color identification and top mark shape identification are performed using a trained model, similar to the embodiment described above. For example, a trained model that identifies both color and top mark shape may be used, or a trained model that identifies color and a trained model that identifies top mark shape may be used separately.

[0112] Next, the navigation aid identification device 1 determines the country to which the detected position of its own vessel, as detected by the GNSS receiver 7 (see Figure 1), belongs (S33: Processing as the country determination unit 33). For example, the navigation aid identification device 1 determines, based on chart data, which country's territorial waters the coordinates of its own vessel's detected position are located.

[0113] Next, the navigation aid identification device 1 refers to the country-specific port and starboard configuration table and determines whether the markings on the side buoy are port or starboard based on the color identified in S31, the shape of the top mark identified in S32, and the country determined in S33 (S34: Processing as port / starboard determination unit 34).

[0114] Figure 16 shows an example of a country-specific port and starboard configuration table. The country-specific port and starboard configuration table is a table that shows the correspondence between the configuration of a buoy and the content of the marker, and is stored in the non-volatile memory of the navigation aid identification device 1.

[0115] Specifically, the country-specific port / starboard configuration table indicates whether the green and red colors of the side buoys correspond to port or starboard markers in each country. Furthermore, the country-specific port / starboard configuration table also indicates whether the cylindrical and conical shapes of the top marks correspond to port or starboard markers in each country.

[0116] According to the first modified example described above, it becomes possible to distinguish between port and starboard markers based on the markings on the side buoys, regardless of the ship's position.

[0117] [Second variation] The following describes a second modified example. Configurations or processes that overlap with the above embodiment may be denoted by the same reference numerals, and detailed explanations may be omitted.

[0118] The type and markings of buoys can be identified by elements such as the buoy's color, the shape of its top mark, and how it reflects light. However, when identifying markings directly from an image, the contribution of each element is unclear, and the identification accuracy may be insufficient. Therefore, this modified version aims to improve the identification accuracy of markings as described below.

[0119] Figure 17 is a block diagram showing an example configuration of the second identification unit 14B according to the second modified example. The second identification unit 14B includes a color identification unit 41, a first candidate determination unit 42, a shape identification unit 43, a second candidate determination unit 44, a light emission identification unit 45, a third candidate determination unit 46, and a sign content determination unit 47.

[0120] Figure 18 is a flowchart showing an example of the procedure for the sign content identification process S15 according to the second modified example, which is implemented by the second identification unit 14B. The navigation aid identification device 1 executes the information processing shown in the figure according to the program. The sign content identification process S15 corresponds to S15 shown in Figure 7 above.

[0121] Figure 19 shows the color, top mark shape, and illumination pattern corresponding to the markings of buoys. Buoy markings are classified by the buoy's color, top mark shape, and illumination pattern. The illumination pattern refers to the time pattern of when the light is on and off.

[0122] As shown in Figure 18, first, the navigation aid identification device 1 identifies the color candidates of the buoys contained in the second image P2 (S41: processing as the color identification unit 41). Specifically, the navigation aid identification device 1 identifies the color candidates of the buoys in the second image P2 using a trained model. In addition, the navigation aid identification device 1 calculates a first probability, which represents the likelihood of the color candidates, along with the color candidates.

[0123] Next, the navigation aid identification device 1 determines the first candidate for the buoy's marker content corresponding to the identified color candidate (S42: Processing as the first candidate determination unit 42). Specifically, the navigation aid identification device 1 refers to a table representing the correspondence between colors and marker content and determines the marker content corresponding to the color candidate as the first candidate.

[0124] Next, the navigation aid identification device 1 identifies candidate shapes for the top marks of buoys contained in the second image P2 (S43: Processing as the shape identification unit 43). Specifically, the navigation aid identification device 1 uses a trained model to identify candidate shapes for the top marks of buoys in the second image P2. The navigation aid identification device 1 also calculates a second confidence score, which represents the likelihood of each shape candidate, along with the shape candidates.

[0125] Next, the navigation aid identification device 1 determines a second candidate for the buoy's marker content corresponding to the identified shape candidate (S44: Processing as the second candidate determination unit 44). Specifically, the navigation aid identification device 1 refers to a table representing the correspondence between shapes and marker content and determines the marker content corresponding to the shape candidate as the second candidate.

[0126] Next, the navigation aid identification device 1 identifies candidate buoy illumination patterns from multiple time-series second images P2 (S45: processing as illumination pattern identification unit 45). The navigation aid identification device 1 identifies candidate buoy illumination patterns according to predetermined rules.

[0127] Specifically, the navigation aid identification device 1 extracts the time patterns of the buoy's illumination and extinction from multiple time-series second images P2, and selects the standard time pattern that is most similar to the extracted time pattern from among multiple pre-stored standard time patterns as the illumination pattern candidate. The standard time patterns are created based on the illumination patterns of each marker (see Figure 19).

[0128] Furthermore, the navigation aid identification device 1 calculates a third-degree probability, which represents the likelihood of a candidate light pattern, along with the candidate light pattern itself. Specifically, the navigation aid identification device 1 calculates the similarity between the extracted time pattern and the standard time pattern designated as the candidate light pattern as the third-degree probability.

[0129] Next, the navigation aid identification device 1 determines a third candidate for the buoy's marker content corresponding to the identified light pattern candidate (S46: Processing as the third candidate determination unit 46). Specifically, the navigation aid identification device 1 determines the marker content corresponding to the standard time pattern designated as the light pattern candidate as the third candidate.

[0130] Next, the navigation aid identification device 1 determines whether the current time is daytime or nighttime (S47). If it is daytime, it applies the daytime criteria (S48), and if it is nighttime, it applies the nighttime criteria (S49). The current time is the time when the image was generated by camera 2. The criteria are the criteria used to determine the marker content of the buoy.

[0131] Next, the navigation aid identification device 1 determines the buoy's marker content based on the first candidate for marker content determined in S42, the second candidate for marker content determined in S44, and the third candidate for marker content determined in S46 (S50: Processing as the marker content determination unit 47).

[0132] Specifically, the navigation aid identification device 1 determines the buoy's marker content to be the same marker content if at least two of the first, second, and third candidates have the same marker content. For example, if two of the first, second, and third candidates are port markers and the remaining one is a starboard marker, then the port marker is determined to be the marker content.

[0133] Furthermore, the navigation aid identification device 1 may determine the buoy's marking content based on the first, second, and third degrees of certainty. For example, the candidate corresponding to the highest degree of certainty among the first, second, and third degrees of certainty is determined as the marking content. Also, if multiple candidates represent the same marking content, their corresponding degrees of certainty may be added together.

[0134] The navigation aid identification device 1 changes the weighting assigned to the first, second, and third accuracy levels for daytime and nighttime determination criteria. For example, during the day, it prioritizes candidates related to the color and shape of the buoy's top mark that are easily visible in bright environments, while at night, it prioritizes candidates related to the way the buoy glows that are easily visible even in dark environments.

[0135] In other words, in the daytime judgment criteria, the weighting of the first and second degrees of certainty related to the color of the buoy and the shape of the top mark is higher than the weighting of the third degree of certainty related to how the buoy glows. Conversely, in the nighttime judgment criteria, the weighting of the third degree of certainty related to how the buoy glows is higher than the weighting of the first and second degrees of certainty related to the color of the buoy and the shape of the top mark.

[0136] Furthermore, the method for determining the content of signs in this modified example may be applied not only to side signs and directional signs, but also to isolated obstacle signs, safe water area signs, and special signs.

[0137] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications are of course possible for those skilled in the art. [Explanation of Symbols]

[0138] 1 Navigation aid identification device, 2 Camera, 3 Radar, 4 AIS, 5 Radio communication unit, 6 Display unit, 7 GNSS receiver, 8 Gyrocompass, 9 ECDIS, 10 Automatic steering system, 11 First acquisition unit, 12 First identification unit, 13 Second acquisition unit, 14 Second identification unit, 15 Display control unit, 16 Route calculation unit, 17 Model memory unit, 21 Lens unit, 22 Pan / tilt mechanism, 31 Color identification unit, 32 Shape identification unit, 33 Country determination unit, 34 Port / Starboard determination unit, 41 Color identification unit, 42 First candidate determination unit, 43 Shape identification unit, 44 Second candidate determination unit, 45 Illumination pattern identification unit, 46 Third candidate determination unit, 47 Marker content determination unit, 100 Autonomous navigation system

Claims

1. An acquisition unit that acquires images generated by a camera installed on a ship, A mode identification unit that identifies whether the mode of the side buoy included in the aforementioned image is the first mode or the second mode, A country determination unit determines the country to which the position detected by the position detection unit for detecting the position of the aforementioned vessel belongs. A port / starboard determination unit determines whether the marking content of the side buoy is a port / starboard marking or a starboard marking, based on a predetermined correspondence relationship indicating whether the first and second embodiments correspond to port / starboard markings in each country, and based on the shape of the side buoy and the determined country. A navigational aid identification device equipped with the following features.

2. The aspect identification unit identifies whether the color of the side buoy is green or red. The navigation aid identification device according to claim 1.

3. The aforementioned shape identification unit identifies whether the top mark of the side buoy is cylindrical or conical. A navigational aid identification device according to claim 1 or 2.

4. The system further includes a type identification unit for identifying the type of buoy in the aforementioned image, The mode identification unit identifies the mode of the side buoy when the type of the buoy is a side buoy. A navigation aid identification device according to any one of claims 1 to 3.

5. The system further includes a display control unit that displays a symbol representing the marking content of the side buoy determined by the port / starboard determination unit on the image, electronic chart, or radar image, based on the marking content of the side buoy determined by the port / starboard determination unit, the position of the side buoy in the image, and the imaging direction of the camera. A navigation aid identification device according to any one of claims 1 to 4.

6. The system further includes a consistency determination unit that determines the consistency between the marker content of the side buoy determined by the port / starboard determination unit and the marker content represented by the navigation aid data recorded in the electronic chart, based on the marker content of the side buoy determined by the port / starboard determination unit, the position of the side buoy in the image, the imaging direction of the camera, and the position of the vessel. A navigation aid identification device according to any one of claims 1 to 5.

7. The system further includes a display control unit that displays the consistency determination result on the image, the electronic chart, or the radar image. The navigation aid identification device according to claim 6.

8. A navigational aid identification device according to claim 4, wherein the type identification unit further identifies that the form of the buoy included in the image is a safe waters marker, If the buoys identified in the image include a port side marker and a starboard side marker determined by the port / starboard determination unit, or if they include a port side marker or a starboard side marker determined by the port / starboard determination unit and a safe water area marker identified by the type identification unit, a route calculation unit calculates the ship's route or route width based on the position of the buoys in the image and the imaging direction of the camera, An autonomous navigation system equipped with [the following features].

9. A navigation aid identification device according to any one of claims 1 to 7, A virtual sign acquisition unit acquires data representing the position and content of a virtual sign, A route calculation unit calculates the ship's route or route width based on the marking content of the side buoy determined by the port / starboard determination unit, the position of the virtual marking, and the marking content of the virtual marking; An autonomous navigation system equipped with [the following features].

10. A navigation aid identification device according to any one of claims 1 to 7, A position detection unit for detecting the position of the aforementioned vessel, A route calculation unit sets the turning points that the vessel should pass through, based on the marking content of the side buoy determined by the port and starboard determination unit, the position of the side buoy in the image, the imaging direction of the camera, and the position of the vessel. An autonomous navigation system equipped with [the following features].

11. A navigation aid identification device according to any one of claims 1 to 7, A direction detection unit for detecting the bow direction of the aforementioned vessel, A route calculation unit sets the direction in which the vessel should navigate based on the markings of the side buoys determined by the port and starboard determination unit, the imaging direction of the camera, and the bow bearing of the vessel. An autonomous navigation system equipped with [the following features].

12. The vehicle further includes an automatic steering system that performs autonomous navigation control based on the markings of the side buoys determined by the port and starboard determination unit. An autonomous navigation system according to any one of claims 8 to 11.

13. By computer, Images generated by cameras installed on the ship are acquired. Identify whether the side buoy included in the aforementioned image is of the first or second form, The position detection unit that detects the position of the aforementioned vessel determines the country to which the detected position belongs. Based on a predetermined correspondence indicating whether the first and second embodiments correspond to port markings or starboard markings in each country, it is determined from the form of the side buoy and the determined country whether the marking content of the side buoy is a port marking or a starboard marking. Navigation aid identification method.

14. To acquire images generated by cameras installed on ships, Identify whether the side buoy included in the aforementioned image is of the first or second form. The position detection unit that detects the position of the aforementioned vessel determines the country to which the detected position belongs, and Based on a predetermined correspondence indicating whether the first and second embodiments correspond to port markings or starboard markings in each country, it is determined from the shape of the side buoy and the determined country whether the marking content of the side buoy is a port marking or a starboard marking. A program that causes a computer to execute something.

Images