Navigation aid identification device, autonomous navigation system, navigation aid identification method, and program

Abstract

[Problem] To provide a sea mark identification device that can improve the accuracy of identifying mark content. [Solution] A sea mark identification device comprises: an acquisition unit that acquires an image generated by a camera installed on a ship; a color identification unit that identifies a color candidate of a buoy contained in the image; a shape identification unit that identifies a shape candidate of a top mark of the buoy; a light pattern identification unit that identifies a light pattern candidate of the buoy from a plurality of images in a time series; and a mark content decision unit that, on the basis of a first candidate for mark content of the buoy corresponding to the color candidate, a second candidate for mark content of the buoy corresponding the shape candidate, and a third candidate for mark content of the buoy corresponding to the light pattern candidate, decides the mark content of the buoy.

Description

Technical Field

[0001] The present invention relates to a route sign identification device, an autonomous navigation system, a route sign identification method, and a program.

Background Art

[0002] Patent Document 1 discloses an automatic visual recognition device that automatically identifies route signs. The document describes that if the extracted information on the color, shape, and light emission time of a route sign matches predetermined data, the route sign is determined.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when the extraction accuracy of color, shape, and light emission time is not sufficient, there is a risk that the sign content cannot be identified if a match is made a condition.

[0005] The present invention has been made in view of the above problems, and its main object is to provide a route sign identification device, an autonomous navigation system, a route sign identification method, and a program capable of improving the identification accuracy of sign content.

Means for Solving the Problems

[0006] To solve the above problems, a navigation aid identification device according to one aspect of the present invention includes: an acquisition unit that acquires an image generated by a camera installed on a ship; a color identification unit that identifies candidate colors of buoys included in the image; a shape identification unit that identifies candidate shapes of the top mark of the buoy; a light-emitting type identification unit that identifies candidate ways of illuminating the buoy from a plurality of images in a time series; and a sign content determination unit that determines the sign content of the buoy based on a first candidate sign content of the buoy corresponding to the color candidate, a second candidate sign content of the buoy corresponding to the shape candidate, and a third candidate sign content of the buoy corresponding to the light-emitting type candidate.

[0007] In the above embodiment, the mark content determination unit may determine that at least two of the first candidate, second candidate, and third candidate have the same mark content as the mark content of the buoy.

[0008] In the above embodiment, the color identification unit may calculate a first probability representing the likelihood of the color candidate together with the color candidate; the shape identification unit may calculate a second probability representing the likelihood of the shape candidate together with the shape candidate; the light emission identification unit may calculate a third probability representing the likelihood of the light emission candidate together with the light emission candidate; and the marking content determination unit may determine the marking content of the buoy based on the sum of the first probability, the second probability, and the third probability.

[0009] In the above embodiment, the mark content determination unit may change the determination criteria for determining the mark content of the buoy according to the time the image was generated.

[0010] In the above embodiment, the sign content determination unit may change the weighting assigned to the first accuracy, second accuracy, and third accuracy, respectively, according to the time the image was generated.

[0011] In the above embodiment, the color identification unit may use a trained model to identify color candidates for the buoys in the image.

[0012] In the above embodiment, the shape recognition unit may use a trained model to identify candidate shapes for the top marks of the buoys in the image.

[0013] In the above embodiment, the light emission identification unit may identify candidate light emission patterns for the buoy according to a predetermined rule.

[0014] In the above embodiment, the display control unit may further include a display control unit that displays a symbol representing the buoy's marker content on the first image, an electronic chart, or a radar image based on the marker content of the buoy, the position of the buoy in the first image, and the imaging direction of the camera.

[0015] In the above embodiment, the system may further include a consistency determination unit that determines the consistency between the marker content of the buoy and the marker content represented by the navigation aid data recorded in the electronic chart, based on the marker content of the buoy, the position of the buoy in the first image, the imaging direction of the camera, and the position of the vessel.

[0016] In the above embodiment, the system may further include a display control unit that displays the consistency determination result on the first image, the electronic chart, or the radar image.

[0017] Furthermore, an autonomous navigation system in another aspect of the present invention may include the above-described navigation aid identification device and a navigation route calculation unit that calculates the ship's navigation route or navigation route width based on the position of the buoys in the first image and the imaging direction of the camera, when the markings of the plurality of buoys include at least two of the port side markings, starboard side markings, and safe waters markings.

[0018] Furthermore, an autonomous navigation system in another aspect of the present invention may include the above-mentioned navigation aid identification device, a virtual marker acquisition unit that acquires data representing the position and content of a virtual marker, and a route calculation unit that calculates the ship's route or route width based on the content of the buoy, the position of the virtual marker, and the content of the virtual marker.

[0019] Furthermore, an autonomous navigation system in another aspect of the present invention may include the above-mentioned navigation aid identification device, a position detection unit for detecting the position of the vessel, and a route calculation unit for setting a course turning point that the vessel should pass through based on the markings of the buoy, the position of the buoy in the first image, the imaging direction of the camera, and the position of the vessel.

[0020] Furthermore, an autonomous navigation system in another aspect of the present invention may include the above-mentioned navigation aid identification device, a direction detection unit for detecting the heading of the vessel, and a route calculation unit for setting the direction in which the vessel should navigate based on the markings of the buoy, the imaging direction of the camera, and the heading of the vessel.

[0021] Furthermore, in the above embodiment, an automatic steering system that performs autonomous navigation control based on the markings on the buoy may be further provided.

[0022] Furthermore, another embodiment of the present invention provides a method for identifying navigational aids, which involves acquiring an image generated by a camera installed on a ship, identifying candidate colors of the buoys included in the image, identifying candidate shapes of the top marks of the buoys, identifying candidate ways of illuminating the buoys from a plurality of images in a time series, and determining the marker content of the buoys based on a first candidate for the marker content of the buoys corresponding to the color candidates, a second candidate for the marker content of the buoys corresponding to the shape candidates, and a third candidate for the marker content of the buoys corresponding to the illuminating style candidates.

[0023] Furthermore, a program in another aspect of the present invention causes a computer to perform the following actions: acquire images generated by a camera installed on a ship; identify color candidates for buoys included in the images; identify shape candidates for the top mark of the buoys; identify light-reflecting candidates for the buoys from a plurality of images in a time series; and determine the marking content of the buoys based on a first candidate for the marking content of the buoys corresponding to the color candidates, a second candidate for the marking content of the buoys corresponding to the shape candidates, and a third candidate for the marking content of the buoys corresponding to the light-reflecting candidates. [Effects of the Invention]

[0024] According to the present invention, it is possible to improve the identification accuracy of the sign content.

Brief Description of the Drawings

[0025] [Figure 1] It is a block diagram showing a configuration example of a shipborne system. [Figure 2] It is a diagram for explaining the sign content of a buoy and the like. [Figure 3] It is a block diagram showing a functional configuration example of a route sign identification device. [Figure 4] It is a diagram showing an example of a first image. [Figure 5] It is a diagram showing an example of identification by a first identification unit. [Figure 6] It is a diagram showing an example of a second image. [Figure 7] It is a flowchart showing an example of the procedure of a route sign identification method. [Figure 8] It is a flowchart showing an example of the procedure of a sign content identification process. [Figure 9] It is a diagram showing an example of a buoy management database. [Figure 10] It is a diagram showing an example of display by a display unit. [Figure 11] It is a diagram showing another example of display by a display unit. [Figure 12] It is a block diagram showing another configuration example of a route sign identification device. [Figure 13] It is a block diagram showing yet another configuration example of a route sign identification device. [Figure 14] It is a block diagram showing a configuration example of a second identification unit according to a first modification example. [Figure 15] It is a flowchart showing an example of the procedure of a left / right side sign identification process. [Figure 16] It is a diagram showing an example of a country-specific left / right side mode table. [Figure 17] It is a block diagram showing a configuration example of a second identification unit according to a second modification example. [Figure 18] It is a flowchart showing an example of the procedure of a sign content identification process. [Figure 19]This is a diagram to explain how buoys reflect light. [Modes for carrying out the invention]

[0026] Embodiments of the present invention will be described below with reference to the drawings.

[0027] Figure 1 is a block diagram showing an example configuration of the autonomous navigation system 100. The autonomous navigation system 100 is an ICT system installed on a ship. Hereinafter, a ship equipped with the autonomous navigation system 100 will be referred to as "our ship".

[0028] The autonomous navigation system 100 includes a navigation aid identification device 1, a camera 2, a radar 3, an AIS 4, a radio communication unit 5, a display unit 6, a GNSS receiver 7, a gyrocompass 8, an ECDIS 9, and an automatic steering system 10. These devices are connected to a network N, such as a LAN, and are capable of network communication with each other.

[0029] The navigation aid identification device 1 is a computer that includes a CPU, RAM, ROM, non-volatile memory, and input / output interfaces. The CPU of the navigation aid identification device 1 performs information processing according to a program loaded from ROM or non-volatile memory into RAM.

[0030] The program may be supplied via an information storage medium such as an optical disc or memory card, or via a communication network such as the Internet or LAN.

[0031] Camera 2 is a digital camera that captures images of the exterior of the vessel and generates image data. Camera 2 is installed, for example, on the bridge of the vessel, facing the bow. Camera 2 is a visible light camera capable of capturing images in at least the visible range. It may also be capable of capturing images in the infrared range.

[0032] In this embodiment, camera 2 is a camera having pan-tilt and optical zoom functions, a so-called PTZ camera. Camera 2 pans, tilts, or zooms in response to commands from the navigation aid identification device 1.

[0033] Radar 3 emits radio waves around the vessel and receives the reflected waves, generating echo data based on the received signals. Radar 3 also identifies targets from the echo data and generates target tracking data (TT data) that represents the position and speed of the targets.

[0034] The Automatic Identification System (AIS) 4 receives AIS data from other vessels or shore-based control systems in the vicinity of the vessel. While AIS is not the only option, a VHF Data Exchange System (VDES) may also be used. The AIS data includes information such as the position and speed of other vessels.

[0035] Furthermore, AIS4 may acquire AIS data representing the position and content of virtual markers. AIS4 is an example of a virtual marker acquisition unit. Virtual markers using AIS are so-called virtual AIS navigational aids.

[0036] The radio communication unit 5 includes various radio equipment for enabling communication with other ships or land-based air traffic control, such as radio equipment for the very high frequency band, medium high frequency band, and short frequency band.

[0037] The display unit 6 is, for example, a display device with a touch sensor, a so-called touch panel. The display unit device may include, for example, a liquid crystal display device or an organic EL display device. It is not limited to a touch sensor; other pointing devices such as a trackball or mouse may also be used.

[0038] The display unit 6 displays images captured by the camera 2, radar images generated by the radar 3, electronic charts, or composite images created by combining radar images and electronic charts.

[0039] The GNSS receiver 7 detects the ship's position based on radio waves received from the GNSS (Global Navigation Satellite System). The GNSS receiver 7 is an example of a position detection unit that detects the ship's position.

[0040] The gyrocompass 8 detects the ship's heading. The gyrocompass 8 is an example of a heading detection unit that detects the ship's heading. Other types of compasses, such as GPS compasses, may be used instead of a gyrocompass.

[0041] The ECDIS (Electronic Chart Display and Information System) 9 obtains the ship's position from the GNSS receiver 7 and displays the ship's position on the electronic chart. The ECDIS 9 also displays the ship's planned route on the electronic chart. A GNSS plotter may be used instead of the ECDIS.

[0042] The automatic steering system 10 calculates a target rudder angle to point the bow toward the target course based on the target course obtained from the navigation aid identification device 1 and the heading obtained from the gyrocompass 8, and drives the steering gear so that the rudder angle approaches the target rudder angle. The automatic steering system 10 may also control the engine.

[0043] In this embodiment, the navigation aid identification device 1 is an independent device, but it is not limited to this and may be integrated with other devices such as ECDIS9. In other words, the functions of the navigation aid identification device 1 may be realized by other devices such as ECDIS10.

[0044] Furthermore, although the display unit 2 is an independent device in this embodiment, it is not limited to this, and a display unit provided by another device such as ECDIS9 may be used as the display unit 2 that displays the image generated by the navigation aid identification device 1.

[0045] Figure 2 is a diagram illustrating the markings on buoys. Buoys are navigational aids floating on the sea surface, also known as buoys. The types and markings of buoys are classified by their color and the shape of their top mark.

[0046] Buoys are categorized into side markers, azimuth markers, isolated obstacle markers, safe water markers, and special markers. Side markers include port markers and starboard markers. Port and starboard refer to left and right when facing the source of the water. Azimuth markers include north azimuth markers, east azimuth markers, south azimuth markers, and west azimuth markers.

[0047] For isolated obstacle markers, safe water area markers, and special markers, the marker content is not further subdivided. In other words, the type of buoy itself can be said to represent the marker content.

[0048] Incidentally, buoys floating on the sea surface are small compared to ships, making it difficult to identify the markings on buoys that are far away from ships. Therefore, in this embodiment, the accuracy of identifying the markings is improved by acquiring images in stages, as described below.

[0049] Figure 3 is a block diagram showing an example of the functional configuration of a navigation aid identification device 1 according to an embodiment. The navigation aid identification device 1 includes a first acquisition unit 11, a first identification unit 12, a second acquisition unit 13, a second identification unit 14, a display control unit 15, a route calculation unit 16, and a consistency determination unit 18.

[0050] These functional units are realized by the CPU of the navigation aid identification device 1 performing information processing according to a program. Note that some functional units, such as the display control unit 15 or the route calculation unit 16, may be realized by a separate computer, such as the ECDIS 9 or the automatic steering system 10, which is separate from the navigation aid identification device 1.

[0051] Furthermore, the navigation aid identification device 1 includes a model storage unit 17 for storing learned models. This storage unit is located in the non-volatile memory of the navigation aid identification device 1. However, the model storage unit 17 may be located outside the navigation aid identification device 1.

[0052] The first acquisition unit 11 acquires the first image generated by the camera 2. Specifically, the first acquisition unit 11 sequentially acquires multiple time-series first images generated by the camera 2 and provides them sequentially to the first identification unit 12.

[0053] The first image is an image captured when camera 2 is in its standard state. The standard state is, for example, when the optical zoom magnification is at its minimum and the imaging direction is facing the bow of the ship. Except for the period controlled by the second acquisition unit 13, camera 2 repeatedly generates the first image in the standard state.

[0054] The multiple first images in the time series may be, for example, multiple still images (frames) included in a video, or multiple still images individually generated by capturing images at predetermined time intervals.

[0055] Figure 4 shows an example of a first image P1 acquired by the first acquisition unit 11. This figure shows an example in which the first image P1 includes the ship's hull SP along with the port side marker LL and starboard side marker LR floating on the sea surface in front of the ship.

[0056] The first identification unit 12 identifies the position of the buoy in the first image P1. Specifically, the first identification unit 12 identifies the position of the buoy in the first image P1 using the first trained model stored in the model storage unit 17. In addition, the first identification unit 12 may further identify the type of buoy along with its position in the first image P1.

[0057] The first pre-trained model is generated by machine learning using training images as input data and the labels (or labels indicating the type of buoy) and locations of buoys within the training images as training data. The first pre-trained model thus generated estimates the labels (or labels indicating the type of buoy), locations, and accuracy of buoys within the first image P1. The location of a buoy is represented, for example, by the coordinates of the bounding box surrounding the buoy.

[0058] The first pre-trained model may be an object detection model such as SSD (Single Shot MultiBox Detector), YOLO (You Only Look Once), or Mask R-CNN. However, the first pre-trained model may also be a region segmentation model such as Semantic Segmentation or Instance Segmentation, or a feature point detection model such as Keypoint Detection.

[0059] Figure 5 shows an example of the identification of the first image P1 by the first identification unit 12. The figure shows an example in which the port side marker LL and the starboard side marker LR are identified as buoys (or side buoys) and enclosed by a boundary box BB.

[0060] The second acquisition unit 13 acquires a second image with higher resolution than the first image P1, corresponding to a subregion of the first image P1 that includes the position of the buoy. The subregion is, for example, a bounding box BB (see Figure 5) identified by the first identification unit 12.

[0061] Camera 2 is equipped with a lens unit 21 that provides an optical zoom function and a pan-tilt mechanism 22 that provides a pan-tilt function, and the second acquisition unit 13 acquires a second image by controlling the lens unit 21 and the pan-tilt mechanism 22 of Camera 2.

[0062] Specifically, the second acquisition unit 13 controls the lens unit 21 to cause the camera 2 to expand the real-space range corresponding to a sub-region of the first image P1 and capture the image, thereby acquiring the second image. By utilizing this optical zoom function, a second image with higher resolution than the first image P1 is acquired.

[0063] Furthermore, the second acquisition unit 13 controls the pan-tilt mechanism 22 to orient the imaging direction of the camera 2 to a real-space range corresponding to a sub-region of the first image P1. The second acquisition unit 13 sets a target value for the imaging direction of the camera 2 according to the position of the buoy in the first image P1 identified by the first identification unit 12.

[0064] Figure 6 shows an example of a second image P2 acquired by the second acquisition unit 13. This figure shows an example where the second image P2 includes a port side marker LL. In the second image P2, the color of the port side marker LL and the shape of the top mark TM are easier to identify than in the first image P1 (see Figure 4).

[0065] As shown in Figure 5 above, if multiple buoys (in the illustrated example, port marker LL and starboard marker LR) are identified in the first image P1, the second acquisition unit 13 will have the camera 2 sequentially image each of the multiple buoys, thereby acquiring a second image P2 for each of the multiple buoys.

[0066] The second identification unit 14 identifies the markings of the buoy from the second image P2. Specifically, the second identification unit 14 uses the second trained model stored in the model storage unit 17 to identify the markings of the buoy from the second image P2.

[0067] The second pre-trained model is generated using machine learning, with the training images as input data and the labels of the buoy markings within the training images as training data. This generated second pre-trained model estimates the labels and accuracy of the buoy markings within the second image P2.

[0068] For the second pre-trained model, for example, a model of the same type as the first pre-trained model described above may be used. In this case, the first and second pre-trained models will each have different first and second pre-trained parameters incorporated into a common inference program.

[0069] This is not the only example; the second pre-trained model may be an object recognition model that only identifies objects and does not detect their positions.

[0070] Furthermore, the second pre-trained model may include a pre-trained model for side signs that specializes in identifying the content of side signs, and a pre-trained model for directional signs that specializes in identifying the content of directional signs.

[0071] Figures 7 and 8 are flowcharts illustrating an example of the procedure for identifying navigational aids implemented by the navigational aid identification device 1. These figures primarily show the processes performed by the navigational aid identification device 1, specifically the processes related to image acquisition and identification of the aid's content.

[0072] The CPU of the navigation aid identification device 1 functions as a first acquisition unit 11, a first identification unit 12, a second acquisition unit 13, and a second identification unit 14 by executing the information processing shown in these figures according to the program.

[0073] As shown in Figure 7, first, the navigation aid identification device 1 acquires a first image P1 (see Figure 4) from the camera 2 (S11: Processing as the first acquisition unit 11).

[0074] Next, the navigation aid identification device 1 uses the first trained model to identify the position and type of buoy in the first image P1 (S12: Processing as the first identification unit 12).

[0075] Next, the navigation aid identification device 1 determines whether the type of buoy identified in the first image P1 is a side buoy or a bearing buoy (S13).

[0076] If the type of buoy is a lateral buoy or a bearing buoy (S13 → YES), the navigation aid identification device 1 controls the camera 2 to acquire a second image P2 (see Figure 6) which is an enlarged image of the buoy (S14: Processing as the second acquisition unit 13).

[0077] Next, the navigation aid identification device 1 performs an indicator content identification process to identify the indicator content of the buoy from the second image P2 (S15: processing as the second identification unit 14).

[0078] As shown in Figure 8, in the marker content identification process S15, if the type of buoy is a side buoy (S21 → side buoy), the navigational aid identification device 1 uses a second learned model for side buoys to identify whether the marker content is a port buoy or a starboard buoy (S22).

[0079] On the other hand, if the type of buoy is a azimuth buoy (S21 → azimuth buoy), the navigation aid identification device 1 uses a trained model for azimuth buoys as a second trained model to identify whether the buoy is a north azimuth buoy, east azimuth buoy, south azimuth buoy, or west azimuth buoy (S23).

[0080] Furthermore, if the buoy is not a side buoy or a bearing buoy (S13→NO), that is, if the buoy is an isolated obstacle marker, a safe water area marker, or a special marker, the navigation aid identification device 1 will not acquire the second image P2. This is because, in the case of those buoys, the type itself represents the marker content.

[0081] If multiple buoys are identified in the first image P1 in S12, the navigation aid identification device 1 performs S13 to S15 for all identified buoys (S16). That is, for all buoys that are either lateral buoys or azimuth buoys, the device acquires the second image P2 and identifies the marker content.

[0082] According to the embodiment described above, the mark content is identified from a second image P2, which has a higher resolution than the first image P1 and is imaged after being magnified based on the position of the buoy identified in the first image P1. This makes it possible to improve the accuracy of identifying the mark content.

[0083] Furthermore, according to this embodiment, the type of buoy is identified in the first image P1 and then the marking content is identified from the second image P2. This allows the marking content to be narrowed down according to the type of buoy, thereby further improving the accuracy of marking content identification.

[0084] The method is not limited to this; the buoy and its location may be identified from the first image P1, and the type of buoy and the markings on it may be identified from the second image P2.

[0085] Figure 9 shows an example of a buoy management database. The buoy management database is a database for managing information on identified or acquired buoys and is stored in the non-volatile memory of the navigation aid identification device 1. The buoy management database includes not only information on buoys identified from images from camera 2, but also information on virtual markers acquired by AIS4.

[0086] The buoy management database includes fields such as "Identifier," "Type," "Marking Content," "Location in Image," "Actual Location," and "Virtual Buoy." The "Identifier" is an identifier used to identify a buoy. "Virtual Buoy" indicates whether or not it is a virtual buoy.

[0087] "Type" indicates the type of buoy. "Marking Content" indicates the marking content of the buoy. If "Type" is a side marker or azimuth marker, then "Marking Content" will be entered as a port marker or north azimuth marker, etc. On the other hand, if "Type" is an isolated obstacle marker, safe water marker, or special marker, no data will be entered in "Marking Content".

[0088] "Image Position" represents the buoy's position within the first image P1 (see Figure 4). Note that for virtual buoys, no data is entered in "Image Position". "Actual Position" represents the actual position of the buoy. The actual position of a buoy identified from the image of camera 2 is calculated based on the buoy's image position and the imaging direction of camera 2.

[0089] Returning to the explanation of Figure 3, the display control unit 15 generates display data related to the buoy and outputs it to the display unit 6. Specifically, the display control unit 15 displays a symbol representing the buoy's markings on the first image P1, an electronic chart, or a radar image, etc., based on the markings of the identified buoy, the position of the buoy in the first image P1, and the imaging direction of the camera 2.

[0090] For example, as shown in Figure 10, the display control unit 15 displays an image on the display unit 6 in which symbols ML and MR, representing the contents of the port side marker LL and starboard side marker LR, are attached in association with their positions in the first image P1. The symbols ML and MR include, for example, strings of characters representing the contents of the markers.

[0091] Furthermore, as shown in Figure 11, the display control unit 15 displays an image on the display unit 6 in which symbols TL and TR, representing the contents of the port side marker LL and starboard side marker LR, are attached to positions corresponding to the actual positions of the port side marker LL and starboard side marker LR within the composite image CP, which is a composite image obtained by combining the electronic chart and the radar image. The symbols TL and TR have, for example, shapes that represent the contents of the markers.

[0092] The composite image CP displays the ship's symbol SF, its planned route RT, the point of change of course DF on the planned route RT, and the symbols EL of other ships, etc.

[0093] Furthermore, symbols VL and VR representing the content of the virtual marker may be displayed in the composite image CP. Symbols VL and VR have the same shape as symbols TL and TR. It is preferable that symbols VL and VR are displayed in a way that makes them distinguishable from symbols TL and TR, for example, by changing their transparency.

[0094] The route calculation unit 16 calculates a target course, i.e., bearing, turning point, and course, for autonomous navigation control based on the identified buoy markings. The calculated target course is provided to the automatic steering system 10, which performs autonomous navigation control. Here, misrecognition of navigation markings can cause serious accidents when performing autonomous navigation control. Therefore, by performing autonomous navigation control by identifying navigation markings with improved accuracy according to the present invention, an autonomous navigation system that can withstand operation in real environments can be realized.

[0095] As shown in Figure 5, the route calculation unit 16 calculates the planned route or route width of the vessel based on the positions of the port side marker LL and the starboard side marker LR in the first image P1 and the imaging direction of camera 2, when the buoys identified in the first image P1 include the port side marker LL and the starboard side marker LR. Specifically, the route calculation unit 16 sets the planned route RT of the vessel so that it passes between the port side marker LL and the starboard side marker LR from the vessel's position, based on the actual positions of the port side marker LL and the starboard side marker LR calculated from the positions of the port side marker LL and the starboard side marker LR in the first image P1 and the imaging direction of camera 2 (see Figure 11). In addition, if the buoy identified in the first image P1 includes a port marker LL or starboard marker LR and a safe waters marker, the route calculation unit 16 may set the vessel's planned route RT between the port marker LL or starboard marker LR and the safe waters marker.

[0096] Furthermore, the route calculation unit 16 may calculate the distance between the port side marker LL and the starboard side marker LR as the route width W, based on the actual positions of the port side marker LL and the starboard side marker LR calculated from the positions of the port side marker LL and the starboard side marker LR in the first image P1 and the imaging direction of the camera 2. The calculated route width W may be displayed in the first image P1 displayed on the display unit 6, or it may be displayed in the composite image CP which is a composite of the electronic chart and the radar image (see Figure 11).

[0097] The route calculation unit 16 may set turning points that the vessel should pass through based on the marking content of the identified buoy, the position of the buoy in the first image P1, the imaging direction of the camera 2, and the vessel's own position. Specifically, the route calculation unit 16 sets one or more turning points DF for setting the planned route RT for the vessel entering or leaving a port, based on the marking content of the identified side markers and bearing buoys, the actual position of those buoys calculated from the position of those buoys in the first image P1 and the imaging direction of the camera 2, and the vessel's own position (see Figure 11). Not limited to this, the route calculation unit 16 may also set one or more turning points for setting an avoidance route to avoid obstacles or special areas, based on the marking content of the identified isolated obstacle markers or special markers, the actual position of those buoys calculated from the position of those buoys in the first image P1 and the imaging direction of the camera 2, and the vessel's own position.

[0098] The route calculation unit 16 may set the direction the ship should navigate based on the markings of the identified buoys, the imaging direction of the camera 2, and the ship's heading. For example, the route calculation unit 16 maintains or adjusts the direction the ship should navigate so that buoys such as side buoys continue to be included in multiple time-series first images P1. The route calculation unit 16 may also use the positions of the buoys in the first images P1 to set the direction the ship should navigate so that it heads towards the space between the port and starboard markers, or in a direction along multiple port or starboard markers.

[0099] The route calculation unit 16 may calculate a target course, i.e., bearing, turning points, and route, for autonomous navigation control based on the position and content of virtual markers, in addition to the marking content of the identified buoys. Specifically, when data for virtual port marker VL and virtual starboard marker VR is acquired, the route calculation unit 16 may set the planned route RT of its own vessel so that it passes not only between port marker LL and starboard marker LR identified in the first image P1, but also between virtual port marker VL and virtual starboard marker VR.

[0100] The consistency determination unit 18 determines the consistency between the buoy's marker content and the marker content represented by the navigational aid data recorded in the electronic chart, based on the identified buoy's marker content, the buoy's position in the first image P1, the imaging direction of the camera 2, and the ship's position. Specifically, the consistency determination unit 18 calculates the actual position of the buoy from the buoy's position in the first image P1, the imaging direction of the camera 2, and the ship's position, and extracts navigational aid data corresponding to the buoy's actual position from the navigational aid data recorded in the electronic chart. It then determines whether the identified buoy's marker content matches the marker content represented by the extracted navigational aid data.

[0101] The display control unit 15 displays the determination result from the alignment determination unit 18 on the first image P1, electronic chart, or radar image, etc. For example, the display control unit 15 displays a symbol indicating whether or not alignment is achieved, associated with the buoy, on the first image P1 (see Figure 10) or composite image CP (see Figure 11), etc., displayed on the display unit 6. Alternatively, the display control unit 15 may display a symbol representing the marking content (symbols ML, MR in Figure 10 or symbols TL, TR in Figure 11, etc.) only on buoys that have been aligned.

[0102] The configuration of the navigation aid identification device 1 is not limited to the example shown in Figure 3. For example, as shown in Figure 12, the second acquisition unit 13A may be an image processing unit that acquires a second image by increasing the resolution of a portion of the first image. By performing this resolution increase, a second image with higher resolution than the first image is acquired.

[0103] The system is not limited to this, however, the first acquisition unit 11 may acquire a first image by performing a decimation or averaging process on the original image generated by the camera 2, and the second acquisition unit 13 may acquire a second image by cutting out a region corresponding to a sub-region of the first image from the original image. In this case, a second image with higher resolution than the first image will also be acquired.

[0104] Furthermore, as shown in Figure 13, the second acquisition unit 13B may be a camera control unit that acquires a second image by having an auxiliary camera 3, which has a higher resolution than camera 2, capture an area of ​​real space corresponding to a sub-region of the first image. By using the auxiliary camera 3 in this way, a second image with a higher resolution than the first image is acquired.

[0105] The auxiliary camera 3, like camera 2 shown in Figure 3 above, is equipped with a lens unit 31 that provides optical zoom functionality and a pan-tilt mechanism 32 that provides pan-tilt functionality. The lens unit 31 of the auxiliary camera 3 has a higher magnification than the lens unit 21 of camera 2.

[0106] [First variation] The first modified example is described below. Configurations or processes that overlap with the above embodiment may be denoted by the same reference numerals, and detailed explanations may be omitted.

[0107] The interpretation of port and starboard buoys can differ from country to country. Therefore, in this modified example, port and starboard buoys are distinguished regardless of the ship's position, as explained below.

[0108] Figure 14 is a block diagram showing an example configuration of the second identification unit 14A according to the first modified example. The figure mainly shows the functional unit for identifying the marking content of the side buoy, which is one of the functional units realized in the second identification unit 14A.

[0109] The second identification unit 14A includes a color identification unit 31, a shape identification unit 32, a country determination unit 33, and a port / starboard determination unit 34. The color identification unit 31 and the shape identification unit 32 are examples of the form identification unit.

[0110] When the type of buoy identified by the first identification unit 12 (type identification unit) shown in Figure 3 is a lateral buoy, the functional unit of the second identification unit 14A identifies the markings of the lateral buoy included in the second image P2 (see Figure 6).

[0111] Figure 15 is a flowchart showing an example of the procedure for the port and starboard marker identification process S22 according to the first modified example, which is implemented by the second identification unit 14A. The navigation marker identification device 1 performs the information processing shown in the figure according to the program.

[0112] The port and starboard buoy identification process S22 corresponds to S22 shown in Figure 8 above. That is, the port and starboard buoy identification device 1 executes the port and starboard buoy identification process S22 when the type of buoy identified in S12 shown in Figure 7 above is a side buoy.

[0113] First, the navigation aid identification device 1 identifies whether the side buoy included in the second image P2 is green or red (S31: processing as color identification unit 31). Green and red are examples of the first and second embodiments.

[0114] Next, the navigation aid identification device 1 identifies whether the top mark of the side buoy included in the second image P2 is cylindrical or conical (S32: processing as the shape identification unit 32). The cylindrical and conical shapes are examples of the first and second embodiments.

[0115] Color identification and top mark shape identification are performed using a trained model, similar to the embodiment described above. For example, a trained model that identifies both color and top mark shape may be used, or a trained model that identifies color and a trained model that identifies top mark shape may be used separately.

[0116] Next, the navigation aid identification device 1 determines the country to which the detected position of its own vessel, as detected by the GNSS receiver 7 (see Figure 1), belongs (S33: Processing as the country determination unit 33). For example, the navigation aid identification device 1 determines, based on chart data, which country's territorial waters the coordinates of its own vessel's detected position are located.

[0117] Next, the navigation aid identification device 1 refers to the country-specific port and starboard configuration table and determines whether the markings on the side buoy are port or starboard based on the color identified in S31, the shape of the top mark identified in S32, and the country determined in S33 (S34: Processing as port / starboard determination unit 34).

[0118] Figure 16 shows an example of a country-specific port and starboard configuration table. The country-specific port and starboard configuration table is a table that shows the correspondence between the configuration of a buoy and the content of the marker, and is stored in the non-volatile memory of the navigation aid identification device 1.

[0119] Specifically, the country-specific port / starboard configuration table indicates whether the green and red colors of the side buoys correspond to port or starboard markers in each country. Furthermore, the country-specific port / starboard configuration table also indicates whether the cylindrical and conical shapes of the top marks correspond to port or starboard markers in each country.

[0120] According to the first modified example described above, it becomes possible to distinguish between port and starboard markers based on the markings on the side buoys, regardless of the ship's position.

[0121] [Second variation] The following describes a second modified example. Configurations or processes that overlap with the above embodiment may be denoted by the same reference numerals, and detailed explanations may be omitted.

[0122] The type and markings of buoys can be identified by elements such as the buoy's color, the shape of its top mark, and how it reflects light. However, when identifying markings directly from an image, the contribution of each element is unclear, and the identification accuracy may be insufficient. Therefore, this modified version aims to improve the identification accuracy of markings as described below.

[0123] Figure 17 is a block diagram showing an example configuration of the second identification unit 14B according to the second modified example. The second identification unit 14B includes a color identification unit 41, a first candidate determination unit 42, a shape identification unit 43, a second candidate determination unit 44, a light emission identification unit 45, a third candidate determination unit 46, and a sign content determination unit 47.

[0124] Figure 18 is a flowchart showing an example of the procedure for the sign content identification process S15 according to the second modified example, which is implemented by the second identification unit 14B. The navigation aid identification device 1 executes the information processing shown in the figure according to the program. The sign content identification process S15 corresponds to S15 shown in Figure 7 above.

[0125] Figure 19 shows the color, top mark shape, and illumination pattern corresponding to the markings of buoys. Buoy markings are classified by the buoy's color, top mark shape, and illumination pattern. The illumination pattern refers to the time pattern of when the light is on and off.

[0126] As shown in Figure 18, first, the navigation aid identification device 1 identifies the color candidates of the buoys contained in the second image P2 (S41: processing as the color identification unit 41). Specifically, the navigation aid identification device 1 identifies the color candidates of the buoys in the second image P2 using a trained model. In addition, the navigation aid identification device 1 calculates a first probability, which represents the likelihood of the color candidates, along with the color candidates.

[0127] Next, the navigation aid identification device 1 determines the first candidate for the buoy's marker content corresponding to the identified color candidate (S42: Processing as the first candidate determination unit 42). Specifically, the navigation aid identification device 1 refers to a table representing the correspondence between colors and marker content and determines the marker content corresponding to the color candidate as the first candidate.

[0128] Next, the navigation aid identification device 1 identifies candidate shapes for the top marks of buoys contained in the second image P2 (S43: Processing as the shape identification unit 43). Specifically, the navigation aid identification device 1 uses a trained model to identify candidate shapes for the top marks of buoys in the second image P2. The navigation aid identification device 1 also calculates a second confidence score, which represents the likelihood of each shape candidate, along with the shape candidates.

[0129] Next, the navigation aid identification device 1 determines a second candidate for the buoy's marker content corresponding to the identified shape candidate (S44: Processing as the second candidate determination unit 44). Specifically, the navigation aid identification device 1 refers to a table representing the correspondence between shapes and marker content and determines the marker content corresponding to the shape candidate as the second candidate.

[0130] Next, the navigation aid identification device 1 identifies candidate buoy illumination patterns from multiple time-series second images P2 (S45: processing as illumination pattern identification unit 45). The navigation aid identification device 1 identifies candidate buoy illumination patterns according to predetermined rules.

[0131] Specifically, the navigation aid identification device 1 extracts the time patterns of the buoy's illumination and extinction from multiple time-series second images P2, and selects the standard time pattern that is most similar to the extracted time pattern from among multiple pre-stored standard time patterns as the illumination pattern candidate. The standard time patterns are created based on the illumination patterns of each marker (see Figure 19).

[0132] Furthermore, the navigation aid identification device 1 calculates a third-degree probability, which represents the likelihood of a candidate light pattern, along with the candidate light pattern itself. Specifically, the navigation aid identification device 1 calculates the similarity between the extracted time pattern and the standard time pattern designated as the candidate light pattern as the third-degree probability.

[0133] Next, the navigation aid identification device 1 determines a third candidate for the buoy's marker content corresponding to the identified light pattern candidate (S46: Processing as the third candidate determination unit 46). Specifically, the navigation aid identification device 1 determines the marker content corresponding to the standard time pattern designated as the light pattern candidate as the third candidate.

[0134] Next, the navigation aid identification device 1 determines whether the current time is daytime or nighttime (S47). If it is daytime, it applies the daytime criteria (S48), and if it is nighttime, it applies the nighttime criteria (S49). The current time is the time when the image was generated by camera 2. The criteria are the criteria used to determine the marker content of the buoy.

[0135] Next, the navigation aid identification device 1 determines the buoy's marker content based on the first candidate for marker content determined in S42, the second candidate for marker content determined in S44, and the third candidate for marker content determined in S46 (S50: Processing as the marker content determination unit 47).

[0136] Specifically, the navigation aid identification device 1 determines the buoy's marker content to be the same marker content if at least two of the first, second, and third candidates have the same marker content. For example, if two of the first, second, and third candidates are port markers and the remaining one is a starboard marker, then the port marker is determined to be the marker content.

[0137] Furthermore, the navigation aid identification device 1 may determine the buoy's marking content based on the first, second, and third degrees of certainty. For example, the candidate corresponding to the highest degree of certainty among the first, second, and third degrees of certainty is determined as the marking content. Also, if multiple candidates represent the same marking content, their corresponding degrees of certainty may be added together.

[0138] The navigation aid identification device 1 changes the weighting assigned to the first, second, and third accuracy levels for daytime and nighttime determination criteria. For example, during the day, it prioritizes candidates related to the color and shape of the buoy's top mark that are easily visible in bright environments, while at night, it prioritizes candidates related to the way the buoy glows that are easily visible even in dark environments.

[0139] In other words, in the daytime judgment criteria, the weighting of the first and second degrees of certainty related to the color of the buoy and the shape of the top mark is higher than the weighting of the third degree of certainty related to how the buoy glows. Conversely, in the nighttime judgment criteria, the weighting of the third degree of certainty related to how the buoy glows is higher than the weighting of the first and second degrees of certainty related to the color of the buoy and the shape of the top mark.

[0140] Furthermore, the method for determining the content of signs in this modified example may be applied not only to side signs and directional signs, but also to isolated obstacle signs, safe water area signs, and special signs.

[0141] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications are of course possible for those skilled in the art. [Explanation of Symbols]

[0142] 1 Navigation aid identification device, 2 Camera, 3 Radar, 4 AIS, 5 Radio communication unit, 6 Display unit, 7 GNSS receiver, 8 Gyrocompass, 9 ECDIS, 10 Automatic steering system, 11 First acquisition unit, 12 First identification unit, 13 Second acquisition unit, 14 Second identification unit, 15 Display control unit, 16 Route calculation unit, 17 Model memory unit, 21 Lens unit, 22 Pan / tilt mechanism, 31 Color identification unit, 32 Shape identification unit, 33 Country determination unit, 34 Port / Starboard determination unit, 41 Color identification unit, 42 First candidate determination unit, 43 Shape identification unit, 44 Second candidate determination unit, 45 Illumination pattern identification unit, 46 Third candidate determination unit, 47 Marker content determination unit, 100 Autonomous navigation system

Claims

1. An acquisition unit that acquires images generated by a camera installed on a ship, A color identification unit that identifies color candidates for buoys included in the aforementioned image, A shape identification unit for identifying candidate shapes for the top mark of the buoy, A light-reflecting unit that identifies candidate light-reflecting patterns for the buoy from multiple images in a time series, A first candidate determination unit that determines a first candidate for the buoy's marking content corresponding to the aforementioned color candidate, A second candidate determination unit for determining a second candidate for the marking content of a buoy corresponding to the shape candidate, A third candidate determination unit that determines a third candidate for the marking content of a buoy corresponding to the aforementioned light pattern candidate, A marking content determination unit that determines the marking content of the buoy based on the first candidate, the second candidate, and the third candidate, Equipped with, The mark content determination unit determines that at least two of the first candidate, second candidate, and third candidate have the same mark content as the mark content of the buoy. Navigation aid identification device.

2. An acquisition unit that acquires images generated by a camera installed on a ship, A color identification unit that identifies color candidates for buoys included in the aforementioned image, A shape identification unit for identifying candidate shapes for the top mark of the buoy, A light-reflecting unit that identifies candidate light-reflecting patterns for the buoy from multiple images in a time series, A first candidate determination unit that determines a first candidate for the buoy's marking content corresponding to the aforementioned color candidate, A second candidate determination unit for determining a second candidate for the marking content of a buoy corresponding to the shape candidate, A third candidate determination unit that determines a third candidate for the marking content of a buoy corresponding to the aforementioned light pattern candidate, A marking content determination unit that determines the marking content of the buoy based on the first candidate, the second candidate, and the third candidate, Equipped with, The color identification unit calculates a first probability, which represents the likelihood of the color candidate, along with the color candidate. The shape identification unit calculates a second probability, which represents the likelihood of the shape candidate, along with the shape candidate. The light pattern identification unit calculates a third probability, which represents the likelihood of the light pattern candidate, along with the light pattern candidate. The marking content determination unit determines the marking content of the buoy based on the sum of the first accuracy, the second accuracy, and the third accuracy. Navigation aid identification device.

3. The mark content determination unit changes the determination criteria for determining the mark content of the buoy according to the time the image was generated. A navigational aid identification device according to claim 1 or 2.

4. The sign content determination unit changes the weighting assigned to the first accuracy, second accuracy, and third accuracy, respectively, according to the time the image was generated. The navigation aid identification device according to claim 2.

5. The color identification unit identifies color candidates for the buoys in the image using a trained model. A navigation aid identification device according to any one of claims 1 to 4.

6. The shape recognition unit identifies candidate shapes of the top marks of the buoys in the image using a trained model. A navigation aid identification device according to any one of claims 1 to 5.

7. The light emission identification unit identifies candidate light emission patterns for the buoy according to predetermined rules. A navigation aid identification device according to any one of claims 1 to 6.

8. The system further includes a display control unit that displays a symbol representing the buoy's markings on the image, electronic chart, or radar image based on the markings of the buoy, the position of the buoy in the image, and the imaging direction of the camera. A navigation aid identification device according to any one of claims 1 to 7.

9. The system further includes a consistency determination unit that determines the consistency between the buoy's marker content and the marker content represented by navigational aid data recorded in an electronic chart, based on the marker content of the buoy, the position of the buoy in the image, the imaging direction of the camera, and the position of the vessel. A navigation aid identification device according to any one of claims 1 to 8.

10. The system further includes a display control unit that displays the consistency determination result on the image, the electronic chart, or the radar image. The navigation aid identification device according to claim 9.

11. A navigation aid identification device according to any one of claims 1 to 10, A route calculation unit calculates the ship's route or route width based on the position of the buoys in the image and the imaging direction of the camera, when the markings of multiple buoys include at least two of the following: port side markings, starboard side markings, and safe waters markings. An autonomous navigation system equipped with [the following features].

12. A navigation aid identification device according to any one of claims 1 to 10, A virtual sign acquisition unit acquires data representing the position and content of a virtual sign, A route calculation unit that calculates the ship's route or route width based on the marking content of the buoy, the position of the virtual marking, and the marking content of the virtual marking, An autonomous navigation system equipped with [the following features].

13. A navigation aid identification device according to any one of claims 1 to 10, A position detection unit for detecting the position of the aforementioned vessel, A route calculation unit that sets the turning points the vessel should pass through based on the markings of the buoy, the position of the buoy in the image, the imaging direction of the camera, and the position of the vessel, An autonomous navigation system equipped with [the following features].

14. A navigation aid identification device according to any one of claims 1 to 10, A direction detection unit for detecting the bow direction of the aforementioned vessel, A route calculation unit that sets the direction in which the vessel should navigate based on the markings of the buoy, the imaging direction of the camera, and the heading of the vessel, An autonomous navigation system equipped with [the following features].

15. The system further includes an automatic steering system that performs autonomous navigation control based on the markings of the buoys. An autonomous navigation system according to any one of claims 11 to 14.

16. Images generated by cameras installed on the ship are acquired. Identify the color candidates of the buoys included in the aforementioned image, Identify candidate shapes for the top mark of the aforementioned buoy, From multiple images in a time series, a candidate pattern of illumination for the buoy is identified. Determine the first candidate for the buoy's marking content corresponding to the aforementioned color candidate. A second candidate for the buoy's marking content corresponding to the aforementioned shape candidate is determined. A third candidate for the buoy's marking content corresponding to the aforementioned candidate for the way it glows is determined. Based on the first candidate, the second candidate, and the third candidate, the marking content of the buoy is determined. A method for identifying navigational aids, The determination of the marking content is made by determining that if at least two of the first candidate, second candidate, and third candidate have the same marking content, then that identical marking content is determined to be the marking content of the buoy. Navigation aid identification method.

17. Images generated by cameras installed on the ship are acquired. Identify the color candidates of the buoys included in the aforementioned image, Identify candidate shapes for the top mark of the aforementioned buoy, From multiple images in a time series, a candidate pattern of illumination for the buoy is identified. Determine the first candidate for the buoy's marking content corresponding to the aforementioned color candidate. A second candidate for the buoy's marking content corresponding to the aforementioned shape candidate is determined. A third candidate for the buoy's marking content corresponding to the aforementioned candidate for the way it glows is determined. Based on the first candidate, the second candidate, and the third candidate, the marking content of the buoy is determined. A method for identifying navigational aids, The identification of the color candidates involves calculating a first probability value representing the likelihood of the color candidates, along with the color candidates themselves. The identification of the shape candidates involves calculating a second degree of certainty, which represents the likelihood of the shape candidates, along with the shape candidates themselves. The identification of the aforementioned light-emitting pattern candidates is performed by calculating a third degree of certainty, which represents the likelihood of the aforementioned light-emitting pattern candidates, along with the aforementioned light-emitting pattern candidates. The determination of the marking content is based on the sum of the first, second, and third accuracy values, thereby determining the marking content of the buoy. Navigation aid identification method.

18. To acquire images generated by cameras installed on ships, To identify the color candidates of the buoys included in the aforementioned image, To identify candidate shapes for the top mark of the buoy, Identifying candidate patterns of illumination for the buoy from multiple images in a time series, To determine the first candidate for the buoy's marking content corresponding to the aforementioned color candidate, To determine a second candidate for the marking content of the buoy corresponding to the aforementioned shape candidate, To determine a third candidate for the buoy's marking content corresponding to the aforementioned candidate for the way it glows, and Based on the first candidate, the second candidate, and the third candidate, the content of the buoy's markings is determined. Have the computer run it, The determination of the marking content is made by determining that if at least two of the first candidate, second candidate, and third candidate have the same marking content, then that identical marking content is determined to be the marking content of the buoy. program.

19. To acquire images generated by cameras installed on ships, To identify the color candidates of the buoys included in the aforementioned image, To identify candidate shapes for the top mark of the buoy, Identifying candidate patterns of illumination for the buoy from multiple images in a time series, To determine the first candidate for the buoy's marking content corresponding to the aforementioned color candidate, To determine a second candidate for the marking content of the buoy corresponding to the aforementioned shape candidate, To determine a third candidate for the buoy's marking content corresponding to the aforementioned candidate for the way it glows, and Based on the first candidate, the second candidate, and the third candidate, the content of the buoy's markings is determined. Have the computer run it, The identification of the color candidates involves calculating a first probability value representing the likelihood of the color candidates, along with the color candidates themselves. The identification of the shape candidates involves calculating a second degree of certainty, which represents the likelihood of the shape candidates, along with the shape candidates themselves. The identification of the aforementioned light-emitting pattern candidates is performed by calculating a third degree of certainty, which represents the likelihood of the aforementioned light-emitting pattern candidates, along with the aforementioned light-emitting pattern candidates. The determination of the marking content is based on the sum of the first, second, and third accuracy values, thereby determining the marking content of the buoy. program.

Images