Navigational Aids Identification Device, Autonomous Navigation System, Navigational Aids Identification Method, and Program

Abstract

[Problem] To provide a channel marker identification device that can increase the identification accuracy for marker contents. [Solution] A channel marker identification device provided with: a first acquisition unit for acquiring a first image that includes a buoy from a camera installed in a ship; a first identification unit for identifying the position of the buoy in the first image; a second acquisition unit for acquiring a second image having a higher resolution than that of the first image and corresponding to a partial area of the first image which includes the position of the buoy; and a second identification unit for identifying marker contents of the buoy from the second image.

Description

[Technical Field]

[0001] The present invention relates to a navigational beacon identification device, an autonomous navigation system, a navigational beacon identification method, and a program. [Background technology]

[0002] Patent Document 1 discloses an automatic visual recognition device that automatically identifies navigational aids. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 4-76562 Summary of the Invention [Problem to be solved by the invention]

[0004] However, buoys floating on the sea surface are smaller than ships, and it is difficult to identify the markings on a buoy that is far away from the ship.

[0005] The present invention has been made in view of the above-mentioned problems, and a main object of the present invention is to provide a navigational beacon identification device, an autonomous navigation system, a navigational beacon identification method, and a program that can improve the accuracy of identifying beacon contents. [Means for solving the problem]

[0006] In order to solve the above problem, one embodiment of the navigational mark identification device of the present invention comprises a first acquisition unit that acquires a first image including a buoy from a camera installed on a ship, a first identification unit that identifies the position of the buoy in the first image, a second acquisition unit that acquires a second image having a higher resolution than the first image and corresponding to a partial region of the first image that includes the position of the buoy, and a second identification unit that identifies the marking content of the buoy from the second image.

[0007] In the above aspect, the camera may have an optical zoom function, and the second acquisition unit may acquire the second image by causing the camera to enlarge and capture an area corresponding to the partial region of the first image.

[0008] In the above aspect, the camera may have a pan-tilt function, and the second acquisition unit may direct the imaging direction of the camera to a range corresponding to the partial area of ​​the first image based on the position of the buoy.

[0009] In the above aspect, the second acquisition unit may acquire the second image by increasing the resolution of the partial region of the first image.

[0010] In the above aspect, the second acquisition unit may acquire the second image by causing an auxiliary camera having a higher resolution than the camera to capture an image of a range corresponding to the partial region of the first image.

[0011] In the above aspect, the first identification unit may use a first trained model to identify the position of the buoy in the first image, and the second identification unit may use a second trained model to identify the marking content of the buoy from the second image.

[0012] In the above aspect, the first trained model and the second trained model may have first trained parameters and second trained parameters that are different from each other, respectively, incorporated into a common inference program.

[0013] In the above aspect, the first identification unit may further identify the type of the buoy.

[0014] In the above aspect, when the type of the buoy is a side beacon, the second identification unit may identify whether the beacon content of the buoy is a port beacon or a starboard beacon.

[0015] In the above aspect, when the type of the buoy is a directional indicator, the second identification unit may identify whether the indicator content of the buoy is a north directional indicator, an east directional indicator, a south directional indicator, or a west directional indicator.

[0016] In the above aspect, when the type of the buoy is a side beacon, the second identification unit may use a trained model for side beacons to identify whether the beacon content of the buoy is a port beacon or a starboard beacon, and when the type of the buoy is a direction beacon, the second identification unit may use a trained model for direction beacons to identify whether the beacon content of the buoy is a north direction beacon, an east direction beacon, a south direction beacon, or a west direction beacon.

[0017] In the above aspect, the second acquisition unit may not acquire the second image when the type of the buoy is an isolated hazard beacon, a safe water beacon, or a special beacon.

[0018] In the above aspect, when multiple buoys are identified from the first image, the second acquisition unit and the second identification unit may acquire the second image and identify the sign content for each of the multiple buoys.

[0019] In the above aspect, the navigation system may further include a display control unit that displays a symbol representing the marking content of the buoy on the first image, electronic nautical chart, or radar image based on the marking content of the buoy, the position of the buoy in the first image, and the imaging direction of the camera.

[0020] In the above aspect, the system may further include a consistency determination unit that determines the consistency between the marking content of the buoy and the marking content represented by the navigational aid data recorded on the electronic nautical chart, based on the marking content of the buoy, the position of the buoy in the first image, the imaging direction of the camera, and the position of the ship.

[0021] In the above aspect, the navigation system may further include a display control unit that displays the consistency determination result on the first image, the electronic nautical chart, or the radar image.

[0022] In addition, an autonomous navigation system according to another aspect of the present invention may include the above-mentioned navigational beacon identification device, and a route calculation unit that calculates the route or route width of the ship based on the position of the buoys in the first image and the imaging direction of the camera when the beacon contents of the multiple buoys include at least two of a port beacon, a starboard beacon, and a safe waters beacon.

[0023] In addition, an autonomous navigation system according to another aspect of the present invention may include the above-mentioned navigational beacon identification device, a virtual sign acquisition unit that acquires data representing the position and content of a virtual sign, and a route calculation unit that calculates the route or route width of the ship based on the beacon content of the buoy, the position of the virtual sign, and the beacon content of the virtual sign.

[0024] In addition, an autonomous navigation system according to another aspect of the present invention may include the above-mentioned navigational beacon identification device, a position detection unit that detects the position of the ship, and a route calculation unit that sets a waypoint through which the ship should pass based on the beacon content of the buoy, the position of the buoy in the first image, the imaging direction of the camera, and the position of the ship.

[0025] In addition, an autonomous navigation system according to another aspect of the present invention may include the above-described navigational beacon identification device, a direction detection unit that detects the bow direction of the ship, and a route calculation unit that sets the direction in which the ship should navigate based on the beacon content of the buoy, the imaging direction of the camera, and the bow direction of the ship.

[0026] In the above aspect, the vessel may further include an automatic steering device that performs autonomous navigation control based on the marker content of the buoy.

[0027] Another aspect of the present invention provides a method for identifying navigational marks, which comprises acquiring a first image including a buoy from a camera installed on a ship, identifying the position of the buoy within the first image, acquiring a second image having a higher resolution than the first image and corresponding to a partial region of the first image that includes the position of the buoy, and identifying the markings of the buoy from the second image.

[0028] In addition, another aspect of the program of the present invention causes a computer to perform the following steps: acquire a first image including a buoy from a camera installed on a ship; identify the position of the buoy in the first image; acquire a second image having a higher resolution than the first image corresponding to a partial area of ​​the first image including the position of the buoy; and identify the marking content of the buoy from the second image. [Effects of the Invention]

[0029] According to the present invention, it is possible to improve the accuracy of identifying sign content. [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 is a block diagram illustrating a configuration example of an autonomous navigation system. [Figure 2] FIG. 1 is a diagram for explaining the markings on the buoys. [Figure 3] FIG. 2 is a block diagram showing an example of a functional configuration of a navigational beacon identification device. [Figure 4] FIG. 2 is a diagram showing an example of a first image. [Figure 5] FIG. 10 is a diagram showing an example of recognition by a first recognition unit. [Figure 6] FIG. 10 is a diagram showing an example of a second image. [Figure 7] FIG. 1 is a flowchart showing an example of the procedure of a method for identifying navigational marks. [Figure 8] FIG. 10 is a flowchart showing an example of a procedure for sign content identification processing. [Figure 9] FIG. 10 is a diagram illustrating an example of a buoy management database. [Figure 10] FIG. 10 is a diagram showing an example of display by a display unit. [Figure 11] FIG. 10 is a diagram showing another example of display by the display unit. [Figure 12] FIG. 10 is a block diagram showing another example of the configuration of a navigational beacon identification device. [Figure 13] FIG. 10 is a block diagram showing yet another example of the configuration of the navigational beacon identification device. [Figure 14]FIG. 10 is a block diagram showing an example of the configuration of a second identification unit according to a first modified example. [Figure 15] FIG. 10 is a flowchart showing an example of a procedure for port and starboard sign identification processing. [Figure 16] FIG. 10 is a diagram showing an example of a country-specific starboard side configuration table. [Figure 17] FIG. 10 is a block diagram showing an example of the configuration of a second identification unit according to a second modified example. [Figure 18] FIG. 10 is a flowchart showing an example of a procedure for sign content identification processing. [Figure 19] This is a diagram to explain how a buoy shines. DETAILED DESCRIPTION OF THE INVENTION

[0031] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0032] 1 is a block diagram showing an example of the configuration of an autonomous navigation system 100. The autonomous navigation system 100 is an ICT system installed on a ship. Hereinafter, the ship on which the autonomous navigation system 100 is installed will be referred to as the "own ship."

[0033] The autonomous navigation system 100 includes a navigational beacon identification device 1, a camera 2, a radar 3, an AIS 4, a wireless communication unit 5, a display unit 6, a GNSS receiver 7, a gyrocompass 8, an ECDIS 9, and an automatic steering device 10. These devices are connected to a network N such as a LAN, and are capable of network communication with each other.

[0034] The navigational beacon identification device 1 is a computer including a CPU, RAM, ROM, non-volatile memory, an input / output interface, etc. The CPU of the navigational beacon identification device 1 executes information processing in accordance with a program loaded from the ROM or non-volatile memory to the RAM.

[0035] The program may be supplied via an information storage medium such as an optical disk or a memory card, or may be supplied via a communication network such as the Internet or a LAN.

[0036] Camera 2 is a digital camera that captures images of the outside of the ship and generates image data. Camera 2 is installed, for example, on the bridge of the ship, facing the bow direction. Camera 2 is a visible light camera that can capture images in at least the visible range. It may also be capable of capturing images in the infrared range as well as the visible range.

[0037] In this embodiment, the camera 2 is a so-called PTZ camera, which has pan / tilt and optical zoom functions. The camera 2 pans, tilts, or zooms in response to commands from the navigational beacon identification device 1.

[0038] The radar 3 emits radio waves around the ship and receives the reflected waves, generating echo data based on the received signals. The radar 3 also identifies targets from the echo data and generates target tracking data (TT data) that indicate the position and speed of the targets.

[0039] The AIS (Automatic Identification System) 4 receives AIS data from other ships around the ship or from land-based control. Instead of AIS, a VDES (VHF Data Exchange System) may also be used. The AIS data includes the position and speed of other ships.

[0040] The AIS 4 may also acquire AIS data indicating the position and content of virtual beacons. The AIS 4 is an example of a virtual beacon acquisition unit. A virtual beacon using an AIS is a so-called virtual AIS navigational aid.

[0041] The wireless communication unit 5 includes various types of wireless equipment, such as wireless equipment for the ultra-short wave band, the medium wave band, and the short wave band, for realizing communication with other ships or land-based control.

[0042] The display unit 6 is, for example, a display device with a touch sensor, a so-called touch panel. The display device may be, for example, a liquid crystal display device or an organic EL display device. The display device is not limited to a touch sensor, and other pointing devices such as a trackball or a mouse may also be used.

[0043] The display unit 6 displays an image captured by the camera 2, a radar image generated by the radar 3, an electronic nautical chart, or a composite image obtained by combining a radar image and an electronic nautical chart.

[0044] The GNSS receiver 7 detects the ship's own position based on radio waves received from a GNSS (Global Navigation Satellite System). The GNSS receiver 7 is an example of a position detection unit that detects the ship's own position.

[0045] The gyrocompass 8 detects the heading of the ship. The gyrocompass 8 is an example of a direction detection unit that detects the heading of the ship. The gyrocompass is not limited to a gyrocompass, and other types of direction indicators such as a GPS compass may also be used.

[0046] An ECDIS (Electronic Chart Display and Information System) 9 acquires the ship's position from the GNSS receiver 7 and displays the ship's position on an electronic chart. The ECDIS 9 also displays the ship's planned route on the electronic chart. A GNSS plotter may be used instead of an ECDIS.

[0047] The automatic steering device 10 calculates a target rudder angle for directing the bow toward the target course based on the target course acquired from the navigational mark identification device 1 etc. and the bow direction acquired from the gyrocompass 8, and drives the steering gear so that the rudder angle of the steering gear approaches the target rudder angle. The automatic steering device 10 may also control the engine.

[0048] In this embodiment, the navigational beacon identification device 1 is an independent device, but is not limited to this and may be integrated with another device such as the ECDIS 9. In other words, the function of the navigational beacon identification device 1 may be realized by another device such as the ECDIS 10.

[0049] In addition, in this embodiment, the display unit 2 is also an independent device, but this is not limited to this, and a display unit provided in another device such as ECDIS 9 may be used as the display unit 2 that displays the image generated by the navigational mark identification device 1.

[0050] Figure 2 is a diagram for explaining the markings of buoys. A buoy is a navigational marker that floats on the sea surface and is also called a buoy. The type and markings of buoys are classified according to the color of the buoy and the shape of its top mark.

[0051] Buoys are classified as side marks, direction marks, isolated hazard marks, safe water marks, and special marks. Side marks include port marks and starboard marks. Port and starboard indicate left and right when heading towards the water source. Direction marks include north mark, east mark, south mark, and west mark.

[0052] The contents of the buoys are not further subdivided for isolated hazard markers, safe water markers, and special markers. In other words, the type of buoy itself represents the contents of the buoy.

[0053] However, buoys floating on the sea surface are smaller than ships, and it is difficult to identify the markings on buoys that are far away from the ship. Therefore, in this embodiment, the accuracy of identifying the markings is improved by acquiring images in stages as described below.

[0054] 3 is a block diagram showing an example of the functional configuration of the navigational mark identification device 1 according to the embodiment. The navigational mark identification device 1 includes a first acquisition unit 11, a first identification unit 12, a second acquisition unit 13, a second identification unit 14, a display control unit 15, a route calculation unit 16, and a consistency determination unit 18.

[0055] These functional units are realized by executing information processing according to a program by the CPU of the navigational mark identification device 1. Note that some functional units such as the display control unit 15 or the route calculation unit 16 may be realized by a computer separate from the navigational mark identification device 1, which is included in the ECDIS 9 or the automatic steering device 10, etc.

[0056] The navigational mark identification device 1 also includes a model storage unit 17 that stores the trained model. This storage unit is provided in a non-volatile memory of the navigational mark identification device 1. However, the model storage unit 17 may be provided outside the navigational mark identification device 1.

[0057] The first acquisition unit 11 acquires a first image generated by the camera 2. Specifically, the first acquisition unit 11 sequentially acquires a plurality of first images in time series generated by the camera 2 and provides them to the first identification unit 12 sequentially.

[0058] The first image is an image captured when the camera 2 is in a standard state. The standard state is, for example, a state in which the optical zoom magnification is minimum and the imaging direction is facing the bow direction. The camera 2 repeatedly generates the first image in the standard state except for the period controlled by the second acquisition unit 13.

[0059] The plurality of first images in time series may be, for example, a plurality of still images (frames) included in a moving image, or may be a plurality of still images individually generated by capturing images at predetermined time intervals.

[0060] 4 is a diagram showing an example of a first image P1 acquired by the first acquisition unit 11. The figure shows an example in which the first image P1 includes the hull SP of the ship as well as the port marker LL and starboard marker LR floating on the sea surface ahead of the ship.

[0061] The first identification unit 12 identifies the position of the buoy in the first image P1. Specifically, the first identification unit 12 identifies the position of the buoy in the first image P1 using the first trained model stored in the model storage unit 17. Furthermore, the first identification unit 12 may further identify the type of the buoy along with the position of the buoy in the first image P1.

[0062] The first trained model is generated by machine learning using training images as input data and the labels (or labels of buoy types) and positions of buoys in the training images as training data. The first trained model generated in this manner estimates the labels (or labels of buoy types), positions, and accuracy of the buoys in the first image P1. The positions of the buoys are represented, for example, by the coordinates of a bounding box surrounding the buoy.

[0063] The first trained model may be, for example, an object detection model such as SSD (Single Shot MultiBox Detector), YOLO (You Only Look Once), or Mask R-CNN. However, the first trained model may be, for example, a region segmentation model such as Semantic Segmentation or Instance Segmentation, or a feature point detection model such as Keypoint Detection.

[0064] 5 is a diagram showing an example of classification of the first image P1 by the first classification unit 12. The figure shows an example in which the port marking LL and the starboard marking LR are each identified as a buoy (or a side buoy) and are surrounded by a bounding box BB.

[0065] The second acquisition unit 13 acquires a second image having a higher resolution than the first image P1, the second image corresponding to a partial region of the first image P1 that includes the position of the buoy. The partial region is, for example, a bounding box BB (see FIG. 5) identified by the first identification unit 12.

[0066] The camera 2 is equipped with a lens unit 21 that realizes an optical zoom function and a pan-tilt mechanism 22 that realizes a pan-tilt function, and the second acquisition unit 13 acquires the second image by controlling the lens unit 21 and the pan-tilt mechanism 22 of the camera 2.

[0067] Specifically, the second acquisition unit 13 acquires the second image by controlling the lens unit 21 to cause the camera 2 to capture an enlarged image of the real space corresponding to the partial region of the first image P1. By using the optical zoom function in this way, the second image having a higher resolution than the first image P1 is acquired.

[0068] The second acquisition unit 13 also controls the pan-tilt mechanism 22 to direct the imaging direction of the camera 2 to the range of real space corresponding to the partial region of the first image P1. The second acquisition unit 13 sets a target value for the imaging direction of the camera 2 according to the position of the buoy in the first image P1 identified by the first identification unit 12.

[0069] 6 is a diagram showing an example of the second image P2 acquired by the second acquisition unit 13. The figure shows an example in which the second image P2 includes the port mark LL. In the second image P2, the color of the port mark LL, the shape of the top mark TM, etc. are easier to distinguish than in the first image P1 (see FIG. 4).

[0070] As shown in Figure 5 above, when multiple buoys (in the illustrated example, the port marker LL and the starboard marker LR) are identified in the first image P1, the second acquisition unit 13 acquires a second image P2 for each of the multiple buoys by having the camera 2 sequentially capture an image of each of the multiple buoys.

[0071] The second identification unit 14 identifies the marking content of the buoy from the second image P2. Specifically, the second identification unit 14 identifies the marking content of the buoy from the second image P2 using the second trained model stored in the model storage unit 17.

[0072] The second trained model is generated by machine learning using training images as input data and labels of the sign content of the buoy in the training images as training data. The second trained model generated in this way estimates the labels and accuracy of the sign content of the buoy in the second image P2.

[0073] The second trained model may be, for example, the same type of model as the first trained model. In this case, the first trained model and the second trained model are each constructed by incorporating different first trained parameters and second trained parameters into a common inference program.

[0074] Without being limited to this, the second trained model may be an object identification model that only identifies an object and does not detect the position of the object.

[0075] The second trained model may also include a trained model for side signs that is specialized in identifying the sign content of side signs, and a trained model for directional signs that is specialized in identifying the sign content of directional signs.

[0076] 7 and 8 are flow charts showing an example of the procedure of the method for identifying navigational marks implemented by the navigational mark identification device 1. The figures mainly show the processes related to image acquisition and identification of the contents of the marks among the processes executed by the navigational mark identification device 1.

[0077] The CPU of the navigational mark identification device 1 executes the information processing shown in these figures in accordance with the programs, thereby functioning as a first acquisition unit 11, a first identification unit 12, a second acquisition unit 13, and a second identification unit .

[0078] As shown in FIG. 7, first, the navigational mark identification device 1 acquires a first image P1 (see FIG. 4) from the camera 2 (S11: processing as the first acquisition unit 11).

[0079] Next, the navigational mark identification device 1 uses the first learned model to identify the position and type of the buoy in the first image P1 (S12: processing as the first identification unit 12).

[0080] Next, the navigational mark identification device 1 determines whether the type of the buoy identified in the first image P1 is a side buoy or an azimuth buoy (S13).

[0081] If the type of buoy is a side buoy or an azimuth buoy (S13 → YES), the navigational mark identification device 1 controls the camera 2 to acquire a second image P2 (see Figure 6) of the buoy, which is an enlarged image of the buoy (S14: processing as the second acquisition unit 13).

[0082] Next, the navigational mark identification device 1 executes a mark content identification process for identifying the mark content of the buoy from the second image P2 (S15: processing as the second identification unit 14).

[0083] As shown in Figure 8, in the sign content identification process S15, if the type of buoy is a side buoy (S21 → side buoy), the navigational mark identification device 1 uses a trained model for side marks as the second trained model to identify whether the sign content is a port beacon or a starboard beacon (S22).

[0084] On the other hand, if the type of buoy is a direction buoy (S21 → direction buoy), the navigational mark identification device 1 uses a trained model for direction buoys as the second trained model to identify whether the sign content is a north direction buoy, an east direction buoy, a south direction buoy, or a west direction buoy (S23).

[0085] If the type of buoy is not a side buoy or a direction buoy (S13→NO), that is, if the type of buoy is an isolated hazard mark, a safe waters mark, or a special mark, the navigational mark identification device 1 does not acquire the second image P2, because for these buoys, the type itself represents the mark content.

[0086] If multiple buoys are identified in the first image P1 in S12, the navigational mark identification device 1 executes S13 to S15 for all identified buoys (S16). That is, for all buoys that are side buoys or azimuth buoys, the second image P2 is acquired and the mark contents are identified.

[0087] According to the embodiment described above, the sign content is identified from the second image P2, which has a higher resolution than the first image P1 and is enlarged and captured based on the position of the buoy identified in the first image P1, thereby making it possible to improve the accuracy of identifying the sign content.

[0088] Furthermore, according to the embodiment, the type of buoy is identified in the first image P1 and then the sign content is identified from the second image P2, so that the sign content can be narrowed down according to the type of buoy, thereby enabling further improvement in the accuracy of identifying the sign content.

[0089] Alternatively, the buoy and its position may be identified from the first image P1, and the type of buoy and the marking details may be identified from the second image P2.

[0090] 9 is a diagram showing an example of a buoy management DB (database). The buoy management DB is a database for managing information on identified or acquired buoys, and is provided in the non-volatile memory of the navigational beacon identification device 1. The buoy management DB includes not only information on buoys identified from images taken by the camera 2, but also information on virtual beacons acquired by the AIS 4.

[0091] The buoy management DB includes fields such as "identifier," "type," "sign content," "position in image," "actual position," and "virtual buoy." "Identifier" is an identifier for identifying a buoy. "Virtual buoy" indicates whether it is a virtual buoy or not.

[0092] "Type" indicates the type of buoy. "Sign content" indicates the sign content of the buoy. If "Type" is a side sign or a direction sign, "Sign content" is entered as a port sign or a north direction sign, etc. On the other hand, if "Type" is an isolated obstacle sign, a safe waters sign, or a special sign, no data is entered in "Sign content".

[0093] "Position in image" indicates the position of the buoy in the first image P1 (see Figure 4). In the case of a virtual buoy, no data is entered into "Position in image". "Actual position" indicates the actual position of the buoy. The actual position of the buoy identified from the image of camera 2 is calculated based on the position of the buoy in the image and the imaging direction of camera 2.

[0094] Returning to the explanation of Fig. 3, the display control unit 15 generates display data related to the buoy and outputs it to the display unit 6. Specifically, the display control unit 15 displays a symbol representing the marking content of the buoy on the first image P1, electronic chart, radar image, or the like, based on the marking content of the identified buoy, the position of the buoy in the first image P1, the imaging direction of the camera 2, and the like.

[0095] 10, the display control unit 15 displays an image on the display unit 6 in which symbols ML and MR representing the contents of the port sign LL and starboard sign LR are attached in association with the positions of these signs in the first image P1. The symbols ML and MR include, for example, character strings representing the contents of the signs.

[0096] 11, the display control unit 15 displays an image on the display unit 6 in which symbols TL and TR representing the contents of the port marking LL and starboard marking LR are added at positions corresponding to the actual positions of these marks in a composite image CP obtained by combining an electronic nautical chart and a radar image. The symbols TL and TR have shapes that represent the contents of the marks, for example.

[0097] The composite image CP displays the symbol SF of the own ship, the planned route RT of the own ship, a waypoint DF on the planned route RT, and the symbol EL of another ship.

[0098] Furthermore, symbols VL and VR representing the sign contents of the virtual sign may be displayed in the composite image CP. The symbols VL and VR have the same shape as the symbols TL and TR. It is preferable that the symbols VL and VR are displayed so as to be distinguishable from the symbols TL and TR, for example, by changing the transparency.

[0099] The route calculation unit 16 calculates a target course for autonomous navigation control, i.e., a bearing, waypoints, and route, based on the identified buoy beacon information. The calculated target course is provided to the automatic steering device 10, which performs autonomous navigation control. Here, erroneous recognition of the identified navigation beacon information during autonomous navigation control can cause a serious accident. Therefore, by performing autonomous navigation control using the improved accuracy of the present invention to identify the navigation beacon information, an autonomous navigation system that can withstand ship maneuvering in real environments can be realized.

[0100] 5, when the buoys identified in the first image P1 include the port mark LL and the starboard mark LR, the route calculation unit 16 calculates the planned route or route width of the ship based on the positions of the port mark LL and the starboard mark LR in the first image P1 and the imaging direction of the camera 2. Specifically, the route calculation unit 16 sets the planned route RT of the ship so that it passes between the port mark LL and the starboard mark LR from the ship's position based on the actual positions of the port mark LL and the starboard mark LR calculated from the positions of the port mark LL and the starboard mark LR in the first image P1 and the imaging direction of the camera 2 (see FIG. 11). Not limited to this, if the buoy identified in the first image P1 includes a port marker LL or a starboard marker LR and a safe water area marker, the route calculation unit 16 may set the ship's planned route RT between the port marker LL or the starboard marker LR and the safe water area marker.

[0101] Furthermore, the route calculation unit 16 may calculate the distance between the port mark LL and the starboard mark LR as the route width W based on the positions of the port mark LL and the starboard mark LR in the first image P1 and the actual positions of the port mark LL and the starboard mark LR calculated from the imaging direction of the camera 2. The calculated route width W may be displayed in the first image P1 displayed on the display unit 6, or may be displayed in a composite image CP that combines an electronic chart and a radar image (see FIG. 11).

[0102] The route calculation unit 16 may set a waypoint through which the ship should take, based on the beacon content of the identified buoy, the position of the buoy in the first image P1, the imaging direction of the camera 2, and the ship's own position. Specifically, the route calculation unit 16 sets one or more waypoints DF for setting a planned route RT for the ship entering or departing a port, based on the beacon content of the identified side beacon and direction buoy, the actual positions of the buoys calculated from the positions of the buoys in the first image P1 and the imaging direction of the camera 2, and the ship's own position (see FIG. 11 ). Alternatively, the route calculation unit 16 may set one or more waypoints for setting an avoidance route to avoid an obstacle or special area, based on the beacon content of the identified isolated obstacle beacon or special beacon, the actual positions of the buoys calculated from the positions of the buoys in the first image P1 and the imaging direction of the camera 2, and the ship's own position.

[0103] The route calculation unit 16 may set the heading in which the ship should navigate based on the marking content of the identified buoy, the imaging direction of the camera 2, and the ship's heading. For example, the route calculation unit 16 maintains or adjusts the heading in which the ship should navigate so that a buoy such as a side buoy continues to be included in the multiple first images P1 in a time series. The route calculation unit 16 may also use the position of the buoy in the first image P1 to set the heading in which the ship should navigate so that the ship heads between the port and starboard marks or heads in a direction along multiple port or starboard marks.

[0104] The route calculation unit 16 may calculate a target course for autonomous navigation control, that is, a direction, waypoints, and route, based on the positions and content of virtual beacons in addition to the beacon content of the identified buoy. Specifically, when data on the virtual port beacon VL and the virtual starboard beacon VR is acquired, the route calculation unit 16 may set the planned route RT of the ship so that it passes not only between the port beacon LL and starboard beacon LR identified in the first image P1, but also between the virtual port beacon VL and the virtual starboard beacon VR.

[0105] The consistency determination unit 18 determines the consistency between the beacon content of the identified buoy and the beacon content represented by the navigational aid data recorded on the electronic chart, based on the beacon content of the identified buoy, the position of the buoy in the first image P1, the imaging direction of the camera 2, and the ship's position. Specifically, the consistency determination unit 18 calculates the actual position of the buoy from the position of the buoy in the first image P1, the imaging direction of the camera 2, and the ship's position, extracts navigational aid data corresponding to the actual position of the buoy from the navigational aid data recorded on the electronic chart, and determines whether the beacon content of the identified buoy matches the beacon content represented by the extracted navigational aid data.

[0106] The display control unit 15 displays the determination result by the matching determination unit 18 on the first image P1, an electronic nautical chart, a radar image, or the like. For example, the display control unit 15 displays a symbol indicating whether or not there is a match in association with the buoy on the first image P1 (see FIG. 10) or the composite image CP (see FIG. 11) displayed on the display unit 6. Alternatively, the display control unit 15 may display a symbol indicating the marking content (symbols ML and MR in FIG. 10 or symbols TL and TR in FIG. 11, or the like) only on matching buoys.

[0107] The configuration of the navigational mark identification device 1 is not limited to the example shown in Fig. 3. For example, as shown in Fig. 12, the second acquisition unit 13A may be an image processing unit that acquires a second image by increasing the resolution of a partial region of the first image. By increasing the resolution in this manner, a second image having a higher resolution than the first image is acquired.

[0108] Alternatively, the first acquisition unit 11 may acquire the first image by thinning or averaging the original image generated by the camera 2, and the second acquisition unit 13 may acquire the second image by cutting out an area corresponding to a partial area of ​​the first image from the original image. This also allows the acquisition of a second image with higher resolution than the first image.

[0109] 13, the second acquisition unit 13B may be a camera control unit that acquires the second image by causing an auxiliary camera 3 having a higher resolution than the camera 2 to capture an image of a range of real space corresponding to a partial region of the first image. By using the auxiliary camera 3 in this way, a second image having a higher resolution than the first image is acquired.

[0110] 3, the auxiliary camera 3 includes a lens unit 31 that realizes an optical zoom function and a pan-tilt mechanism 32 that realizes a pan-tilt function. The lens unit 31 of the auxiliary camera 3 has a higher magnification than the lens unit 21 of the camera 2.

[0111] [First Modification] The following describes the first modified example. The same reference numerals are used to denote configurations and processes that overlap with those of the above embodiment, and detailed descriptions thereof may be omitted.

[0112] The interpretation of port and starboard side buoys may differ depending on the country. Therefore, in this modified example, the port and starboard side buoys are distinguished from each other regardless of the ship's position, as will be explained below.

[0113] 14 is a block diagram showing a configuration example of the second identification unit 14A according to the first modified example. The figure mainly shows the functional unit for identifying the marking contents of the side buoys among the functional units realized in the second identification unit 14A.

[0114] The second identification section 14A includes a color identification section 31, a shape identification section 32, a country determination section 33, and a port / starboard determination section 34. The color identification section 31 and the shape identification section 32 are examples of mode identification sections.

[0115] When the type of buoy identified by the first identification unit 12 (type identification unit) shown in Figure 3 above is a side buoy, the functional unit of the second identification unit 14A identifies the sign content of the side buoy included in the second image P2 (see Figure 6).

[0116] 15 is a flow chart showing an example of the procedure of the port and starboard mark identification processing S22 according to the first modified example, which is realized by the second identification unit 14A. The navigational mark identification device 1 executes the information processing shown in the drawing in accordance with a program.

[0117] The port and starboard mark identification processing S22 corresponds to S22 shown in Fig. 8. That is, the navigational mark identification device 1 executes the port and starboard mark identification processing S22 when the type of buoy identified in S12 shown in Fig. 7 is a side buoy.

[0118] First, the navigational mark identification device 1 identifies whether the color of the side buoy included in the second image P2 is green or red (S31: processing as the color identification unit 31). Green and red are examples of the first and second modes.

[0119] Next, the navigational mark identification device 1 identifies whether the top mark of the side buoy included in the second image P2 is cylindrical or conical (S32: processing as the shape identification unit 32). The cylindrical shape and the conical shape are examples of the first and second modes.

[0120] The color and the shape of the top mark are identified using trained models, as in the above embodiment. For example, a trained model that identifies both the color and the shape of the top mark may be used, or a trained model that identifies the color and a trained model that identifies the shape of the top mark may be used separately.

[0121] Next, the navigational mark identification device 1 determines the country to which the detected position of the ship, detected by the GNSS receiver 7 (see FIG. 1), belongs (S33: processing as the country determination unit 33). For example, the navigational mark identification device 1 determines, based on nautical chart data, which country's territorial waters include the coordinates of the detected position of the ship.

[0122] Next, the navigational mark identification device 1 refers to the country-specific port and starboard side configuration table and determines whether the mark content of the side buoy is a port side mark or a starboard side mark based on the color identified in S31, the shape of the top mark identified in S32, and the country determined in S33 (S34: processing as port and starboard side determination unit 34).

[0123] 16 is a diagram showing an example of a country-by-country starboard side configuration table. The country-by-country starboard side configuration table is a table showing the correspondence between the buoy configuration and the beacon content, and is stored in the non-volatile memory of the navigational mark identification device 1.

[0124] Specifically, the country-specific port and starboard configuration table indicates whether the green and red colors of side buoys correspond to port and starboard beacons in each country. The country-specific port and starboard configuration table also indicates whether the cylindrical and conical shapes of top marks correspond to port and starboard beacons in each country.

[0125] According to the first modified example described above, it is possible to distinguish between the port and starboard beacons on the side buoys regardless of the ship's position.

[0126] [Second Modification] The second modified example will be described below. Configurations and processes that overlap with those of the above embodiment will be denoted by the same reference numerals, and detailed descriptions thereof may be omitted.

[0127] The type and marking details of a buoy can be identified by factors such as the color of the buoy, the shape of the top mark, and the way it glows, but when identifying the marking details directly from an image, the contribution of each factor cannot be determined and the identification accuracy may be insufficient. Therefore, in this modified example, the identification accuracy of the marking details is improved as described below.

[0128] 17 is a block diagram showing an example of the configuration of the second identification unit 14B according to the second modification. The second identification unit 14B includes a color identification unit 41, a first candidate determination unit 42, a shape identification unit 43, a second candidate determination unit 44, a light pattern identification unit 45, a third candidate determination unit 46, and a sign content determination unit 47.

[0129] 18 is a flow diagram showing an example of the procedure of the sign content identification process S15 according to the second modification, which is realized by the second identification unit 14B. The navigational mark identification device 1 executes the information processing shown in the figure in accordance with a program. The sign content identification process S15 corresponds to S15 shown in FIG. 7 above.

[0130] Figure 19 shows the color, shape of the top mark, and lighting pattern corresponding to the buoy's markings. The buoy's markings are classified by the buoy's color, shape of the top mark, and lighting pattern. The lighting pattern is the time pattern of lighting on and off.

[0131] As shown in Fig. 18, first, the navigational mark identification device 1 identifies color candidates of the buoy included in the second image P2 (S41: processing as the color identification unit 41). Specifically, the navigational mark identification device 1 identifies color candidates of the buoy in the second image P2 using a trained model. In addition, the navigational mark identification device 1 calculates the color candidates as well as a first accuracy that indicates the likelihood of the color candidates.

[0132] Next, the navigational mark identification device 1 determines the first candidate for the mark content of the buoy corresponding to the identified color candidate (S42: processing as the first candidate determination unit 42). Specifically, the navigational mark identification device 1 refers to a table showing the correspondence between colors and mark content, and determines the mark content corresponding to the color candidate as the first candidate.

[0133] Next, the navigational mark identification device 1 identifies shape candidates of the top mark of the buoy included in the second image P2 (S43: processing as the shape identification unit 43). Specifically, the navigational mark identification device 1 identifies shape candidates of the top mark of the buoy in the second image P2 using the trained model. In addition, the navigational mark identification device 1 calculates a second accuracy indicating the likelihood of the shape candidate along with the shape candidate.

[0134] Next, the navigational mark identification device 1 determines a second candidate for the mark content of the buoy corresponding to the identified shape candidate (S44: processing as the second candidate determination unit 44). Specifically, the navigational mark identification device 1 refers to a table showing the correspondence relationship between shapes and mark content, and determines the mark content corresponding to the shape candidate as the second candidate.

[0135] Next, the navigational mark identifying device 1 identifies the lighting pattern candidates of the buoy from the plurality of second images P2 in time series (S45: processing as the lighting pattern identifying unit 45). The navigational mark identifying device 1 identifies the lighting pattern candidates of the buoy according to a predetermined rule.

[0136] Specifically, the navigational mark identification device 1 extracts the time pattern of the buoys turning on and off from a plurality of second images P2 in time series, and selects the standard time pattern that is most similar to the extracted time pattern from a plurality of pre-stored standard time patterns as the lighting pattern candidate. The standard time pattern is created based on the lighting pattern of each beacon content (see Figure 19).

[0137] The navigational beacon identification device 1 also calculates the lighting pattern candidate and a third accuracy indicating the likelihood of the lighting pattern candidate. Specifically, the navigational beacon identification device 1 calculates the similarity between the extracted time pattern and the standard time pattern selected as the lighting pattern candidate as the third accuracy.

[0138] Next, the navigational mark identification device 1 determines a third candidate for the sign content of the buoy corresponding to the identified lighting pattern candidate (S46: processing as the third candidate determination unit 46). Specifically, the navigational mark identification device 1 determines the sign content corresponding to the standard time pattern that has been determined as the lighting pattern candidate as the third candidate.

[0139] Next, the navigational mark identification device 1 determines whether the current time is daytime or nighttime (S47), and if it is daytime, applies the daytime judgment criteria (S48), and if it is nighttime, applies the nighttime judgment criteria (S49). The current time is the time when the image was generated by the camera 2. The judgment criteria are used to determine the marking content of the buoy.

[0140] Next, the navigational mark identification device 1 determines the mark content of the buoy based on the first candidate mark content determined in S42, the second candidate mark content determined in S44, and the third candidate mark content determined in S46 (S50: processing as the mark content determination unit 47).

[0141] Specifically, when at least two of the first, second, and third candidates have the same beacon content, the navigational mark identification device 1 determines the same beacon content as the beacon content of the buoy. For example, when two of the first, second, and third candidates are port beacons and the remaining one is a starboard beacon, the port beacon is determined as the beacon content.

[0142] The navigational mark identification device 1 may also determine the mark content of the buoy based on the first, second, and third accuracies. For example, the candidate corresponding to the highest accuracy among the first, second, and third accuracies is determined as the mark content. In addition, when multiple candidates represent the same mark content, the corresponding accuracies may be added together.

[0143] The navigational mark identification device 1 changes the weighting assigned to each of the first accuracy, second accuracy, and third accuracy depending on the daytime judgment criterion and the nighttime judgment criterion. For example, during the daytime, priority is given to candidates related to the color of a buoy and the shape of a top mark that are easily visible in a bright environment, and during the night, priority is given to candidates related to the light pattern of a buoy that is easily visible in a dark environment.

[0144] That is, in the daytime judgment criteria, the weighting of the first and second probabilities related to the color of the buoy and the shape of the top mark is higher than the weighting of the third probabilities related to the way the buoy shines. Conversely, in the nighttime judgment criteria, the weighting of the third probabilities related to the way the buoy shines is higher than the weighting of the first and second probabilities related to the color of the buoy and the shape of the top mark.

[0145] The method for determining sign content according to this modified example may be applied not only to side signs and directional signs, but also to isolated obstacle signs, safe water signs, and special signs.

[0146] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and it goes without saying that various modifications can be made by those skilled in the art. [Explanation of symbols]

[0147] 1 Navigational Aid Identification Device, 2 Camera, 3 Radar, 4 AIS, 5 Radio Communication Unit, 6 Display Unit, 7 GNSS Receiver, 8 Gyrocompass, 9 ECDIS, 10 Automatic Steering Device, 11 First Acquisition Unit, 12 First Identification Unit, 13 Second Acquisition Unit, 14 Second Identification Unit, 15 Display Control Unit, 16 Route Calculation Unit, 17 Model Memory Unit, 21 Lens Unit, 22 Pan-Tilt Mechanism, 31 Color Identification Unit, 32 Shape Identification Unit, 33 Country Determination Unit, 34 Port / Starboard Determination Unit, 41 Color Identification Unit, 42 First Candidate Determination Unit, 43 Shape Identification Unit, 44 Second Candidate Determination Unit, 45 Light Pattern Identification Unit, 46 Third Candidate Determination Unit, 47 Beacon Content Determination Unit, 100 Autonomous Navigation System

Claims

1. a first acquisition unit that acquires a first image including the buoy from a camera installed on the ship; a first identification unit that identifies the position of the buoy within the first image; a second acquisition unit that acquires a second image having a higher resolution than the first image, the second image corresponding to a partial region of the first image that includes the position of the buoy; a second identification unit that identifies the marking content of the buoy from the second image; A navigational aid identification device comprising:

2. The camera has an optical zoom function, the second acquisition unit acquires the second image by causing the camera to capture an image of an enlarged range corresponding to the partial region of the first image.

2. The navigational beacon identification device according to claim 1.

3. The camera has a pan / tilt function, The second acquisition unit directs the imaging direction of the camera to a range corresponding to the partial region of the first image based on the position of the buoy.

3. The navigational beacon identification device according to claim 2.

4. the second acquisition unit acquires the second image by increasing the resolution of the partial region of the first image.

2. The navigational beacon identification device according to claim 1.

5. the second acquisition unit acquires the second image by causing an auxiliary camera having a higher resolution than the camera to capture an image of a range corresponding to the partial region of the first image.

2. The navigational beacon identification device according to claim 1.

6. The first identification unit identifies the position of the buoy in the first image using a first trained model; The second identification unit identifies the marking content of the buoy from the second image using a second trained model.

6. A navigational beacon identification device according to any one of claims 1 to 5.

7. The first trained model and the second trained model are configured such that different first trained parameters and second trained parameters are incorporated into a common inference program.

7. The navigational beacon identification device according to claim 6.

8. The first identification unit further identifies the type of the buoy.

8. A navigational beacon identification device according to any one of claims 1 to 7.

9. The second identification unit, when the type of the buoy is a side beacon, identifies whether the beacon content of the buoy is a port beacon or a starboard beacon.

9. The navigational beacon identification device according to claim 8.

10. When the type of the buoy is a direction indicator, the second identification unit identifies whether the indicator content of the buoy is a north direction indicator, an east direction indicator, a south direction indicator, or a west direction indicator.

10. The navigational beacon identification device according to claim 8 or 9.

11. The second identification unit is If the type of the buoy is a side marking, using a trained model for side marks, identify whether the marking content of the buoy is a port marking or a starboard marking; If the type of the buoy is a directional sign, using a trained model for directional signs, identify whether the sign content of the buoy is a north directional sign, an east directional sign, a south directional sign, or a west directional sign.

11. A navigational beacon identification device according to any one of claims 8 to 10.

12. The second acquisition unit does not acquire the second image when the type of the buoy is an isolated hazard beacon, a safe water beacon, or a special beacon.

12. A navigational beacon identification device according to any one of claims 8 to 11.

13. When a plurality of the buoys are identified from the first image, the second acquisition unit and the second identification unit acquire the second image and identify the sign content for each of the plurality of the buoys.

13. A navigational beacon identification device according to any one of claims 1 to 12.

14. a display control unit that displays a symbol representing the marking content of the buoy on the first image, an electronic nautical chart, or a radar image based on the marking content of the buoy, the position of the buoy in the first image, and the imaging direction of the camera; 14. A navigational beacon identification device according to any one of claims 1 to 13.

15. a consistency determination unit that determines consistency between the sign content of the buoy and the sign content represented by the navigational aid data recorded on the electronic nautical chart based on the sign content of the buoy, the position of the buoy in the first image, the imaging direction of the camera, and the position of the ship; 15. A navigational beacon identification device according to any one of claims 1 to 14.

16. Further, a display control unit is provided that displays the consistency determination result on the first image, the electronic nautical chart, or the radar image.

16. The navigational beacon identification device according to claim 15.

17. A navigational beacon identification device according to any one of claims 1 to 16; a route calculation unit that calculates the route or route width of the ship based on the position of the buoys in the first image and the imaging direction of the camera when the beacon contents of the buoys include at least two of a port beacon, a starboard beacon, and a safe water beacon; An autonomous navigation system comprising:

18. A navigational beacon identification device according to any one of claims 1 to 16; a virtual sign acquisition unit that acquires data representing the position and content of a virtual sign; a route calculation unit that calculates the route or route width of the ship based on the marking content of the buoy, the position of the virtual marking, and the marking content of the virtual marking; An autonomous navigation system comprising:

19. A navigational beacon identification device according to any one of claims 1 to 16; a position detection unit that detects the position of the ship; a route calculation unit that sets a waypoint through which the vessel should take based on the markings of the buoy, the position of the buoy in the first image, the imaging direction of the camera, and the position of the vessel; An autonomous navigation system comprising:

20. A navigational beacon identification device according to any one of claims 1 to 16; a heading detection unit for detecting a bow heading of the ship; a route calculation unit that sets a direction in which the vessel should navigate based on the markings of the buoy, the imaging direction of the camera, and the bow direction of the vessel; An autonomous navigation system comprising:

21. The vessel further includes an automatic steering device that performs autonomous navigation control based on the marker content of the buoy.

21. The autonomous navigation system according to any one of claims 17 to 20.

22. By a computer, acquiring a first image including the buoy from a camera mounted on the vessel; Identifying the location of the buoy within the first image; acquiring a second image having a higher resolution than the first image, the second image corresponding to a partial region of the first image including the position of the buoy; identifying marking content of the buoy from the second image; Navigation aid identification method.

23. acquiring a first image including the buoy from a camera mounted on the vessel; identifying the location of the buoy within the first image; acquiring a second image having a higher resolution than the first image, the second image corresponding to a partial region of the first image including the position of the buoy; and identifying marking content of the buoy from the second image; A program that causes a computer to execute the following.

Images