Information processing device, information processing method, and program

Abstract

This invention provides an information processing device, an information processing method, and a program that can enhance safety during ship navigation in a new way. [Solution] The information processing device according to this disclosure includes a control unit that performs: a receiving process for receiving detection target information, including location information of a predetermined detection target detected by a detection unit of each of the multiple first vessels in navigation; a storage process for aggregating and storing the received detection target information; and a transmission process for transmitting at least one of the received detection target information and predetermined output data generated using the detection target information under predetermined generation conditions to a predetermined second vessel.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, in order to enhance the safety when a ship is navigating, a system such as AIS (Automatic Identification System) is known as a means for sharing the position information of one's own ship with other ships. However, among small ships such as fishing boats, many ships do not have AIS installed, and there are problems regarding information sharing.

[0003] In contrast, for example, in Patent Document 1, a system has been proposed that transmits the position information of one's own ship to other ships using a device equipped with a GPS function so that even ships without AIS can share the position information of their own ships.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, it is realistically difficult to install devices equipped with AIS or GPS on all ships.

[0006] Therefore, the present disclosure has been made in view of the above problems, and an object thereof is to provide an information processing apparatus, an information processing method, and a program capable of enhancing the safety during ship navigation by a new method.

Means for Solving the Problems

[0007] According to this disclosure, the receiving process receives detection target information, including location information of a predetermined detection target detected by the detection unit of each of the first vessels, from a plurality of first vessels in navigation. A storage process that aggregates and stores the received detection target information, An information processing device is provided, which includes a control unit that performs a transmission process to transmit to a predetermined second vessel at least one of the received detection target information and predetermined output data generated using the detection target information under predetermined generation conditions. [Effects of the Invention]

[0008] This disclosure provides an information processing device, an information processing method, and a program that can enhance safety during ship navigation in a new way. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of a system according to one embodiment of the present disclosure. [Figure 2] This figure shows an example of an information processing device according to the present invention. [Figure 3] This is a flowchart illustrating a series of control operations in the system according to the same embodiment. [Figure 4] This figure shows an example of the information to be detected according to the same embodiment. [Figure 5] An example of a ship detection system according to the same embodiment is shown. [Figure 6] An example of a vessel according to the same embodiment is shown. [Figure 7] This is a flowchart illustrating a series of controls in the detection system according to the same embodiment. [Modes for carrying out the invention]

[0010] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0011] Figure 1 is a conceptual diagram of a system using the information processing device of this embodiment. Figure 1 illustrates two ships A1 and A2 that are in transit and can communicate with the information processing device 100, as well as a ship B, a floating object C, and a person D that cannot communicate with the information processing device 100.

[0012] The information processing device 100 shown in Figure 1 functions as a management server (e.g., a cloud server). As shown in Figure 2, the information processing device 100 comprises a control unit 120, a storage unit 130, an input unit 140, an output unit 150, and a communication unit 160. In this example, the control unit 120, storage unit 130, input unit 140, output unit 150, and communication unit 160 are implemented in an information processing device (computer), but the system is not limited to this configuration. The system may be configured by dividing it into multiple devices, or each part (at least a part of it) may be provided independently.

[0013] The control unit 120 includes a processor (arithmetic unit) such as a CPU and is configured to execute various information processing based on a program stored in the storage unit 130.

[0014] The storage unit 130 can store various types of information that have been stored in advance, information generated by the control unit 120, user operation information input via the input unit 140, and information received from ships or external devices via the communication unit 160. The storage unit 130 includes a main memory composed of a volatile storage device such as DRAM (Dynamic Random Access Memory), and an auxiliary storage device composed of a non-volatile storage device such as flash memory or HDD (Hard Disc Drive).

[0015] The storage unit 130 can store two-dimensional or three-dimensional map data (map information) corresponding to the real space, received information, various types of information generated based on the received information, various types of information regarding the detection target, condition information for various processes executed by the control unit, and the like. The various processes can store information regarding various conditions such as, for example, the matching process, position prediction process, target estimation process, etc., which will be described later. The storage unit 130 can store information regarding various parameters indicating the shapes (3D model data, etc.) of various ships, characteristics (characteristics regarding output, speed, turning amount), navigation history information, navigation route information, tidal information, and the like.

[0016] The input unit 140 receives input information based on operations from the user, etc. The input unit 140 is composed of a mechanical button, switch, operation lever, touch panel, keyboard, mouse, etc. The input unit 140 may be provided with a microphone capable of voice input, etc.

[0017] The output unit 150 outputs various types of information as images (videos), sounds, etc. The output unit 150 may be provided with, for example, a display unit such as a liquid crystal monitor or touch panel for displaying images, a sound output unit such as a speaker for displaying sounds, a vibration generation unit (vibration device) for generating vibrations, and the like.

[0018] The communication unit 160 is connected to a network such as the Internet or wireless communication, and receives and transmits data from ships, external devices (management servers, control devices, terminal devices used by users, etc.), etc.

[0019] Note that the control unit 120 can estimate the position, type, size, orientation, etc. of an obstacle by, for example, analyzing image data acquired by a camera (target estimation process). The control unit 120 may estimate the type of the obstacle by selecting any one from the candidates of the types of obstacles stored in the storage unit 130 in advance. The information on the type of the obstacle may include type information such as ships, specific ship classifications (fishing boats, water taxis, small multi-purpose boats, pleasure fishing boats, passenger ships, traffic boats, work boats, fire boats / guard boats, pleasure yachts, pleasure motor boats, special work boats, tankers, cargo ships, etc.), marine structures, people (including swimming, surfing, yachts, paddle boards, etc.), rocks, shores, driftwood, seaweed, marine organisms, and the like. The information on the type of the obstacle is stored in association with the information on the size (area, height).

[0020] In the obstacle information estimation process, the control unit 120 may estimate the position, type, size, orientation, etc. of the obstacle by using a machine learning technique using a learning model learned in advance. When the control unit 120 estimates the position of the obstacle based on the image data acquired by the camera, after estimating the three-dimensional position (coordinates) of the obstacle in the camera coordinate system, based on the information on the position (coordinates) and the shooting orientation of the camera, the camera coordinate system can be converted into the world coordinate system by coordinate conversion processing, and the position (coordinates) of the obstacle on the world coordinate system can be calculated. The control unit 120 can estimate the bow direction of the ship as an obstacle, the direction of the face of a person (front direction), the direction of an object used such as a surfboard (traveling direction), etc. by analyzing the image data from the camera or analyzing three-dimensional data (point cloud data, model data, etc.).

[0021] The control unit 120 of the information processing device can perform the following: receiving processing to receive detection target information, including location information of a predetermined detection target detected by the detection unit of each of the multiple first vessels in navigation; storing the received detection target information; and transmitting at least one of the received detection target information and predetermined output data generated using the detection target information under predetermined generation conditions to a predetermined second vessel. According to this embodiment, information detected by the first vessel can be stored and transmitted to other second vessels. As a result, for example, location information of a vessel that does not have AIS or GPS can be shared from the first vessel to the second vessel via the information processing device, thereby improving safety during navigation. Note that in Figure 1, vessel A1 may function as the first vessel and vessel A2 may function as the second vessel, or vice versa. Also, vessels A1 and A2 may function as the first vessel, or vessels A1 and A2 may function as the second vessel. Furthermore, at least one of vessels A1 and A2 may function as the first vessel, and other vessels not shown may function as the second vessel. In either case, if the detection targets such as vessel B, floating object C, and person D can be detected by either of the first vessels, they can be shared with other second vessels that have not detected them, thereby increasing safety during navigation.

[0022] For example, as shown in Figure 3, the control unit 120 performs a receiving process to acquire detection target information from the first vessel (S101). The receiving process only needs to receive detection target information from at least one of the first vessels. Furthermore, the information processing device maintains a state where it can continuously receive information as long as the control unit 120 does not stop its receiving function.

[0023] Next, the control unit 120 aggregates the detection target information and performs storage processing (S102). Aggregating and storing means that information obtained from multiple different vessels is combined and centrally managed. Storage processing is performed each time a reception process is completed. The control unit 120 may assign unique identification information (such as an ID containing a number or characters) to each detection target and store it in the detection target information table. Alternatively, it may store map information that reflects the location (coordinates) of the detection target in pre-stored map information.

[0024] Figure 4 shows an example of a detection target information table. In the example in Figure 4, the detection target information includes the type of detection target, location, attitude, movement speed, movement direction, size, detection time (time detected by the first vessel), reception time (time received by the information processing device), and transmitting vessel (the vessel that detected the information and transmitted it to the information processing device), each of which is aggregated and managed in relation to the identification information. Note that the detection target information may also include other information; for example, image information taken by the first vessel's camera for each detection target may be associated and stored. Furthermore, the detection target information received by the information processing device only needs to include location information; for example, attitude, movement speed, etc., may be unknown. In the example in Figure 4, for example, for the "driftwood" with identification information "B0003" transmitted from vessel "A1", information regarding attitude and movement could not be detected and is therefore unknown (indicated by "-" in the figure). Also, the attitude and movement direction information is expressed as a numerical value up to 360, with one of the cardinal directions (east, west, north, south) being 0°, but it is not limited to this. Furthermore, while the size is expressed in three stages—"large," "medium," and "small"—in the illustrated example, it is not limited to these three stages; there may be two or fewer stages, or four or more stages. The stages may be expressed numerically, as may the width (maximum width, diameter, etc.), height, area, volume, etc., be expressed numerically. This information can be estimated by known methods by analyzing image data acquired by the first vessel. Such analysis may be performed on the first vessel, or on an information processing device that receives the image data from the first vessel.

[0025] The control unit 120 may update the stored information repeatedly at predetermined intervals, regardless of whether or not reception processing has been performed. For example, if the detected object is moving, the control unit 120 may update the location information by predicting and storing the current location based on the elapsed time from a past time.

[0026] Then, the control unit 120 performs a process to determine the vessels to which the information will be transmitted (S103). The vessels to which the information will be transmitted may be all vessels within the communication range, or only some of them. Some of the vessels may be, for example, the vessel that sent the transmission request, but are not limited to this, and may be limited to only vessels that meet other predetermined conditions. Other conditions may be, for example, vessels that have transmitted their own information (ship type, position, owner information, purpose information, etc.), or only vessels that have been pre-approved and registered by the system for communication. In this case, identifiers may be added to the transmitted information or encryption processing may be performed so that only specific vessels can obtain the information. Such conditions for determining the vessels to which the information will be transmitted are stored in the storage unit in advance.

[0027] Furthermore, the control unit 120 performs the process of transmitting output information to a predetermined second vessel (S104). The control unit 120 can, for example, transmit the detection target information received so far to the vessel (second vessel) determined in the process of determining the transmission target vessel (S103). The output information to be transmitted may be the detection target information received from all first vessels in its original data format, or it may be transmitted in a predetermined data format. Alternatively, various data generated based on the detection target information received from the first vessel may be transmitted.

[0028] Furthermore, the transmission process may be repeated each time a reception or storage process occurs, each time the storage unit is updated, at regular intervals, or in response to a reception request from the second vessel.

[0029] Furthermore, the second vessel to be subjected to the transmission process may be all vessels capable of communication, or it may be only some of the vessels.

[0030] Here, the types of objects to be detected may include other vessels, floating objects, people, and fixed objects such as rocks and shorelines. Examples of other vessels include fishing boats, water taxis, small multi-purpose boats, recreational fishing boats, passenger ships, transport boats, workboats, fireboats / patrol boats, pleasure yachts, pleasure motorboats, special workboats, tankers, cargo ships, etc. Examples of floating objects include driftwood, seaweed, marine life, etc. Examples of people include swimmers, surfers, sailboats, paddleboarders, etc.

[0031] The information to be detected is not limited to location information, but may also include information such as orientation, whether or not it is moving, speed of movement, path of movement, type (other vessels, floating objects, people, etc.), and size. Various types of information are associated with identification information (such as a unique ID) assigned to each detected object and stored in aggregated manner. In addition, location information and orientation information may be stored in association with time information. Furthermore, various types of information such as location information may include not only the time of information acquisition (such as the current time at the time of detection) but also predicted location information after a predetermined time. Size information may include at least one of the following in plan view: maximum diameter (m), area (m^2), volume (m^3), and height from the water surface (m).

[0032] The control unit 120 may transmit at least one of the detection target information and output data in response to the transmission request information received from the second vessel. For example, the second vessel transmits transmission request information (signal) to the information processing device via wireless communication, satellite communication, the internet, etc., to request information about the detection target. When the information processing device receives the transmission request information, it can transmit the detection target information to the second vessel that sent the transmission request information in the same manner.

[0033] In this embodiment, the transmission request information includes the position information of the second vessel, and the control unit 120 may transmit to the second vessel information regarding detection targets located within a predetermined distance range from the second vessel's position. Specifically, the control unit 120 can extract detection targets located within a predetermined radius (100m, 200m, 500m, 1km, etc.) centered on the position information (coordinate information) of the second vessel from the information in the storage unit and generate output data. The output data may include all information about the detection targets included in the range, or it may consist of only some of the information. For example, the detection target information shown in Figure 4 may consist only of position information, or only of type and position information, or it may include other information such as image data. The transmission request information may also include time information (for example, the current time when the second vessel transmitted the transmission request signal). In that case, the control unit of the information processing device may generate output data based on the time information. For example, the system may extract only the information detected within a predetermined range (within 1 minute, 10 minutes, 30 minutes, 1 hour, etc.) from the given time and transmit it to the second vessel. Alternatively, the system may predict the position of each detected object based on the elapsed time from the detection time of each object shown in Figure 4 to the transmission request time of the second vessel, taking into account the speed and direction of movement, and generate and transmit output data including the predicted position (position prediction processing). In this way, the control unit of the information processing device can generate information corresponding to the time information included in the transmission request information and transmit it to the second vessel.

[0034] The control unit 120 may generate output data that reflects the detection target information on a pre-stored map and transmit it to the second vessel during the transmission process. In this case, the map data can be displayed on a display unit such as a monitor installed on the second vessel, making it easy to understand the location of the detection target. The map data may not be limited to the location information of the detection target, but may also display various information such as type, size, direction of movement, and speed of movement using text or diagrams (icons, etc.). This allows for a quick confirmation of where and what kind of detection target exists. Alternatively, the second vessel may generate map data similar to the output data by reflecting the location information and other information contained in the detection target information received from the information processing device onto the map data that is pre-stored on the second vessel.

[0035] In this embodiment, the detection target information includes the speed information (movement speed) and direction information (movement direction) of the detection target, and the control unit may generate output data that includes predicted position information of the detection target predicted based on the speed information and direction information. This allows the movement of the detection target to be communicated to the second vessel, thereby increasing safety and improving the navigation efficiency of the second vessel. That is, for example, if the direction of movement of the detection target is away from the second vessel, there is no need to avoid it unnecessarily, and there is no need to take an unnecessary detour, thus improving the navigation efficiency of the second vessel.

[0036] In this embodiment, the control unit may further perform a matching process to integrate duplicate detection target information by comparing multiple received detection target information, and transmit output data based on the information after the matching process to the second vessel. With this configuration, for example, if the same detection target is detected by multiple first vessels, the risk of mistakenly identifying one detection target as having multiple instances can be avoided. Information regarding the conditions for the matching process is stored in the storage unit in advance.

[0037] In this embodiment, the control unit may determine that two detection targets are the same if their positional information is within a predetermined distance during the matching process. For example, the control unit may determine that "B0001" and "B0005" in Figure 4 are the same detection target if their positional information is within a predetermined distance range. In that case, the control unit may prioritize keeping "B0005," which has the most recent time information, and delete and update "B0001."

[0038] In this embodiment, the control unit may determine that two detection targets are the same if their location information and type information satisfy predetermined conditions during the matching process. For example, the control unit may determine that "B0001" and "B0005" in Figure 4 are the same type and their location information is within a predetermined distance range. In this case, the control unit may prioritize keeping "B0005" with the most recent time information and delete and update "B0001". By adding type information as well as location information to the matching conditions, more accurate matching processing becomes possible. In addition, arbitrary information such as size information, movement speed, and movement direction can be added to the conditions, and the combination of these can also be changed arbitrarily.

[0039] In this embodiment, the control unit may store the received location information of the detected object in association with time information during the storage process. The time information may include, for example, the detection time when the first vessel detected the object, the reception time when the information processing device received the information about the detected object, or both, as shown in Figure 4.

[0040] In this embodiment, the detection target may include floating objects, people, or other objects such as fixed objects. In this way, information about obstacles that cannot be obtained by AIS or the like can be acquired.

[0041] In this embodiment, an information processing method is provided in which an information processing device performs the following: receiving processing to receive detection target information, including location information of a predetermined detection target detected by the detection unit of each of the multiple first vessels in navigation; storing the received detection target information; and transmitting at least one of the received detection target information and predetermined output data generated using the detection target information under predetermined generation conditions to a predetermined second vessel.

[0042] In this embodiment, a program is provided that causes a control unit to execute the following: a receiving process for receiving detection target information, including location information of a predetermined detection target detected by the detection unit of each of the multiple first vessels in navigation; a storage process for aggregating and storing the received detection target information; and a transmission process for transmitting at least one of the received detection target information and predetermined output data generated using the detection target information under predetermined generation conditions to a predetermined second vessel.

[0043] In this embodiment, the detection target information may include information regarding the accuracy of the detection unit (including error information). For example, the accuracy information may be expressed as 0 to 100% (or 0 to 1), and the control unit of the information processing device can store the accuracy information and generate output information including the accuracy information. For example, if different accuracy information is obtained for the same detection target position, the control unit may adopt and store the position information with the higher accuracy value, or it may perform weighting corresponding to the accuracy values ​​and calculate the position between multiple detection target information as the position information of the detection target. For example, based on a first position (x1, y1) information with a sensor detection accuracy of 0.6 and a second position (x2, y2) information with a sensor detection accuracy of 0.3, the control unit may estimate the position of the detection target to be shifted from the midpoint between the first and second positions by the ratio of the difference in accuracy to the first position. Such estimation conditions are stored in the storage unit in advance.

[0044] In this embodiment, the information to be detected may include information about the type (category) of the detection unit, such as a sensor or camera. The control unit of the information processing device can estimate the accuracy from the information about the type of the detection unit. Then, output information can be generated based on the accuracy, as described above.

[0045] In this embodiment, the information transmitted from the first vessel may include wind speed and direction obtained by the first vessel (its own vessel), estimated wave height, wavelength, and current, and this information can also be transmitted to the second vessel and shared. The second vessel can obtain various detection information that it cannot obtain on its own vessel, enabling safer and more efficient navigation planning.

[0046] In this embodiment, the information to be detected is not limited to the first vessel; it may also be received from sensors installed on land, such as in ports or on fixed buoys, or collected from the air by a mobile device such as a drone. In other words, it can be received from any device equipped with a detection unit and a transmission unit, and shared (transmitted) to a second vessel, etc.

[0047] In this embodiment, the second vessel may be any vessel that does not have or does not have a function for detecting other vessels. This makes it possible to enhance the safety of vessels that are unable to detect the position of other vessels.

[0048] In this embodiment, the first vessel may receive information such as its current position, destination position, intermediate positions, and estimated arrival times at each position, as well as information on the navigation route, and transmit this information to the second vessel.

[0049] Here, we will describe an example of a first vessel (e.g., vessels A1 and A2) capable of communicating with an information processing device. The second vessel may also have the same functions as the first vessel. That is, it may be capable of displaying information about the object to be detected received from the information processing device, or generating a travel path based on that detected object information.

[0050] The first vessel is equipped with detection units such as sensors and cameras, and can detect the positions of surrounding objects (detection targets). For example, the vessel is equipped with a camera, and by analyzing the images captured by the camera, it is possible to estimate the relative positions (3D coordinates in a camera coordinate system with the camera as the origin) of vessels, floating objects, people, etc., contained in the images with respect to the vessel (e.g., the camera) using known techniques. The vessel also has a self-position estimation function and can estimate the position of the detection target in real space (world coordinate system) based on the vessel's position information and the relative position information of the target to the vessel. Note that the position estimation of the detection target may be performed by other known methods. Furthermore, the vessel's position information may be estimated as the current position using coordinate information as positioning data acquired by a GNSS module, for example, or it may be estimated by self-position estimation processing based on a comparison between map data (3D point cloud data) acquired from Lidar and map information stored in a memory unit.

[0051] The ship can store the information detected by the detection unit, associating it with the time of detection. It can also transmit the detection information and the detection time information to an information processing device.

[0052] A ship may receive and acquire various information from other ships, including its own position information (ship type, size, attitude, speed, direction of travel, route information, etc.), and may also transmit the position information of the other ship to an information processing device. This allows the information processing device to transmit not only detection information but also the received position information of the other ship.

[0053] Figure 5 shows an example of an obstacle estimation system (hereinafter also simply referred to as "the system") installed on a first vessel according to this embodiment. The system of this embodiment can be installed on any type of vessel and is particularly suitable for relatively small vessels. Specifically, the system of this embodiment can be used on fishing boats, water taxis, small multi-purpose boats, recreational fishing boats, passenger ships, transport boats, workboats, fireboats / patrol boats, pleasure yachts, pleasure motorboats, special work vessels, etc., and can also be used on large passenger ships, tankers, cargo ships, etc.

[0054] The system of this embodiment is an obstacle estimation system 1 for detecting obstacles (objects to be detected) when a ship is navigating. This system 1 includes, for example, a data acquisition unit 10, a control unit 20, a storage unit 30, an input unit 40, an output unit 50, a communication unit 60, and a ship operation control unit 70. Each component is connected by wire or wireless and can communicate with each other. In this example, the control unit 20, storage unit 30, input unit 40, output unit 50, and communication unit 60 are implemented in an information processing device (computer), but the system is not limited to this, and each component may be independent.

[0055] The data acquisition unit 10 is equipped with multiple data acquisition means such as various sensors and cameras, and acquires various sensor data and image data. The data acquisition unit may include, for example, cameras, radar devices such as Lidar (Light detection and ranging), marine radar, and millimeter-wave radar, positioning devices such as ultrasonic ranging sensors, GNSS modules, and QZSS modules, AIS, inertial measuring units (IMUs), inertial sensors, wind direction sensors, wind speed sensors, speed sensors (ground speed sensors, water speed sensors), heading sensors, acceleration sensors, gyro sensors, magnetic compasses, satellite compasses, temperature sensors, humidity sensors, barometric pressure sensors, altitude sensors, infrared sensors, etc. The data acquisition unit 10 can acquire information about the ship's current position (coordinates), attitude (heading such as the direction of the bow), moving speed, direction of movement, heading and turning amount, information about the surrounding environment, and information about obstacles such as other ships and people. AIS is a system for sending and receiving various information between ships and between ships and land-based facilities, and for example, ship information such as the position, attitude (course), moving speed, and destination of each ship can be exchanged via wireless communication. The compass sensor may be a magnetic compass sensor that uses the Earth's magnetic field to calculate the ship's heading, a gyrocompass, a GPS compass, or the like.

[0056] The control unit 20 includes a processor (arithmetic unit) such as a CPU and is configured to perform various information processing based on a program stored in the storage unit 30. The control unit 20 can perform ship information estimation processing to estimate the ship's position information and ship's attitude information based on data (first data) from the data acquisition unit 10. The ship's position information can be estimated as the current position using coordinate information as positioning data acquired by the GNSS module. The ship's position information may also be estimated by a self-position estimation process based on a comparison between map data (3D point cloud data) acquired from Lidar and map information stored in the storage unit. The ship's position information is expressed as 2D or 3D coordinate information on a specific coordinate system and may be expressed in latitude and longitude. The ship's attitude is expressed as azimuth (angle) information on a specific 2D or 3D coordinate system. The attitude information may be expressed in east, west, north, and south directions.

[0057] The control unit 20 can perform a ship information correction process to correct the ship's position information and attitude information based on the data (second data) from the data acquisition unit 10. The control unit 20 can correct the ship's position information and attitude information based, for example, on speed data acquired from a Lidar or camera.

[0058] The control unit 20 can perform a speed estimation process to estimate the speed of its own vessel based on the data from the data acquisition unit 10. For example, it can calculate the distance traveled during a predetermined time (0.01 seconds, 0.1 seconds, 1 second, etc.) from the position information from a predetermined time ago and the current position information, and estimate the speed from the distance traveled during the predetermined time.

[0059] The control unit 20 can perform obstacle information estimation processing to estimate the position and size of obstacles based on the ship's position information and attitude information corrected by the ship's information correction processing, as well as data from the data acquisition unit 10 (third data). The control unit 20 can also estimate the type of obstacle based on the ship's position information and attitude information corrected by the ship's information correction processing, as well as data from the data acquisition unit 10 (third data).

[0060] The control unit 20 can estimate the type of obstacle by, for example, selecting one of the candidate types of obstacles that are pre-stored in the storage unit 30. The information on the type of obstacle can include type information such as ships, specific ship classifications (fishing boats, water taxis, small multi-purpose boats, recreational fishing boats, passenger ships, transport boats, workboats, fireboats / patrol boats, pleasure yachts, pleasure motorboats, special workboats, etc.), marine structures, people (including swimmers, surfers, sailboats, paddleboarders, etc.), rocks, shores, driftwood, seaweed, and marine organisms. The information on the type of obstacle is stored in association with size (area, height) information.

[0061] The control unit 20 can estimate the position, orientation, type, and size of an obstacle by, for example, performing image analysis on image data acquired by the camera. In the obstacle information estimation process, the control unit 20 may use machine learning techniques with a pre-trained learning model to estimate the position, orientation, type, and size of the obstacle. When the control unit 20 estimates the position of an obstacle based on image data acquired by the camera, it can first estimate the 3D position (coordinates) of the obstacle in the camera coordinate system, and then, based on the camera's position (coordinates) and shooting direction information, perform a coordinate transformation process to convert the camera coordinate system to the world coordinate system and calculate the position (coordinates) of the obstacle in the world coordinate system. The control unit 20 can estimate the bow direction of a ship, the direction of a person's face (forward direction), and the direction of an object being used (direction of movement), such as a surfboard, by performing image analysis on image data from the camera or by analyzing 3D data (point cloud data, model data, etc.). The control unit 20 can estimate the three-dimensional position and attitude of obstacles on or underwater based not only on image data from the camera as described above, but also on sensor data from radar, Lidar, or any combination thereof.

[0062] The memory unit 30 may store pre-associated information about candidate types of obstacles, their corresponding size (information regarding area and height in a plan view), and their size range (upper and lower limits). For example, the area range for a ship could be 3 m² or more and 9200 m² or less, and the height range for a ship (height above water) could be 0.5 m or more and 100 m or less. Similarly, the area range for a person could be 0.2 m² or more and 5 m² or less, and the height range for a person (height above water) could be 0.1 m or more and 3 m or less. The memory unit 30 stores information about the appropriate avoidance distance associated with obstacle information such as the position, type, and size of the obstacle. The memory unit 30 may also store information about a calculation formula for calculating the appropriate avoidance distance. For example, the appropriate avoidance distance could be calculated according to the size (plan view area) of a ship as an obstacle. The parameters used in the calculation may include information such as the type of obstacle, movement speed, and movement direction. The memory unit 30 may contain information about a program that simulates the ship's movement along its planned route and whether or not it comes into contact with obstacles. The memory unit 30 stores information about each data acquisition means. The information about the data acquisition means may include, but is not limited to, the position of the data acquisition means on the ship (for example, its relative position to a specific point such as the ship's center or center of gravity) and its relative orientation (for example, its direction relative to the ship's bow).

[0063] The control unit 20 may estimate the speed, direction, and attitude of obstacles based on the ship's position information and attitude information corrected by the ship's information correction process, the ship's speed information, direction of movement information, and data from the data acquisition unit 10 (third data). The control unit 20 can acquire relative speed information and direction information of obstacles such as other ships with respect to the ship based on data from the Lidar. The control unit 20 can also acquire speed information and direction information of obstacles using data from the radar device.

[0064] In obstacle information estimation processing, the control unit 20 can analyze image data acquired by the camera to estimate information such as the type, size, location, shape, and color of an obstacle. The control unit 20 may also estimate obstacle information based on any of the obstacle information estimated from the image data and obstacle-related information pre-stored in the memory unit. For example, the control unit 20 may estimate the size of an obstacle from the image data, refer to the memory unit, and determine (estimate) the type of obstacle based on candidate obstacles associated with a size range that matches that size. Alternatively, the control unit 20 may analyze image data acquired by the camera to estimate the type of obstacle, refer to the memory unit, and determine (estimate) the size of the obstacle based on candidate sizes associated with that type. Furthermore, the control unit 20 may estimate any of the obstacle type, size, and location based not only on image data acquired by the camera, but also on detection data acquired from a radar device, detection data from Lidar, or a combination thereof. For example, Lidar can generate 3D data of an obstacle from distance data to multiple points on the obstacle, and based on this 3D data, the 3D shape, type, size, and position of the obstacle can be estimated. In this way, the control unit 20 can select an obstacle from the candidates and estimate the type, size, etc. of the detected obstacle by comparing the information of obstacle candidates stored in the storage unit 30 with the data on obstacles acquired from the data acquisition unit 10. It is also possible for the control unit 20 to estimate the position and size of the obstacle based solely on the data from the data acquisition unit 10.

[0065] The control unit 20 can store various types of information, such as generated information, in the storage unit 30, update the information in the storage unit 30, output it from the output unit 50, and transmit it to external devices or other ships via the communication unit 60. In addition, the control unit 20 can receive information acquired by the data acquisition unit 10, user operation information input via the input unit 40, and information received by the communication unit 60, and perform information processing based on that information.

[0066] The memory unit 30 can store various types of information, including pre-stored information, information acquired from the data acquisition unit 10, information received from external devices or other vessels via the communication unit 60, information generated by the control unit 20, and information input via the input unit 40. It is composed of non-volatile memory or a hard disk. The memory unit 30 can store information on various parameters indicating the shape of the vessel (3D model data, etc.), characteristics (characteristics related to output, speed, turning amount), information on obstacles, 2D or 3D map information, navigation history information, navigation plan information, tidal information, and the like.

[0067] The input unit 40 receives input information based on user operations, etc. The input unit 40 consists of mechanical buttons, switches, operating levers, touch panels, etc. The input unit 40 may also be equipped with a microphone capable of voice input, etc.

[0068] The output unit 50 outputs various information in the form of images (videos), sound, etc. The output unit 50 may include, for example, a display unit such as an LCD monitor or touch panel for displaying images, an audio output unit such as a speaker for displaying sound, and a vibration generating unit (vibration device) for generating vibrations.

[0069] The communications unit 60 is connected to networks such as the Internet and wireless communication, and can transmit data to external devices (management servers, smartphones and tablet terminals used by users, etc.) and other ships, and receive data from external devices.

[0070] The ship's motion control unit 70 may include an auto-rudder for steering, an auto-throttle for controlling the power unit (engine, motor, etc.) for rotating the propellers and thrusters, etc. Based on instruction information from at least one of the control unit 20, memory unit 30, input unit 40, and communication unit 60, the ship's motion control unit 70 can control the ship's movements (movements such as moving forward, backward, left and right, turning, etc.) and navigate the ship along a predetermined path.

[0071] Figure 6 shows an example of the system installed on ship A. In the example in Figure 6, the data acquisition unit 10 includes a Lidar, camera, GNSS antenna, AIS antenna, and wind direction and speed meter, the control unit 20 is a control unit, the output unit 50 is a monitor, and the ship operation control unit 70 includes an auto rudder and auto throttle. In the example in Figure 6, the Lidar and camera are installed on the front (bow side) and rear (stern side) of ship 100, respectively, enabling high-precision detection of data around ship 100, especially on the front and rear sides. The configuration of the system installed on a ship is not limited to the illustrated example, and other sensors may be included.

[0072] Figure 7 shows an example of a processing flow for the symmetry detection method of the system of this embodiment.

[0073] The control unit 20 performs the following: a self-ship information estimation process (S201) which estimates the self-ship's position information and self-ship's attitude information based on first data acquired from at least one data acquisition means; a self-ship information correction process (S202) which corrects the self-ship's position information and self-ship's attitude information based on second data acquired from at least one data acquisition means; an obstacle information estimation process (S203) which estimates the position and size of obstacles based on the self-ship's position information and self-ship's attitude information corrected in the self-ship information correction process, and third data acquired from at least one data acquisition means; and an output information generation process (S204) which generates output information based on the position and size information of obstacles estimated in the obstacle information estimation process. The first data, second data, and third data may each include data acquired from a common data acquisition means. Furthermore, the first data, second data, and third data may each be data acquired from one data acquisition means, or they may include data acquired from multiple data acquisition means. Note that the self-ship information correction process is not mandatory, and the obstacle information estimation process may be performed after the self-ship information estimation process.

[0074] In the self-ship information estimation process (S201), for example, the three-dimensional position of the ship is estimated based on coordinate information from a GNSS antenna used as a data acquisition means, and the three-dimensional attitude information of the ship is estimated based on azimuth data acquired from an inertial measuring device. The self-ship information estimation process (S201) may be performed based on data acquired from only one data acquisition means, but from the viewpoint of improving estimation accuracy, it is preferable to perform the estimation process based on data acquired from multiple data acquisition means.

[0075] In the self-ship information correction process (S202), the self-ship position information and self-ship attitude information estimated in S101 are corrected based on, for example, speed information acquired from at least one of the Lidar and the camera. The control unit 20 estimates the current position and attitude (after a predetermined time has elapsed from that point in time) using, for example, the position (coordinates), direction of movement information, and speed of movement information at a predetermined time (0.01 seconds, 0.1 seconds, 1 second, etc.) prior. The control unit 20 then compares the estimated position and attitude information with the self-ship position information and self-ship attitude information estimated in the self-ship information estimation process (S201), and if they do not match, the values ​​between the two are adopted as the accurate current position and current attitude information, thereby correcting the self-ship position information and self-ship attitude information. Note that the self-ship information correction process may be performed by other methods as well; for example, the self-ship position information and self-ship attitude information may be corrected using position information and attitude information calculated by self-position and attitude estimation processing based on image data from the camera, or a combination of these methods may be used. Furthermore, from the viewpoint of improving the accuracy of the correction process, it is preferable to perform the correction process based on data acquired from multiple data acquisition means.

[0076] Next, in the obstacle information estimation process (S203), for example, the position and attitude of the camera are estimated based on the ship's position information and attitude information corrected in the ship information correction process (the position and attitude of the camera in the ship's coordinate system), and the position of the obstacle in the world coordinate system is estimated by performing a coordinate transformation using the relative position data of the obstacle in the camera coordinate system. In addition, the type and size of the obstacle are estimated by analyzing the camera image data. The estimation of the position, type, and size of the obstacle may be performed using detection data acquired from the radar system alone or in combination.

[0077] Then, in the output information generation process (S204), output information including various information such as the location, type, and movement speed, direction, and size of the obstacle can be generated and transmitted to the information processing device 100. The transmission process may be performed repeatedly at pre-stored intervals, or it may be performed each time a new target is detected. The output information may also include image data and audio data, and may display the location of the obstacle on the ship's monitor as a map, display the type and size of the obstacle as text or an image, or output audio from the speaker to warn the user. The content of the output information is not particularly limited; for example, navigation route information for avoiding obstacles may be generated and transmitted to the information processing device 100, displayed on a monitor, or applied to an autopilot system for automatic navigation. The output information may also include the current location, destination location, intermediate locations, and the estimated time of arrival at the destination location and intermediate locations.

[0078] The output information may include information on the distance to be maintained away from the obstacle (appropriate avoidance distance). For example, if the obstacle is small (e.g., 1 m² or less), the appropriate avoidance distance may be set to a smaller value (5 m, 10 m, etc.), and the larger the obstacle (e.g., 10 m² or more), the larger the appropriate avoidance distance may be set to a larger value (20 m, 50 m, etc.). Also, for example, if the obstacle is a person, the appropriate avoidance distance may be set to a larger value (20 m, 50 m, etc.), and if it is floating object such as a buoy, driftwood, or seaweed, the appropriate avoidance distance may be set to a smaller value (5 m, 10 m, etc.). The appropriate avoidance distance may be determined based on appropriate avoidance distance information stored in the memory unit in advance, depending on at least one of the type and size of the obstacle. In that case, the information on the type and size of the obstacle and the information on the appropriate avoidance distance are stored in association with each other in advance. Furthermore, if the obstacle is another vessel, the appropriate avoidance distance may be calculated based on the speed and direction of movement of that other vessel.

[0079] In this embodiment, the control unit 20 may perform various processes based on information (including error information) regarding the accuracy (precision) of the detection unit (sensor, camera, etc.) that has been stored in the memory unit in advance. For example, in the ship position estimation process, the control unit 20 may prioritize the use of information from high-precision sensors whose accuracy value is above a predetermined threshold (and not use information from low-precision sensors whose accuracy value is below the threshold), or it may integrate and use the detection information based on weightings stored in the memory unit in advance. For example, the accuracy information may be expressed as 0 to 100% (or as a decimal between 0 and 1), and the control unit of the information processing device can store and update the accuracy information based on input information from the user. Various processes may include at least one of the following: processes for estimating ship position information and ship attitude information, processes for correcting these, and processes for estimating obstacles. In this way, by prioritizing the use of high-precision detection units in information processing, the accuracy of various processes such as ship position, ship attitude, and obstacle detection can be improved. The control unit 20, for example, if it has multiple different sensors capable of estimating the ship's position information, can estimate a position determined based on the weighting of accuracy information between the positions detected by the multiple sensors. If, among the multiple ship position information detected (estimated), different accuracy information is obtained for the same detected target position, the control unit may choose to store the position information with the higher accuracy value, or it may perform weighting corresponding to the accuracy values ​​and calculate the position between the multiple detected target information as the position information of the detected target. For example, based on a first position (x1, y1) information with a sensor detection accuracy of 0.6 and a second position (x2, y2) information with a sensor detection accuracy of 0.3, the control unit may estimate the position of the detected target as a position coordinate that is closer to the first position by the ratio of the difference in accuracy than the midpoint between the first and second positions. Such estimation conditions are stored in the memory unit in advance.

[0080] While preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art of the present disclosure that various modifications or alterations may be conceived within the scope of the technical idea set forth in the claims, and these will naturally also fall within the technical scope of the present disclosure.

[0081] The apparatus described herein may be implemented as a single device, or as a group of devices (e.g., cloud servers) that are partially or entirely connected by a network. For example, the control unit and the storage unit may be implemented as different servers connected to each other by a network.

[0082] The series of processes performed by the apparatus described herein may be implemented using software, hardware, or a combination of software and hardware. Computer programs for implementing each function of the control unit according to this embodiment can be created and implemented on a PC or the like. Furthermore, a computer-readable recording medium containing such a computer program can also be provided. Examples of recording media include magnetic disks, optical disks, magneto-optical disks, and flash memory. Alternatively, the computer program may be distributed without using a recording medium, for example, via a network.

[0083] Furthermore, the processes described using flowcharts in this specification do not necessarily have to be executed in the order shown. Some processing steps may be executed in parallel. Additional processing steps may be adopted, and some processing steps may be omitted.

[0084] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that are obvious to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein.

[0085] Furthermore, the following configurations also fall within the technical scope of this disclosure. (Item 1) A receiving process that receives detection target information, including location information of a predetermined detection target detected by the detection unit of each of the first vessels, from multiple first vessels that are navigating, A storage process that aggregates and stores the received detection target information, An information processing device comprising a control unit that performs a transmission process to transmit to a predetermined second vessel at least one of the received detection target information and predetermined output data generated using the detection target information under predetermined generation conditions. (Item 2) The information processing apparatus according to item 1, wherein the control unit transmits at least one of the detection target information and the output data in response to transmission request information received from the second vessel. (Item 3) The aforementioned transmission request information includes the self-position information of the second vessel, The control unit, The information processing device according to item 2, which transmits to the second vessel information about the detected object located within a predetermined distance range from the second vessel's own position. (Item 4) The information processing device according to item 1, wherein the control unit generates output data that reflects the detection target information on a pre-stored map and transmits it to the second vessel in the transmission process. (Item 5) The detection target information includes speed information and direction of travel information of the detection target, The information processing apparatus according to item 1, wherein the control unit generates output data including predicted position information of the detected target predicted based on the speed information and direction of travel information. (Item 6) The control unit further, By comparing the multiple pieces of detection target information received, a matching process is performed to integrate the overlapping detection target information. The information processing device described in item 1 transmits output data based on the information after the matching process to the second vessel. (Item 7) The control unit, The information processing device according to item 6, which determines that two detection targets are the same target when their positional information is within a predetermined distance in the matching process. (Item 8) The control unit, The information processing device described in item 6, which determines that two detection targets are the same when their location information and type information satisfy predetermined conditions in the matching process. (Item 9) The information processing apparatus according to item 1, wherein the control unit stores the received location information of the detected object in association with time information during the storage process. (Item 10) The detection target includes other vessels, as described in item 1, and is an information processing device. (Item 11) The information processing device described in item 1, wherein the object to be detected includes suspended matter. (Item 12) The detection target includes a person, and the information processing device is as described in item 1. (Item 13) A receiving process that receives detection target information, including location information of a predetermined detection target detected by the detection unit of each of the first vessels, from multiple first vessels that are navigating, A storage process that aggregates and stores the received detection target information, An information processing method comprising: an information processing device performing a transmission process to transmit to a predetermined second vessel at least one of the received detection target information and predetermined output data generated using the detection target information under predetermined generation conditions. (Item 14) A receiving process that receives detection target information, including location information of a predetermined detection target detected by the detection unit of each of the first vessels, from multiple first vessels that are navigating, A storage process that aggregates and stores the received detection target information, A program that causes a control unit to perform a transmission process to transmit to a predetermined second vessel at least one of the received detection target information and predetermined output data generated using the detection target information under predetermined generation conditions. [Explanation of symbols]

[0086] 1. Travel Path Generation System 10 Data acquisition unit 20 Control Unit 30 Storage section 40 Input section 50 Output section 60 Communications Department 70 Ship Operation Control Unit

Claims

1. A receiving process that receives detection target information, including location information of a predetermined detection target detected by the detection unit of each of the first vessels, from multiple first vessels that are navigating, A storage process that aggregates and stores the received detection target information, An information processing device comprising a control unit that performs a transmission process to transmit to a predetermined second vessel at least one of the received detection target information and predetermined output data generated using the detection target information under predetermined generation conditions.

2. The information processing apparatus according to claim 1, wherein the control unit transmits at least one of the detection target information and the output data in response to transmission request information received from the second vessel.

3. The aforementioned transmission request information includes the self-position information of the second vessel, The control unit, The information processing device according to claim 2, which transmits to the second vessel information relating to the detection target located within a predetermined distance range from the second vessel's own position.

4. The information processing apparatus according to claim 1, wherein the control unit generates output data that reflects the detection target information on a pre-stored map and transmits it to the second vessel in the transmission process.

5. The detection target information includes speed information and direction of travel information of the detection target, The information processing apparatus according to claim 1, wherein the control unit generates output data including predicted position information of the detected target predicted based on the speed information and direction of travel information.

6. The control unit further, By comparing the multiple pieces of detection target information received, a matching process is performed to integrate the overlapping detection target information. The information processing apparatus according to claim 1, which transmits output data based on the information after matching processing to a second vessel.

7. The control unit, The information processing apparatus according to claim 6, wherein, in the matching process, if the positional information of two detection targets is within a predetermined distance, it is determined that they are the same detection target.

8. The control unit, The information processing apparatus according to claim 6, wherein, in the matching process, if the location information and type information of the two detection targets satisfy predetermined conditions, it is determined that they are the same detection target.

9. The information processing apparatus according to claim 1, wherein the control unit stores the received location information of the detected object in association with time information during the storage process.

10. The information processing apparatus according to claim 1, wherein the detection target includes other ships.

11. The information processing apparatus according to claim 1, wherein the object to be detected includes floating matter.

12. The information processing apparatus according to claim 1, wherein the detection target includes a person.

13. A receiving process that receives detection target information, including location information of a predetermined detection target detected by the detection unit of each of the first vessels, from multiple first vessels that are navigating, A storage process that aggregates and stores the received detection target information, An information processing method comprising: an information processing device performing a transmission process to transmit to a predetermined second vessel at least one of the received detection target information and predetermined output data generated using the detection target information under predetermined generation conditions.

14. A receiving process that receives detection target information, including location information of a predetermined detection target detected by the detection unit of each of the first vessels, from multiple first vessels that are navigating, A storage process that aggregates and stores the received detection target information, A program that causes a control unit to perform a transmission process to transmit to a predetermined second vessel at least one of the received detection target information and predetermined output data generated using the detection target information under predetermined generation conditions.

Images