Information processing device, information processing method, information processing system, and computer program

Abstract

[Problem] To provide an information processing device, an information processing method, and a computer program that contribute to the realization of appropriate automatic navigation of a waterborne moving body. [Solution] An information processing device comprising: an interface unit that inputs a ship-handling instruction from an operator as linguistic data; a current status acquisition unit that acquires sea condition data indicating the operating status of a waterborne moving body to be operated; a command output unit that, when the sea condition data and the linguistic data are input, outputs command data using a learning model that outputs command data that conforms to the instruction of the linguistic data according to the status indicated by the sea condition data; and a control output unit that outputs control data for operating and controlling the waterborne moving body on the basis of the output of the learning model.

Description

Information processing device, information processing method, information processing system, and computer program

[0001] The present invention relates to an information processing device, an information processing method, an information processing system, and a computer program relating to the navigation of a water vehicle.

[0002] A method for autonomous navigation of water vehicles such as ships and drones has been proposed. Autonomous navigation is achieved by determining the rudder angle and speed to navigate to a predetermined position on a nautical chart using position data, navigation speed, and direction data obtained from a GPS (Global Positioning System) receiver installed on the water vehicle.

[0003] Patent Document 1 discloses a method for realizing automatic navigation by recording ship maneuvering operations such as engine speed and steering angle from departure to sailing out to sea, returning to port, and docking, and then reproducing the recorded ship maneuvering operations. Patent Document 2 proposes a ship maneuvering control that can control in cooperation with other ships.

[0004] Patent No. 5566426 Patent No. 5972201

[0005] The sea on which a surface vehicle navigates can change depending on the sea conditions, such as wind direction and wave height, as well as the navigation conditions of other vehicles on that day. When unexpected changes in other vehicles or weather occur, a surface vehicle that uses a control method that reproduces pre-recorded maneuvers can have difficulty reaching its destination.

[0006] The present disclosure aims to provide an information processing device, an information processing method, an information processing system, and a computer program that contribute to realizing appropriate automatic navigation of a water vehicle.

[0007] One aspect of the information processing device disclosed herein includes an interface unit that inputs maneuvering instructions from an operator as language-type data, a current status acquisition unit that acquires sea condition data indicating the operating status of the water vehicle to be operated, and a command output unit that, when the sea condition data and the language-type data are input, uses a learning model to output command data that conforms to the instructions of the language-type data according to the situation indicated by the sea condition data, and outputs command data.

[0008] In one aspect, a learning model is used that is trained to output command data in response to input of linguistic instructions from an operator and sea state data for maneuvering a surface vehicle, thereby obtaining appropriate command data by taking into account predicted sea state data in response to the operator's instructions.

[0009] One aspect of the information processing device of the present disclosure includes a control output unit that outputs control data for operating and controlling the water vehicle based on the output of the learning model.

[0010] In one aspect, control data for steering equipment such as a steering gear, a propulsion generator, and a power source of a surface vehicle is also output, enabling automatic navigation based on instructions in natural language.

[0011] In one aspect of the information processing device of the present disclosure, the command output unit outputs notification data including at least one of instruction content corresponding to the maneuvering command, sea state data on which the operation control is based, and instruction content corresponding to the command data.

[0012] In one aspect, the command content, the rationale for the operational control of the surface vehicle, and the command content are output and available for the operator to review, allowing the operator to recognize the causal relationship between their own instructions and the navigation control.

[0013] In one aspect of the information processing device of the present disclosure, the language-type data is speech or text in a natural language.

[0014] In one aspect, speech or text in natural language from an operator is accepted as language-type data, so that even if the operator issues an abstract instruction in natural language, command data is automatically output in response to the instruction.

[0015] In one aspect of the information processing device of the present disclosure, the learning model includes a multimodal model and a natural language model.

[0016] In one aspect, the learning model accepts both image data and text, and appropriate command data can be output in natural language based on the events indicated by the image data and the content of the text regarding maneuvering instructions learned in the language model.

[0017] In one aspect of the information processing device of the present disclosure, the learning model is provided with ship-maneuvering knowledge data of the water vehicle as premise data.

[0018] In one aspect, by providing ship-maneuvering knowledge data to the language model as premise data, command data that conforms to the ship-maneuvering knowledge is more likely to be output from the learning model.

[0019] One aspect of the information processing device of the present disclosure causes another device to execute part of the calculations in the learning model.

[0020] In one aspect, a large-scale learning model is required to output command data, which is a response to an instruction in natural language, using a language model. Processing efficiency can be improved by performing the processing on an external server device or external service, rather than performing the processing solely within the information processing device installed on the watercraft. If it is difficult for the watercraft to communicate with these server devices wirelessly, command data may be output using a relatively small language model trained only on natural language related to vessel navigation knowledge.

[0021] One aspect of the information processing device of the present disclosure includes a processing unit that processes sea condition data acquired by the current condition acquisition unit using an algorithm different from that of the learning model, and the command output unit inputs the sea condition data after processing by the processing unit into the learning model.

[0022] In one aspect, before providing sea condition data to the learning model, other learning models of a different type than the learning model or algorithms other than the learning model may be used to generate sea condition data including identification of objects in the image, depth distribution, and situation data estimated from sensor input.

[0023] In one aspect of the information processing device of the present disclosure, the sea condition data includes data obtained from at least one of a camera, radar, sonar, a GPS receiver, a weather sensor, an automatic identification system, and a fish finder.

[0024] In one aspect, command data is output according to the situation indicated by sea condition data obtained from a camera, radar, sonar, GPS receiver, weather sensor, automatic identification system, fish finder, etc.

[0025] In one aspect of the information processing device of the present disclosure, the command output unit inputs data obtained from at least one of the radar, sonar, GPS receiver, weather sensor, automatic identification system, and fish finder as sea condition data into the learning model as image data.

[0026] In one aspect, the distribution of detection results obtained from radar, sonar, GPS receivers, weather sensors, automatic identification systems, and fish finders is input into a learning model as image data. By using a multimodal model that has been trained to be able to verbalize the content of image data, it is expected that information can be extracted from the distribution image with high recognition accuracy and reflected in command data.

[0027] In one aspect of the information processing device of the present disclosure, the command data includes at least one of the steering angle, propulsion force, output from the power source of the water vehicle, and command content for equipment mounted on the water vehicle.

[0028] In one aspect, in response to an instruction that is language-type data, it becomes possible to control any of the steering angle, propulsion force, output from the power source of the water vehicle, and the command content for the equipment installed on the water vehicle.

[0029] In one aspect of the information processing device of the present disclosure, the learning model outputs predicted data of sea state data related to the input maneuvering instruction together with the command data.

[0030] In one aspect, the learning model outputs prediction data based on input sea state data. By outputting the prediction data that is the basis for the command data, the operator can recognize the basis for the output command.

[0031] One aspect of the information processing device of the present disclosure includes a prediction accuracy determination unit that determines the prediction accuracy of the prediction data output by the learning model by comparing it with sea condition data sequentially acquired by the current condition acquisition unit, and the learning model receives feedback of the prediction accuracy and outputs prediction data or command data.

[0032] In one aspect, the prediction accuracy of the prediction data calculated from the input sea state data is verified sequentially, which is expected to improve the prediction accuracy of the situation that is the basis for command data output.

[0033] One aspect of the information processing device of the present disclosure includes a strategy control unit that creates a control strategy for stable navigation of the water vehicle, and the command output unit outputs command data that conforms to the control strategy by inputting the control strategy into the learning model.

[0034] In one aspect, feedback is provided on whether the ship steering control output from the learning model conforms to the control strategy for safe navigation, and it is expected that the learning model will output command data that conforms to safe navigation.

[0035] In one aspect of the information processing device of the present disclosure, the water vehicle is an unmanned vessel.

[0036] In one aspect, autonomous navigation of unmanned ships can be achieved by having an operator remotely communicate maneuvering instructions using language-type data.

[0037] One aspect of the information processing system disclosed herein comprises a control device that controls the operation of a surface vehicle to be operated, and an information processing device that outputs control data for operating and controlling the surface vehicle to the control device, wherein the information processing device comprises an interface unit that inputs maneuvering instructions from an operator as language-type data, a current status acquisition unit that acquires sea condition data indicating the operating status of the surface vehicle via communication, a command output unit that outputs command data using a learning model that, when the sea condition data and the language-type data are input, outputs command data that conforms to the instructions of the language-type data according to the situation indicated by the sea condition data, a control output unit that outputs control data for operating and controlling the surface vehicle based on the output of the learning model, and a communication unit that provides the control data to the control device via communication.

[0038] In one aspect, the information processing device that outputs control data using a learning model that has been trained to output command data in response to input of linguistic instructions from an operator and sea state data may be a device mounted on land or another ship, rather than on the water vehicle, so that appropriate command data that takes into account predicted sea state data can be remotely provided in response to the operator's instructions.

[0039] One aspect of the information processing method disclosed herein involves inputting maneuvering instructions from an operator as language-type data, obtaining sea condition data indicating the operating status of the surface vehicle to be operated, using a learning model that, when the sea condition data and language-type data are input, outputs command data that conforms to the instructions of the language-type data according to the operating environment of the sea condition data, and outputting control data for operating and controlling the surface vehicle based on the output of the learning model.

[0040] One aspect of the computer program of the present disclosure causes a computer to input maneuvering instructions from an operator as language data, acquire sea condition data indicating the operating status of the surface vehicle to be operated, use a learning model that, when the sea condition data and the language data are input, outputs command data that conforms to the instructions of the language data according to the operating environment of the sea condition data, and executes a process of outputting control data for operating and controlling the surface vehicle based on the output of the learning model.

[0041] 1 is a schematic diagram of a navigation system of the present disclosure. FIG. 1 is a block diagram showing the configuration of an information processing device. FIG. 1 is a block diagram showing the configuration of a server device. FIG. 2 is an explanatory diagram of functions based on an information processing program. FIG. 3 is a schematic diagram of a learning model. FIG. 4 is an explanatory diagram of another example of functions based on an information processing program. FIG. 5 is a flowchart showing an example of a command data output processing procedure by an information processing device. FIG. 6 is an explanatory diagram showing an example of a display screen on a display unit. FIG. 7 is an explanatory diagram showing another example of a display screen on a display unit. FIG. 8 is an explanatory diagram of functions based on an information processing program in a second embodiment. FIG. 9 is a schematic diagram of another aspect of a learning model. FIG. 10 is a flowchart showing an example of a processing procedure by an information processing device in a second embodiment. FIG. 11 is an explanatory diagram of functions based on an information processing program in a third embodiment. FIG. 12 is a flowchart showing an example of a command data output processing procedure by an information processing device in a third embodiment.

[0042] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present disclosure will be specifically described with reference to the drawings illustrating embodiments thereof. In the following embodiments, a navigation system in which an information processing device according to the present disclosure is applied to automatic control of a ship will be described.

[0043] The term "ship" in this disclosure refers to a pleasure boat, a fishing boat, a merchant ship, etc. The term "ship" is an example of a waterborne vehicle, including a drone that navigates on the water, a drone that navigates underwater, and an air vehicle that moves within a predetermined height above the water surface, and can be interpreted as any of these.

[0044] First Embodiment Fig. 1 is a schematic diagram of a navigation system 100 according to the present disclosure. The navigation system 100 is a system that realizes automatic navigation of a vessel S. The vessel S may be a water drone, a small vessel such as a pleasure boat or a fishing boat, or a large vessel such as a merchant ship. The navigation system 100 according to the present disclosure realizes automatic navigation of the vessel S by receiving instructions in natural language from an operator and determining command data based on the received instructions and future sea conditions that can be predicted from sea condition data obtained from inside and outside the vessel S.

[0045] The navigation system 100 creates and uses a learning model M1 that is trained to output command data for the ship S when instructions in natural language and sea state data are input. The learning model M1 uses a large multimodal model (LMM) and is trained to input text or voice indicating instructions from an operator, and numerical values ​​and images indicating sea states, and to output command data.

[0046] The navigation system 100 includes an information processing device 1 that outputs control data for operating and controlling the ship S, a control device 2 that controls the navigation of the ship S, and a sensor 3 that acquires navigation data used for control. The information processing device 1 is a computer capable of communication and is connected to the control device 2 and the sensor 3.

[0047] The control device 2 outputs control signals to the steering gear, the propulsion generating device, and / or the power source installed on the ship S. The control device 2 creates control signals corresponding to the command data provided by the information processing device 1 and outputs them to the steering gear, the propulsion generating device, and / or the power source. The control device 2 outputs a signal corresponding to the steering angle to the steering gear. The propulsion generating device is, for example, a screw, a propeller, etc. The control device 2 outputs a signal to the propulsion generating device instructing it to be on or off, and a signal corresponding to the rotation speed when it is on. The power source is an engine or a secondary battery. The control device 2 outputs a signal to the power source instructing the output amount.

[0048] The sensors 3 include a camera 301 that photographs the sea, a radar 302, a sonar 303, a speedometer 304, a GPS receiver 305, and the like that are mounted on the vessel S. The sensors 3 also include an Automatic Identification System (AIS) 306. The sensors 3 also include sensors that monitor the rudder angle of the steering gear, the thrust of the thrust generating device, and the output from the power source. The sensors 3 may also include a fish finder and may also include a weather sensor.

[0049] In the navigation system 100, the information processing device 1 receives instructions from an operator in natural language, and inputs the instructions and images and numerical values ​​indicating sea conditions obtained by the sensor 3 into the learning model M1 to obtain command data. The navigation system 100 automatically derives the command data and outputs it to the control device 2, enabling automatic navigation of the ship S.

[0050] In the navigation system 100 of the present disclosure, the information processing device 1 executes a process of outputting command data obtained by the learning model M1 to the display unit 13 so that the operator can confirm it. At this time, in response to the received instruction, the navigation system 100 outputs text and images indicating the predicted sea conditions and the command data, along with images obtained from the sea condition data, to the display unit 13 of the information processing device 1. This not only enables automatic derivation of command data and automatic navigation, but also allows the operator to confirm what sea condition predictions were used to derive the command data, thereby enabling them to understand the causal relationships of the command data.

[0051] As shown in Figure 1, the navigation system 100 can be connected to a server device 4 or an external server via a network N including communication media such as satellite communication and a carrier network. The server device 4 or the external server provides information obtained from a weather forecast service, a service providing information obtained from observation satellites, and a sea condition prediction service. The external server refers to a group of servers that provide each of these services, and the server device 4 is a device that compiles and provides information as sea condition data used in the navigation system 1000. The server device 4 may also be a device that can collect radar and sonar information provided by multiple ships S that use the navigation system 100 and provide it to other ships S.

[0052] 2 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 is a personal computer and includes a processing unit 10, a storage unit 11, a communication unit 12, a display unit 13, an operation unit 14, and an audio input / output unit 15.

[0053] The processing unit 10 includes one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), etc. The processing unit 10 also includes a temporary storage medium such as a static random access memory (SRAM) or a dynamic random access memory (DRAM). The processing unit 10 reads out an information processing program P1 stored in the storage unit 11 onto the temporary storage medium and executes it, thereby causing a general-purpose computer to execute various processes described below and function as the information processing device 1 of the navigation system 100 of the present disclosure.

[0054] The memory unit 11 is a relatively large-capacity non-volatile memory area such as an SSD or a hard disk. The memory unit 11 stores a program (program product) required for the processing unit 10 to execute processing and reference setting data. The setting data may include identification information of the ship S on which the information processing device 1 is installed, the type, model number, and identification information of the operator (pilot) of the ship S. The program product includes an information processing program P1, images to be displayed on the display unit 13, and their design (arrangement). The memory unit 11 stores a learning model M1. Part or all of the learning model M1 may be stored in the server device 4 or an external server. The memory unit 11 stores general knowledge related to ship control (ship-maneuvering knowledge) and information on the operator operating the ship S. The memory unit 11 stores data related to sea conditions (sea condition data) obtained from the various sensors 3. The memory unit 11 may also store nautical chart data.

[0055] One of the information processing program (computer product) P1 and the learning model M1 stored in the memory unit 11 may be the information processing program P9 or the learning model M9 stored in a computer-readable non-transitory storage medium 9, which the processing unit 10 reads and stores in the memory unit 11. One of the information processing program P1 and the learning model M1 may be downloaded by the processing unit 10 from the server device 4 or another download server via the communication unit 12 and stored in the memory unit 11.

[0056] The communication unit 12 realizes communication with the server device 4 or an external server via the network N. The communication unit 12 may use a communication module that communicates with onshore devices or other ships using an AIS-dedicated frequency. The processing unit 10 can obtain navigation information such as time information, speed, and navigation status from the server device while notifying an onshore maritime traffic center of information via the onshore device using the communication unit 12. The processing unit 10 may also be able to transmit command data to the control device 2 of the ship S to be operated via the communication unit 12. The communication unit 12 may be a wireless communication module that connects to a carrier network, a wireless communication module for Wi-Fi, or a communication module that realizes communication with the server device 4 or an external server via satellite communication.

[0057] The display unit 13 is a display such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 13 is, for example, a display with a built-in touch panel. The processing unit 10 can output command data to the display unit 13 so that the operator can check it.

[0058] The operation unit 14 is a user interface capable of inputting and outputting data to and from the processing unit 10, and may be a touch panel built into the display unit 13, or may include physical buttons, switches, and physical dials. A part of the operation unit 14 may be a remote controller having a button with a function of identifying a direction, and may have a function of identifying the direction in which the remote controller itself is facing when the button is pressed.

[0059] The audio input / output unit 15 includes a speaker and a microphone. The processing unit 10 or the audio input / output unit 15 itself has a voice recognition function that converts audio collected using a microphone into text data. The microphone may be provided in the remote controller described above. The audio input / output unit 15 may output audio data of the audio collected using the microphone to the storage unit 11. The processing unit 10 can output sound effects and audio corresponding to processing using a speaker. Based on the information processing program P1, the processing unit 10 can output audio explaining the command content determined by the processing described below from the speaker of the audio input / output unit 15.

[0060] 3 is a block diagram showing the configuration of the server device 4. The server device 4 includes a processing unit 40, a storage unit 41, and a communication unit 42. The server device 4 may be configured as a single server computer, or may be configured to distribute processing among multiple server computers. The processing unit 40 includes one or more arithmetic processing units such as a CPU, an MPU, or a GPU. The processing unit 40 includes a temporary storage medium such as an SRAM or a DRAM.

[0061] The storage unit 41 is a relatively large-capacity nonvolatile storage area such as an SSD, a hard disk, etc. The storage unit 41 stores a large language model.

[0062] The communication unit 42 realizes communication with the information processing device 1 via the network N. Specifically, the communication unit 42 is a network card. The communication unit 42 may be a wireless communication device that connects to a carrier network, or may be a wireless communication device for Wi-Fi. The processing unit 40 can send and receive data to and from the information processing device 1 via the communication unit 42.

[0063] The server device 4 receives data from the information processing device 1 and executes part of the processing using the learning model M1 performed by the information processing device 1. When the information processing device 1 includes a large-scale language model in the learning model M1, the server device 4 is not essential.

[0064] In the navigation system 100 configured as described above, a method for determining command data by the information processing device 1 will be described. Fig. 4 is an explanatory diagram of functions based on the information processing program P1. The processing unit 10 performs various functions shown in Fig. 4 based on the information processing program P1 and the learning model M1.

[0065] The processing unit 10 functions as an interface unit 101 that accepts instructions from an operator as language-type data. As the interface unit 101, the processing unit 10 accepts voice instructions from the operator as language-type data via the voice input / output unit 15. As the interface unit 101, the processing unit 10 may also accept instructions from the operator via text input in natural language via the operation unit 14. As the interface unit 101, the processing unit 10 may also accept via the operation unit 14 the operator's selection of an instruction from options on a screen displayed on the display unit 13. As the interface unit 101, the processing unit 10 formats the instructions accepted as language-type data and temporarily stores them as instruction content. As the interface unit 101, the processing unit 10 adds text to instructions entered as text so as to predict the time required to reach a target position on a nautical chart based on the instructions. As part of the formatting process, if the accepted instruction includes a statement indicating a target position, the processing unit 10 may execute a process of converting the target position into longitude and latitude information.

[0066] The processing unit 10 functions as a current status acquisition unit 102 that acquires sea state data at the current position of the vessel S from the sensor 3. The processing unit 10 acquires the current position of the vessel S using the GPS receiver 305. The processing unit 10 acquires the vessel speed of the vessel S using the vessel speedometer 304. As the current status acquisition unit 102, the processing unit 10 acquires the rudder angle of the steering gear, the propulsion force of the propulsion force generating device, and the output from the power source from the sensor 3.

[0067] The processing unit 10, as the current status acquisition unit 102, acquires image data of the traveling direction of the ship S captured by a camera 301, which is one of the sensors 3. The camera 301 is installed on the ship S so as to capture the area ahead in the traveling direction of the ship S in its field of view, or multiple cameras may be installed on the front, rear, left and right sides, or the camera may be installed so as to capture a 360° field of view.

[0068] The processing unit 10, as the current status acquisition unit 102, acquires data indicating the surrounding conditions of the vessel S, which is obtained from each of the radar 302 and sonar 303, which are one of the sensors 3. Using the data obtained from the radar 302, the processing unit 10 can detect the presence of other ships, buoys, and birds in the sea around the vessel S. The processing unit 10 may acquire, from the radar 302, image data depicting the distribution of the detection results. Using the data obtained from the sonar 303, the processing unit 10 can detect objects, including fish, present in the sea around the vessel S. The processing unit 10 may acquire the object detection results as image data. The processing unit 10, as the current status acquisition unit 102, may acquire, via the communication unit 12, from the server device 4 or an external server, time information and a state prediction result at the current time based on data from an observation satellite.

[0069] The processing unit 10, as the current status acquisition unit 102, formats data including position data, ship speed, and image data acquired from the sensor 3 to conform to the specifications of data indicating the current status to be input into the learning model M1 described below.

[0070] The processing unit 10, as the current status acquisition unit 102, acquires sea state data obtained from the camera 301, radar 302, sonar 303, speedometer 304, GPS receiver 305, and AIS 306 of the sensor 3 based on a predetermined update period (for example, one second to several tens of seconds), and stores the data in chronological order in the storage unit 11. The processing unit 10 may also acquire data such as the direction in which the ship S is facing. The processing unit 10 may acquire time information when storing the data and associate it with each piece of data.

[0071] The processing unit 10 uses the learning model M1 as the command output unit 103 and outputs command data based on the instruction content and data indicating the current situation. The command data includes rudder angle, propulsive force, and / or output amount from the power source. The processing unit 10 provides ship maneuvering knowledge to the learning model M1 as prerequisite knowledge in the command output unit 103. The command data may include predicted information on the direction of travel as a basis for controlling the navigation of the ship S and may be output in natural language. The learning model M1 will be described in detail below.

[0072] The processing unit 10, as the command output unit 103, may output command data to any device of the sensor 3, or may output command content to the device. For example, the processing unit 10, as the command output unit 103, may output a sensing execution instruction to the sonar 303, which is also the sensor 3. The processing unit 10 may output command data to change the setting of the detection range of the sonar 303, which is also the sensor 3.

[0073] The processing unit 10 functions as a control output unit 104 that shapes the rudder angle, propulsive force, and / or output amount from the power source output from the command output unit 103 for output to the control device 2. As the control output unit 104, the processing unit 10 shapes the values ​​in accordance with the numerical range, units, etc. for the control device 2 installed on the ship S.

[0074] Each time the processing unit 10 receives an instruction from the interface unit 101, it inputs multiple pieces of time-series data spanning different periods of time or the most recent data from the sea state data acquired by the current state acquisition unit 102 together with the instruction content into the learning model M1 and executes processing.

[0075] FIG. 5 is a schematic diagram of the learning model M1. As described above, the learning model M1 employs a multimodal model and accepts both text and image inputs. The learning model M1 is trained to output text containing parameters included in the command data, for example. The learning model M1 uses, for example, LLaVA (Large Language-and Vision Assistant). The learning model M1 is not limited to LLaVA, and other architectures may be employed. In the example shown in FIG. 5, the learning model M1 includes a module that performs computational processing on the input text, such as tokenization and vector data embedding. The learning model M1 performs vector conversion on the input image, encodes (abstracts) the vector-converted data, and further performs processing such as identifying noteworthy data using a transformer or the like. The learning model M1 concatenates the processed data for the text and the processed data for the image data, and then inputs the data into a large-scale language model (LLM). The large-scale language model may be run on the server device 4 or an external server. Before and after inputting the linked data, the processing unit 10 provides the learning model M1 with information that serves as the premise for ship maneuvering knowledge, so that the command data that is output is related to the maneuvering of the ship S.

[0076] The learning model M1 may be one that has been trained using documents and images related to ship-maneuvering knowledge, without using a large-scale language model. The learning model M1 may be configured by dividing the learning model M1 into a rudder angle model that is trained to output a numerical value of the rudder angle from input images and instruction data, using the control history of past ship maneuvering as big data, and a propulsion force model that is trained to output a numerical value of the propulsion force from input images and instruction data.

[0077] The functions of the learning model M1 and the processing unit 10 are not limited to those shown in FIGS. 4 and 5 . The processing unit 10 may also function as a processing unit 107 that pre-processes image data and command data text to be input to the learning model M1. FIG. 6 is an explanatory diagram of another example of functions based on the information processing program P1. The processing unit 107 processes data obtained from the current status acquisition unit 102 and provides the processed data to the command output unit 103. The processing unit 107 may, for example, input image data into a segmentation model and identify objects appearing in the image. The processing unit 107 may, for example, use a depth distribution conversion model to convert the image data into a distribution of distances to objects appearing in the image data. The processing unit 107 may also detect danger or signs of danger from data other than image data obtained from the sensor 30 and input the detection results to the learning model M1. For example, the processing unit may input data obtained by predicting the next sea state from time-series data obtained from the sensor 30 into the learning model M1.

[0078] Use of the learning model M1 of the present disclosure contributes to the realization of safe navigation by being able to deal with unexpected patterns, compared to a process that recognizes voice instructions from an operator and selects from predefined navigation control patterns based on the recognized text. Furthermore, a configuration in which all patterns are stored in the memory unit 11 requires a large amount of storage capacity. Use of the learning model M1 reduces the amount of storage capacity used because command data is output within a range that allows for inference of unexpected patterns.

[0079] The processing unit 10, as the command output unit 103, executes a process of adding or partially modifying wording requesting that a command be issued including specific numerical values ​​for the steering gear, the propulsion generating device, and / or the power source to the language-type data of the text (or voice) of an instruction in natural language received by the interface unit 101. The processing unit 10, as the command output unit 103, may add information indicating in which direction each of the one or more image data received by the interface unit 101 corresponds to, to the language-type data.

[0080] The following describes a process for automatically outputting ship maneuvering command data using the learning model M1 that employs a multimodal model. Figure 7 is a flowchart showing an example of a command data output process performed by the information processing device 1. The processing unit 10 displays a reception screen for receiving instructions on the display unit 13, and executes the following process.

[0081] The processing unit 10 of the information processing device 1 receives an instruction via voice, text input, or image input at the interface unit 101 (step S101). The processing unit 10 outputs the content of the received instruction as text or an image on the display unit 13 (step S102). The processing unit 10 uses the function of the interface unit 101 to format the received instruction as language-type data (step S103).

[0082] The processing unit 10 acquires sea state data from the sensor 3 including a camera using the current state acquisition unit 102 (step S104). The sea state data acquired in step S104 includes image data of images acquired from the camera. The processing unit 10 may acquire sensing data obtained from the radar 302, sonar 303, etc. as sea state data. The sensing data obtained from the radar 302, sonar 303, etc. may be acquired as numerical values, or may be acquired as image data showing a three-dimensional or two-dimensional distribution.

[0083] The processing unit 10 further formats the received instruction language data so that it corresponds to the image data, and adds text instructing a prediction to the language data for the LLM included in the learning model M1 (step S105). In step S105, data obtained from the radar 302 and data obtained from the sonar 303 included in the acquired sea state data may be added to the text as numerical values.

[0084] The processing unit 10, as the command output unit 103, creates text of ship maneuvering knowledge related to the instruction received in step S101 as background knowledge (step S106). In step S106, the processing unit 10 extracts text of ship maneuvering knowledge related to the received instruction from the ship maneuvering knowledge stored in the storage unit 11. The processing unit 10 provides the created text to the learning model M1 (LLM) (step S107). In step S107, if the LLM is running on the server device 4 or an external server, the processing unit 10 provides the text to the LLM via the communication unit 12.

[0085] The processing unit 10, as the command output unit 103, inputs the image data included in the sea condition data acquired in step S104 and the language-type data corrected in step S105 to the learning model M1 (step S108). The processing unit 10 acquires command data output from the learning model M1 (step S109). The command data includes text. The command data may also include image data. The processing unit 10 stores the output (command data) from the learning model M1 acquired in step S109 (step S110).

[0086] The processing unit 10, functioning as the control output unit 104, extracts parameters to be provided to the control device 2 from the text of the acquired command data and formats the parameters for the control device 2 (step S111). The processing unit 10 outputs the formatted parameters to the control device 2 via the control output unit 104 (step S112).

[0087] The processing unit 10, as the command output unit 103, creates text, image, and / or audio notification data to be output on the display unit 13 (step S113). In step S113, the processing unit 10 may include explanatory text, explanatory images, etc. of the situation at the destination predicted from the current situation in the command output unit 103. The processing unit 10 outputs the created text, image, and / or audio from the display unit 13 and / or audio input / output unit 15 (step S114), and ends the process.

[0088] The processing procedure shown in Figure 7 makes it possible to automatically navigate the ship S appropriately based on instructions in natural language from the operator, taking into account predicted sea conditions. Note that step S112 may be omitted, and step S114 for command data may be processed, with the operator of the ship S confirming and approving the command and proposed control data before inputting the control data into the control device 2. Stages or modes may be provided in which the ship S is semi-automatically navigated by presenting only the control content while presenting the rationale to the operator, such as during a test period until learning by the learning model M1 progresses or until reliability is established. The mode may be selected by the operator.

[0089] The functions shown in FIG. 4 and the processing procedure shown in FIG. 7 will be described in more detail. The operator inputs instructions using the operation unit 14 or the voice input / output unit 15 of the information processing device 1. For example, the operator inputs voice, such as "Depart for point ***, ***" into the microphone included in the remote controller of the operation unit 14. FIG. 8 is an explanatory diagram showing an example of the display screen 131 of the display unit 13. The processing unit 10, as the interface unit 101, executes recognition processing on the input voice and displays the recognition result on the display unit 13 (S102). As shown in FIG. 8, the display screen 131 displays an image 132 being captured by the camera 301. The image 132 shows another ship in the area where the ship has left the port and is offshore. The display screen 131 in FIG. 8 also displays text of the instruction content accepted by the interface unit 101.

[0090] FIG. 9 is an explanatory diagram showing another example of the display screen 131 on the display unit 13. The example shown in FIG. 9 is displayed after the display screen 131 shown in FIG. 8, when command data is output from the learning model M1. The display screen 131 includes a command result area 133 containing text of parameters extracted by the command output unit 103 from the output from the learning model M1. As shown in FIG. 9, the command result area 133 displays the content of the command data in text, such as "Depart by turning to the right based on the rules for departure. Stay to the right as other ships are returning." In this way, the information processing device 1 can display a situation predicted from the current situation and determine command data determined from that prediction.

[0091] The text in the command result area 133 indicates the maneuvering knowledge adopted to control the ship S, "Navigate the route determined for each port," and further indicates command data, such as "Move to the right, turn to the right," determined based on the maneuvering knowledge. Using the display screen 131 shown in FIG. 9, the operator can recognize the command data and the basis for determining the command data (departure rules) based on voice or other input. This allows the operator to understand why the steering gear, the propulsion generating device, and / or the power source are operating in the manner that they are.

[0092] As described above, the navigation system 100 makes it possible to automatically navigate the ship S appropriately based on instructions in natural language from the operator, taking into account predicted sea conditions. By presenting the instructions and predicted sea conditions together with the automatically determined command content, it is also possible to appropriately clarify the causal relationship with respect to navigation control.

[0093] [Second embodiment] The information processing device 1 in the second embodiment further has the following functions to increase the reliability of the output of command data using the learning model M1 shown in the first embodiment. The configuration of the navigation system 100 in the second embodiment is similar to the configuration of the navigation system 100 in the first embodiment, except for some of the functions of the information processing device 1 described below. Therefore, the same reference numerals are used for common components, and detailed descriptions are omitted.

[0094] 10 is an explanatory diagram of functions based on the information processing program P1 in the second embodiment. Based on the information processing program P1 of the second embodiment, the processing unit 10 functions as an interface unit 101, a current status acquisition unit 102, a command output unit 103, and a control output unit 104, as well as a prediction accuracy determination unit 105 and a strategy control unit 106, as shown in FIG.

[0095] The processing unit 10 functions as a prediction accuracy determination unit 105 that determines the accuracy of the prediction information included in the output information from the learning model M1. The processing unit 10 inputs instruction data to the learning model M1, using sea state data acquired by the current state acquisition unit 102 at the time of receiving an instruction, to output a situation prediction until reaching the target position. The processing unit 10 quantifies the accuracy based on the magnitude of the difference between the situation predicted by the learning model M1 at any time point until reaching the target position and the situation indicated by the sea state data acquired by the current state acquisition unit 102 after receiving the instruction. The processing unit 10 may perform this process each time a time point corresponding to the prediction data arrives between receiving the instruction and reaching the target position. The processing unit 10, as the prediction accuracy determination unit 105, feeds back to the learning model M1 the calculated accuracy value, assuming that the greater the difference, the lower the prediction accuracy. The prediction accuracy is indicated, for example, by a numerical value or a symbol. If the prediction accuracy is determined to be poor because it is less than a predetermined value (or the magnitude of the difference is equal to or greater than a predetermined value), the processing unit 10 may clearly instruct the learning model M1 to make a new prediction and reacquire the command data (see FIG. 12). In this case, the processing unit 10 may notify the user of the need to make a new prediction and re-output the command data via the display unit 13 and / or the audio input / output unit 15.

[0096] The processing unit 10 functions as a strategy control unit 106 that creates a medium- to long-term control strategy for stably maneuvering the ship S. The processing unit 10 creates the control strategy by referring to ship maneuvering knowledge based on the instruction content received by the interface unit 101 and the sea state data acquired by the current state acquisition unit 102. The processing unit 10 may store and refer to control strategy templates corresponding to the instruction content and sea state data as ship maneuvering knowledge. As the strategy control unit 106, the processing unit 10 determines whether the command data (parameters) output from the command output unit 103 match the control strategy. If it is determined that the command data do not match, the processing unit 10 can also apply the control strategy to the learning model M1 and re-output the command data to achieve more stable ship maneuvering. For example, as the strategy control unit 106, the processing unit 10 creates a medium- to long-term strategy that involves making a rightward evasive maneuver to the destination position and then returning to the route originally determined from the start position to the destination position. In this case, the processing unit 10 determines whether the command data output by the command output unit 103 using the learning model M1 contradicts the medium- to long-term strategy created by the strategy control unit 106 at each change point in the command data up to the target position, and if there is a contradiction, re-outputs the command data. The processing unit 10, as the strategy control unit 106, may create a medium- to long-term control strategy using a learning model that has been trained to output a control strategy when instruction content and sea state data are input.

[0097] The processing unit 10 may function as a strategy control unit 106 that creates a current control strategy for stably maneuvering the vessel S by referring to past control history. In this case, the processing unit 10 stores the rudder angle, propulsive force, and / or output amount from the power source output as a function of the control output unit 104 in the memory unit 11 as control history. As the strategy control unit 106, the processing unit 10 creates a specific strategy for stably maneuvering the vessel S based on changes in the rudder angle, propulsive force, and / or power source from the control history stored in the memory unit 11. For example, the processing unit 10 determines whether a change in the rudder angle output from the control output unit 104 is within a predetermined range, and if it is determined that the change is outside the predetermined range, creates a control strategy for controlling the learning model M1 so that the rudder angle does not change significantly. For example, the processing unit 10 determines whether the most recent change in the propulsive force output from the control output unit 104 is within a predetermined range, and if it is determined that the change is outside the predetermined range, creates a control strategy for controlling the learning model M1 so that the propulsive force does not change significantly. The processing unit 10 inputs the determined control strategy into the learning model M1 as a guideline. The control strategy may be written in natural language as text, or may be represented by numerical values ​​or symbols assigned to pre-set guidelines. If the control strategy contradicts the parameter itself or its change output from the control output unit 104, the command data may be output again and reacquired.

[0098] The function of the prediction accuracy determination unit 105 and / or the function of the strategy control unit 106 may be implemented in the server device 4.

[0099] The processing unit 10 may create a control measure as needed based on sea state data as the measure control unit 106, even while no instructions are being received by the interface unit 101, each time sea state data is acquired by the current state acquisition unit 102. The processing unit 10 may create a control measure when it is determined that the situation has changed, such as when a parameter included in the sea state data has changed significantly.

[0100] The learning model M1 in the second embodiment accepts inputs of text indicating instructions received by the interface unit 101 and sea state data acquired by the current state acquisition unit 102. In addition, as shown in FIG. 10 , the learning model M1 accepts inputs of accuracy output from the prediction accuracy determination unit 105 and control measures output from the measure control unit 106. The processing unit 10 inputs the instruction content, data indicating the current state, accuracy, and data indicating the control measures to the learning model M1 through the function of the command output unit 103. The processing unit 10 extracts parameters from the command data output from the learning model M1 as the control output unit 104 and formats the extracted parameters as control data.

[0101] In the second embodiment, the learning model M1 may be configured to be separated into a situation prediction model and a command output model. FIG. 11 is a schematic diagram of another aspect of the learning model M1. The learning model M1 can be separated into a situation prediction model M11 and a command output model M12. The situation prediction model M11 is trained to input sea state data obtained by the current state acquisition unit 102, predict the situation, and output prediction data. The situation prediction model M11 is a multimodal model that accepts inputs such as image data acquired from the camera 301, mapped images of object detection obtained from the radar 302, and numerical values ​​of the ship speed obtained from the ship speed indicator, and outputs image data and / or numerical values ​​of the prediction results. The prediction data is, for example, sea state data obtained by the sensor 3 after a predetermined time, such as several minutes, has elapsed. The sea state data obtained as prediction data is, for example, the scene captured by the camera 301, the surrounding conditions captured by the radar 302, and / or the detection results of underwater objects detected by the sonar 303. The command output model M12 receives the predicted data output from the situation prediction model M11 and instructions received from the operator, and outputs command data.

[0102] An example of the function of the processing unit 10 as the prediction accuracy determination unit 105 of the second embodiment will be described with reference to a flowchart. Fig. 12 is a flowchart showing an example of a processing procedure by the information processing device 1 in the second embodiment. As in the first embodiment, the processing unit 10 of the information processing device 1 of the second embodiment executes the processing procedure shown in Fig. 7, formats and outputs control data for the control device 2, and then executes the following processing.

[0103] In response to the instruction, the processing unit 10 reads out the stored command data acquired from the learning model M1 by the command output unit 103 and output (step S201). From the read command data, the processing unit 10 extracts prediction data based on the sea state data acquired at the time the instruction was received (step S202). The processing unit 10 identifies the actual sea state data at the time corresponding to the prediction data from the data acquired by the current state acquisition unit 102 (step S203). The time corresponding to the prediction data is a predetermined time after the instruction is received. The predetermined time is the time until the time for which the learning model M1 is instructed to make a prediction. For example, if the processing unit 10 instructs the learning model M1 to output prediction data for sea states 10 minutes from now, the predetermined time is 10 minutes. In step S203, if the time corresponding to the prediction data has not yet arrived, the processing unit 10 waits until the data can be acquired.

[0104] The processing unit 10 compares the prediction data extracted in step S202 with the sea state data acquired in step S203 (step S204). The processing unit 10 determines the prediction accuracy according to the magnitude of the difference (dissociation) between the prediction data and the actual sea state data (step S205). In step S205, the processing unit 10 determines the prediction accuracy to be poorer the greater the difference, and to be better the smaller the difference. The processing unit 10 may calculate the prediction accuracy as a numerical value or may determine it as a symbol.

[0105] The processing unit 10 determines the quality of the prediction accuracy based on whether the value indicating the prediction accuracy is equal to or greater than a predetermined value, within a predetermined range, or whether the symbol indicating the prediction accuracy is a specific symbol (step S206). If the prediction accuracy is determined to be good (S206: YES), it can be assumed that the command output using the prediction data from the learning model M1 is reliable. Therefore, the processing unit 10 continues processing as is. The processing unit 10 determines whether other prediction data is available (step S207). If it is determined that there is no prediction data (S207: NO), the processing unit 10 terminates the prediction accuracy determination process. In step S207, the processing unit 10 determines that there is prediction data if the prediction data based on the sea state data acquired at the time of receiving the instruction includes predictions for multiple time points in the future from the time the instruction was received. If it is determined that there is prediction data (S207: YES), the processing unit 10 returns to step S203. In this case, the processing unit 10 executes steps S204-S207 at the next predicted time point.

[0106] If it is determined in step S206 that the prediction accuracy is not good (S206: NO), the processing unit 10 re-inputs the formatted instruction text (language-type data) corresponding to the command data and the image data included in the measured sea state data acquired in step S203 into the learning model M1 (step S208). The processing unit 10 acquires new command data output from the learning model M1 (step S209). The processing unit 10 stores the output (command data) from the learning model M1 acquired in step S209 (step S210).

[0107] The processing unit 10, functioning as the control output unit 104, extracts parameters to be provided to the control device 2 from the text of the acquired command data and formats the parameters for the control device 2 (step S211). The processing unit 10 outputs the formatted parameters (control data) to the control device 2 via the control output unit 104 (step S212).

[0108] The processing unit 10, as the command output unit 103, creates text, image, and / or audio data to be output on the display unit 13 (step S213). In step S213, the processing unit 10 may include explanatory text, explanatory images, etc. of the situation at the destination predicted from the current situation in the command output unit 103. The processing unit 10 outputs the created text, image, and / or audio from the display unit 13 and / or audio input / output unit 15 (step S214), and the process proceeds to step S207. In step S214, the processing unit 10 may output a message indicating that the prediction accuracy was not good and that the command data was output again.

[0109] The processing unit 10 may provide the prediction accuracy determined in step S205 to the learning model M1, and cause it to re-learn regardless of the quality of the prediction accuracy. This is expected to separately improve the prediction accuracy of the learning model M1.

[0110] The processing procedure by the command output unit 103 of the processing unit 10 is the same as the processing procedure shown in FIG. 7 of the first embodiment, and therefore a description of the display screen 131 will also be omitted.

[0111] The navigation system 100 of the second embodiment makes it possible to automatically navigate the ship S appropriately based on natural language instructions from the operator, taking into account predicted sea conditions. By presenting the instructions and predicted sea conditions along with automatically determined command content, it is also possible to appropriately clarify the causal relationship with respect to navigation control. Furthermore, by improving the accuracy of the prediction of sea conditions that serves as the basis for commands, it is expected that the navigation of the ship S will be controlled based on more appropriate command content. It is also expected that control will be performed to obtain more appropriate command content based on the past navigation history of the ship S.

[0112] [Third embodiment] Figure 13 is an explanatory diagram of functions based on the information processing program P1 of the third embodiment. The processing unit 10 performs the various functions shown in Figure 12 based on the information processing program P1 and the learning model M1. The configuration of the navigation system 100 in the third embodiment is similar to the configuration of the navigation systems 100 in the first and second embodiments, except for some of the functions of the information processing device 1 described below. Therefore, common components are denoted by the same reference numerals and detailed description thereof will be omitted.

[0113] In the third embodiment, as shown in Fig. 13, the function of the strategy control unit 106 is different. The processing unit 10 of the information processing device 1 of the third embodiment receives, through the strategy control unit 106, input of command data output by the command output unit 103, rather than sea state data from the current state acquisition unit 102, and performs the function of determining specific control data to be given to the control device 2 based on the input command data.

[0114] Fig. 14 is a flowchart showing an example of a command data output procedure by the information processing device 1 of the third embodiment. Among the procedure shown in Fig. 14, steps common to the procedure of Fig. 7 of the first embodiment are assigned the same step numbers and detailed descriptions thereof will be omitted.

[0115] In the third embodiment, after storing the command data acquired in step S109 as history (S110), the processing unit 10 determines parameters to be provided to the control device 2 that match the content of the acquired command data based on past maneuvering records, using the function of the strategy control unit 106 (step S131). In step S131, the processing unit 10 makes a decision after, for example, confirming whether or not the parameters that can be extracted from the command data are within a data range for stably maneuvering the ship S. If the parameters are not within the data range, the processing unit 10 may determine parameters within the data range. The processing unit 10 may also extract maneuvering records for sea state data that match the predicted data of the command data from past records, and determine parameters that match the extracted records.

[0116] The processing unit 10 formats the determined parameters for the control device 2 (step S132) and outputs them to the control device 2 (S112).

[0117] The configuration of the navigation system 100 of the third embodiment makes it possible to automatically navigate the ship S appropriately based on natural language instructions from the operator, taking into account predicted sea conditions. By presenting the instructions and predicted sea conditions along with automatically determined command content, it is also possible to appropriately clarify the causal relationship with respect to navigation control. Furthermore, since control data that matches the stable past navigation performance of the ship S is determined and output, it is expected that control will be possible to obtain appropriate command content for each individual ship S.

[0118] [Fourth Embodiment] In the fourth embodiment, the processing unit 10 serves as the command output unit 103 and generates output data from command data obtained using the learning model M1 to be output to the display unit 13 and / or the audio input / output unit 15. The output data includes the numerical values ​​of the propulsive force and / or the output amount from the power source included in the command data, and an image or audio based on the prediction information obtained from the learning model M1.

[0119] The configuration of the navigation system 100 in the fourth embodiment is similar to the configuration of the navigation system 100 in the first embodiment, except for some of the functions of the information processing device 1 described below, and therefore the common components are given the same symbols and detailed explanations are omitted.

[0120] In the fourth embodiment, when creating data to be output as command content in step S114 shown in Fig. 7 as in the first embodiment, the processing unit 10 creates an image showing the state of the destination predicted from the current situation or an image showing the navigation route, and displays it on the display unit 13. For example, the processing unit 10 creates an image to display the route along which the ship will navigate under control based on the command data, superimposed on the image being captured by the camera 301.

[0121] Fig. 15 is an explanatory diagram showing an example of the display screen 131 of the display unit 13 according to the fourth embodiment. Similar to the display screen 131 shown in Fig. 9 according to the first embodiment, the display screen 131 shown in Fig. 15 monitors and displays an image 132 currently being captured by the camera 301. In the fourth embodiment, the processing unit 10 creates and displays a route image 134 showing the navigation route of the vessel S, superimposed on the image 132, as shown in Fig. 15. The output of the route image 134 is not essential, and may be only command data or numerical values ​​of the rudder angle, propulsive force, and / or output amount from the power source.

[0122] In this way, the information processing device 1 in the navigation system 100 determines the control data for automatic navigation using the learning model M1, and can also clarify the content of navigation control by visualizing and presenting the automatically determined command content.

[0123] In the navigation system 100, the ship S shown in FIG. 1 may be an unmanned ship operated by remote communication. FIG. 16 is a schematic diagram of another embodiment of the navigation system 100. In this embodiment of the navigation system 100, the information processing device 1 is installed on a mother ship MS, which is not the ship to be operated. The information processing device 1 may be located on land, like the server device 4. In the navigation system 100 shown in the schematic diagram of FIG. 2, the ship S, which is a drone ship, is the ship to be operated from the mother ship MS. In this case, the information processing device 1 receives instructions from an operator of the information processing device 1 and acquires oceanographic data obtained from the sensor 3 via remote communication via a communication device 5 installed on the ship S. The communication device 5 is a communication module that realizes remote communication, similar to the communication unit 12 of the information processing device 1. The communication device 5 may be a communication module for an AIS-dedicated frequency, a wireless communication module connecting to a carrier network, a wireless communication module for Wi-Fi, or a communication module that realizes communication with the information processing device 1 via satellite communication. The communication device 5 can transmit sea state data obtained from the sensors 3 connected to the control device 2 to the information processing device 1. The communication device 5 stores identification data of the ship S to be maneuvered, and associates the identification data of the ship S when transmitting and receiving data. The information processing device 1 can employ any of the configurations shown in the first to fourth embodiments described above, and provides command data obtained by inputting instructions from an operator and sea state data into a learning model M1 to the control device 2 by remote communication via the communication device 5 of the ship S to be maneuvered. The control device 2 controls the steering gear etc. based on the command data, and may also control any of the devices of the sensors 3 based on the command data.

[0124] The embodiments disclosed above are illustrative in all respects and are not limiting. The scope of the present invention is defined by the claims, and includes all modifications within the meaning and scope of the claims. Forms obtained by appropriately combining the technical means disclosed in each embodiment are also included in the technical scope of the present invention.

[0125] Furthermore, independent claims and dependent claims described in the claims can be combined with each other in any and all combinations, regardless of the reference format. Furthermore, although the claims use a format in which a claim references two or more other claims (multiple claim format), this is not limited to this format. Multiple claims that reference at least one other multiple claim (multi-multi claim format) may also be used.

[0126] The following additional notes are provided regarding the above-described embodiment.

[0127] (Supplementary Note 1) An information processing device comprising: an interface unit that inputs maneuvering instructions from an operator as language-type data; a current status acquisition unit that acquires sea condition data indicating the operating status of the surface vehicle to be operated; and a command output unit that, when the sea condition data and the language-type data are input, outputs command data that conforms to the instructions of the language-type data according to the situation indicated by the sea condition data, using a learning model.

[0128] (Supplementary Note 2) The information processing device according to Supplementary Note 1, further comprising a control output unit that outputs control data for operating and controlling the water vehicle based on the output of the learning model.

[0129] (Supplementary Note 3) The information processing device according to Supplementary Note 1 or 2, wherein the command output unit outputs notification data including at least one of instruction content corresponding to the maneuvering command, sea state data on which the operation control is based, and instruction content corresponding to the command data.

[0130] (Supplementary Note 4) The information processing device according to any one of Supplementary Notes 1 to 3, wherein the language-type data is speech or text in a natural language.

[0131] (Supplementary Note 5) The information processing device according to any one of Supplementary Notes 1 to 4, wherein the learning model includes a multimodal model and a natural language model.

[0132] (Supplementary Note 6) The information processing device according to any one of Supplementary Notes 1 to 5, wherein the learning model is provided with ship-maneuvering knowledge data of the water vehicle as premise data.

[0133] (Supplementary Note 7) The information processing device according to Supplementary Note 5, wherein part of the calculations in the learning model are executed by another device.

[0134] (Appendix 8) An information processing device according to any one of Appendices 1 to 7, further comprising a processing unit that processes the sea state data acquired by the current state acquisition unit using an algorithm different from that of the learning model, and the command output unit inputs the sea state data processed by the processing unit into the learning model.

[0135] (Supplementary Note 9) The information processing device according to any one of Supplementary Notes 1 to 8, wherein the ocean condition data includes data obtained from at least one of a camera, a radar, a sonar, a GPS receiver, a weather sensor, an automatic identification system, and a fish finder.

[0136] (Supplementary Note 10) The information processing device according to Supplementary Note 9, wherein the command output unit inputs data obtained from at least one of the radar, sonar, GPS receiver, weather sensor, automatic vessel identification system, and fish finder as sea condition data into the learning model as image data.

[0137] (Appendix 11) The information processing device described in any one of Appendices 1 to 10, wherein the command data includes at least one of the steering angle, propulsion force, output from a power source, and command content for equipment mounted on the water vehicle.

[0138] (Supplementary Note 12) The information processing device according to any one of Supplementary Notes 1 to 11, wherein the learning model outputs predicted data of sea state data related to the input maneuvering instruction together with the command data.

[0139] (Supplementary Note 13) An information processing device according to Supplementary Note 12, further comprising a prediction accuracy determination unit that determines the prediction accuracy of the prediction data output by the learning model by comparing it with sea state data sequentially acquired by the current state acquisition unit, and the learning model receives feedback of the prediction accuracy and outputs prediction data or command data.

[0140] (Supplementary Note 14) An information processing device according to any one of claims 1 to 13, further comprising a strategy control unit that creates a control strategy for stable navigation of the water vehicle, and the command output unit outputs command data in accordance with the control strategy by inputting the control strategy into the learning model.

[0141] (Supplementary Note 15) The information processing device according to any one of Supplementary Notes 1 to 14, wherein the water vehicle is an unmanned vessel.

[0142] (Appendix 16) An information processing system comprising: a control device that controls the navigation of a surface vehicle to be operated; and an information processing device that outputs control data for operating and controlling the surface vehicle to the control device, wherein the information processing device comprises: an interface unit that inputs maneuvering instructions from an operator as language-type data; a current status acquisition unit that acquires sea state data indicating the operating status of the surface vehicle via communication; a command output unit that outputs command data using a learning model that, when the sea state data and the language-type data are input, outputs command data that matches the instructions of the language-type data according to the situation indicated by the sea state data; a control output unit that outputs control data for operating and controlling the surface vehicle based on the output of the learning model; and a communication unit that provides the control data to the control device via communication.

[0143] (Appendix 17) An information processing method comprising: inputting maneuvering instructions from an operator as language data; acquiring sea condition data indicating the operating status of the surface vehicle to be operated; using a learning model that, when the sea condition data and the language data are input, outputs command data that conforms to the instructions of the language data according to the operating environment of the sea condition data; and outputting control data for operating and controlling the surface vehicle based on the output of the learning model.

[0144] (Appendix 18) A computer program that causes a computer to execute the following process: inputting maneuvering instructions from an operator as language data; acquiring sea condition data indicating the operating status of the surface vehicle to be operated; using a learning model that, when the sea condition data and the language data are input, outputs command data that conforms to the instructions of the language data according to the operating environment of the sea condition data; and outputting control data for operating and controlling the surface vehicle based on the output of the learning model. term

[0145] Not necessarily all objects or advantages may be achieved in accordance with any particular embodiment described herein. Thus, for example, one skilled in the art will appreciate that a particular embodiment may be configured to operate to achieve or optimize one or more advantages as taught herein without necessarily achieving other objects or advantages as taught or suggested herein.

[0146] All processes described herein may be embodied and fully automated by software code modules executed by a computing system including one or more computers or processors. The code modules may be stored on any type of non-transitory computer-readable medium or other computer storage device. Some or all of the methods may be embodied in dedicated computer hardware.

[0147] Many other variations beyond those described herein will be apparent from this disclosure. For example, depending on the embodiment, certain operations, events, or functions of any of the algorithms described herein may be performed in a different sequence, added, merged, or omitted entirely (e.g., not all described acts or events are necessary to execute an algorithm). Furthermore, in certain embodiments, operations or events may be performed in parallel rather than sequentially, e.g., via multithreading, interrupt processing, or multiple processors or processor cores, or on other parallel architectures. Furthermore, different tasks or processes may be performed by different machines and / or computing systems that may function together.

[0148] The various illustrative logical blocks and modules described in connection with the embodiments disclosed herein may be implemented or executed by a machine such as a processor. The processor may be a microprocessor, but alternatively, the processor may be a controller, microcontroller, or state machine, or a combination thereof. The processor may include electrical circuitry configured to process computer-executable instructions. In another embodiment, the processor includes an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable device that performs logical operations without processing computer-executable instructions. A processor may also be implemented as a combination of computing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Although described herein primarily with reference to digital technology, a processor may also include primarily analog elements. For example, some or all of the signal processing algorithms described herein may be implemented by analog circuitry or mixed analog and digital circuitry. The computing environment can include any type of computer system, including, but not limited to, a microprocessor, mainframe computer, digital signal processor, portable computing device, device controller, or computer system based on a computational engine within an appliance.

[0149] Unless otherwise specified, conditional language such as "can," "could," "would," or "potential" is understood within the context in which it is generally used to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language does not generally imply that features, elements, and / or steps are required in any manner in one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included in or performed in any particular embodiment.

[0150] Disjunctive language such as "at least one of X, Y, Z," unless specifically stated otherwise, is understood in its general context to indicate that an item, term, etc. can be either X, Y, Z, or any combination thereof (e.g., X, Y, Z). Thus, such disjunctive language does not generally imply that a particular embodiment requires at least one of X, at least one of Y, or at least one of Z, respectively, to be present.

[0151] Any process descriptions, elements, or blocks in the flow diagrams described herein and / or illustrated in the accompanying drawings should be understood as potentially representing modules, segments, or portions of code, comprising one or more executable instructions for implementing a particular logical function or element in the process. Alternative embodiments are included within the scope of the embodiments described herein, in which elements or functions may be performed out of order, substantially simultaneously, or in reverse order from that shown or described, depending on the functionality involved, as will be understood by those skilled in the art.

[0152] Unless otherwise expressly stated, numeral terms such as "one" should generally be construed to include one or more described items. Thus, phrases such as "one device configured to" are intended to include one or more listed devices. Such one or more listed devices may also be collectively configured to perform the recited reference. For example, "a processor configured to perform the following A, B, and C" may include a first processor configured to perform A and a second processor configured to perform B and C. Additionally, even if a specific number of enumerations of the introduced embodiments are explicitly recited, those skilled in the art should construe such enumerations to typically mean at least the recited number (e.g., the mere enumeration of "two enumerations" without other modifiers typically means at least two enumerations, or two or more enumerations).

[0153] In general, it will be appreciated by those skilled in the art that the terms used herein generally intend "non-limiting" terms (e.g., the term "including" should be interpreted as "including but not limited to at least," the term "having" should be interpreted as "having at least," the term "including" should be interpreted as "including, but not limited to," etc.).

[0154] For purposes of description, the term "horizontal" as used herein is defined as a plane parallel to the plane or surface of the floor of the area in which the described system is used or the plane in which the described method is performed, regardless of its orientation. The term "floor" can be interchanged with the terms "ground" or "water surface." The term "vertical / plumb" refers to a direction perpendicular / vertical to a defined horizontal line. Terms such as "upper," "lower," "below," "top," "side," "higher," "lower," "above," "over," "below," etc. are defined relative to the horizontal plane.

[0155] As used herein, the terms "attach," "connect," "mate," and other related terms, unless otherwise noted, should be interpreted to include detachable, movable, fixed, adjustable, and / or removable connections or couplings. Connections / couplings include direct connections and / or connections with intermediate structures between the two components described.

[0156] Unless otherwise expressly stated, as used herein, numbers preceded by terms such as "approximately," "about," and "substantially" are inclusive of the recited number and also refer to an amount close to the recited amount that performs the desired function or achieves the desired result. For example, "approximately," "about," and "substantially" refer to values ​​less than 10% of the recited numerical value, unless otherwise expressly stated. As used herein, features of the disclosed embodiments preceded by terms such as "approximately," "about," and "substantially" refer to features that have some variability that also perform the desired function or achieve the desired result for that feature.

[0157] Many variations and modifications may be made to the above-described embodiments, and these elements should be understood to be among other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure and are protected by the following claims.

[0158] 100 Navigation system 1 Information processing device 10 Processing unit 101 Interface unit 102 Current status acquisition unit 103 Command output unit 104 Control output unit 105 Prediction accuracy determination unit 106 Strategy control unit 11 Memory unit 12 Communication unit 13 Display unit 131 Display screen 14 Operation unit 15 Voice input / output unit P1 Information processing program M1 Learning model M11 Situation prediction model M12 Command output model 2 Control device 3 Device 301 Camera 302 Radar 303 Sonar 304 Ship speed indicator 305 GPS receiver 4 Server device 42 Communication unit 5 Communication device

Claims

1. An information processing device comprising: an interface unit that inputs maneuvering instructions from an operator as language-type data; a current status acquisition unit that acquires sea condition data indicating the operating status of the surface vehicle to be operated; and a command output unit that, when the sea condition data and the language-type data are input, uses a learning model that outputs command data that matches the instructions of the language-type data according to the situation indicated by the sea condition data, and outputs command data.

2. The information processing device according to claim 1, further comprising a control output unit that outputs control data for operating and controlling the water vehicle based on the output of the learning model.

3. The information processing device according to claim 1, wherein the command output unit outputs notification data including at least one of instruction content corresponding to the maneuvering command, sea state data on which the operation control is based, and instruction content corresponding to the command data.

4. The information processing device according to claim 1, wherein the language-type data is speech or text in a natural language.

5. The information processing device according to any one of claims 1 to 4, wherein the learning model includes a multimodal model and a natural language model.

6. The information processing device according to claim 5, wherein the learning model is provided with navigation knowledge data of the water vehicle as premise data.

7. The information processing device according to claim 5, wherein part of the calculations in the learning model are executed by another device.

8. An information processing device as described in claim 1, further comprising a processing unit that processes the sea state data acquired by the current state acquisition unit using an algorithm different from that of the learning model, and the command output unit inputs the sea state data processed by the processing unit into the learning model.

9. The information processing device according to claim 1, wherein the ocean condition data includes data obtained from at least one of a camera, radar, sonar, GPS receiver, weather sensor, automatic identification system, and fish finder.

10. The information processing device according to claim 9, wherein the command output unit inputs data obtained as sea condition data from at least one of the radar, sonar, GPS receiver, weather sensor, automatic vessel identification system, and fish finder into the learning model as image data.

11. The information processing device according to claim 1, wherein the command data includes at least one of the steering angle, propulsion force, output from the power source of the surface vehicle, and command content for equipment mounted on the surface vehicle.

12. The information processing device according to claim 1, wherein the learning model outputs predicted data of sea state data related to the input maneuvering instructions together with the command data.

13. An information processing device as described in claim 12, further comprising a prediction accuracy determination unit that determines the prediction accuracy of the prediction data output by the learning model by comparing it with sea condition data sequentially acquired by the current condition acquisition unit, and the learning model receives feedback of the prediction accuracy and outputs prediction data or command data.

14. An information processing device as described in claim 1, comprising a strategy control unit that creates a control strategy for stable navigation of the water vehicle, and the command output unit outputs command data in accordance with the control strategy by inputting the control strategy into the learning model.

15. The information processing device according to claim 1, wherein the water vehicle is an unmanned vessel.

16. An information processing system comprising: a control device that controls the operation of a surface vehicle to be operated; and an information processing device that outputs control data for operating and controlling the surface vehicle to the control device, wherein the information processing device comprises: an interface unit that inputs maneuvering instructions from an operator as language-type data; a current status acquisition unit that acquires sea state data indicating the operating status of the surface vehicle via communication; a command output unit that outputs command data using a learning model that, when the sea state data and the language-type data are input, outputs command data that matches the instructions of the language-type data according to the situation indicated by the sea state data; a control output unit that outputs control data for operating and controlling the surface vehicle based on the output of the learning model; and a communication unit that provides the control data to the control device via communication.

17. An information processing method that inputs maneuvering instructions from an operator as language data, acquires sea state data indicating the operating status of the surface vehicle to be operated, uses a learning model that, when the sea state data and language data are input, outputs command data that conforms to the instructions of the language data according to the operating environment of the sea state data, and outputs control data for operating and controlling the surface vehicle based on the output of the learning model.

18. A computer program that causes a computer to execute the following process: inputting maneuvering instructions from an operator as language data; acquiring sea condition data indicating the operating status of the surface vehicle to be operated; using a learning model that, when the sea condition data and language data are input, outputs command data that conforms to the instructions of the language data according to the operating environment of the sea condition data; and outputting control data for operating and controlling the surface vehicle based on the output of the learning model.

Images