Information processing apparatus, information processing method, information processing system, and computer program

Abstract

[Problem] To provide an information processing device, an information processing method, and a computer program that contribute to proper automatic navigation of a waterborne mobile body. [Means for Solution] The information processing device includes: an interface section that inputs a ship operation instruction from an operator as language-type data; a current situation acquisition section that acquires sea state data that shows a running condition of a waterborne mobile body that is an operation target; an instruction output section that outputs instruction data using a learning model that, when the sea state data and the language-type data are input, outputs instruction data that is appropriate for an instruction of the language-type data corresponding to a condition shown by the sea state data; and a control output section that, based on an output of the learning model, outputs control data that performs operation control of the waterborne mobile body.

Description

Technical Field

[0001] This invention relates to an information processing device, information processing method, information processing system, and computer program for navigation of waterborne mobile bodies. Background Technology

[0002] A method for automatic navigation of waterborne mobile bodies such as ships or unmanned vessels is proposed. Automatic navigation is achieved by using location data, navigation speed and bearing data obtained from a Global Positioning System (GPS) receiver mounted on the waterborne mobile body to determine the rudder angle and speed for navigation to a pre-set position on the nautical chart.

[0003] Patent Document 1 discloses a method for achieving automatic navigation by recording ship handling operations such as engine speed and steering angle from departure to sailing on the sea surface, returning to port, and docking, and then reproducing the recorded ship handling operations. Patent Document 2 proposes a ship handling control method that enables cooperative control with vessels other than its own.

[0004] [Existing Technical Documents]

[0005] [Patent Literature]

[0006] Patent Document 1: Japanese Patent No. 5566426

[0007] Patent Document 2: Japanese Patent No. 5972201 Summary of the Invention

[0008] [The problem the invention aims to solve]

[0009] At sea, when mobile vessels are navigating, sea conditions such as wind direction or wave height, or the navigation status of other mobile vessels on the same day, may change due to date and time. In the presence of other unexpected mobile vessels or changes in weather, mobile vessels using control methods that reproduce pre-recorded ship handling operations may find it difficult to reach their target location.

[0010] The purpose of this disclosure is to provide an information processing apparatus, information processing method, information processing system, and computer program that contribute to the realization of appropriate automatic navigation for waterborne mobile bodies.

[0011] [Technical means to solve the problem]

[0012] One aspect of the information processing apparatus disclosed herein includes: an interface unit that inputs ship handling instructions from an operator as linguistic data; a status acquisition unit that acquires sea state data showing the operating status of a mobile watercraft being operated on; and an instruction output unit that outputs instruction data using a learning model, wherein the learning model outputs instruction data suitable for the instructions of the linguistic data corresponding to the status shown by the sea state data when the sea state data and the linguistic data are input.

[0013] In one aspect, regarding the operation of mobile watercraft, a learning model is used that outputs instruction data by inputting verbal instructions from the operator and sea state data. Thus, appropriate instruction data can be obtained by considering predictions of sea conditions in response to the operator's instructions.

[0014] One aspect of the information processing apparatus disclosed herein includes a control output unit that outputs control data for operating the waterborne mobile body based on the output of the learning model.

[0015] In one aspect, even the control data of ship handling equipment such as steering gear, propulsion generating device, and power source for the moving body on the water are also output, so that automatic navigation can be carried out through natural language instructions.

[0016] In one aspect of the information processing apparatus of this disclosure, the instruction output unit outputs notification data, the notification data including at least one of the instruction content corresponding to the ship handling instruction, sea state data as the basis for the operation control, and instruction content corresponding to the instruction data.

[0017] In one aspect, the instructions, the basis for the operation and control of the waterborne mobile body, and the instructions themselves are output, which the operator can then confirm. Thus, the operator can identify the causal relationship of the navigation control instructions given to them.

[0018] In one aspect of the information processing apparatus of this disclosure, the linguistic data is speech or text of natural language.

[0019] In one aspect, speech or text in natural language from the operator is received as linguistic data. Therefore, even if the operator gives abstract instructions in natural language, instruction data can be automatically output accordingly.

[0020] In one aspect of the information processing apparatus disclosed herein, the learning model includes a multimodal model and a natural language model.

[0021] In one aspect, the learning model receives both image data and text, and is able to output appropriate instruction data in natural language based on the phenomena shown in the image data and the text content related to the ship handling instructions learned through the language model.

[0022] In one aspect of the information processing apparatus of this disclosure, the knowledge data on the operation of the waterborne mobile body is provided as prerequisite data to the learning model.

[0023] In one aspect, by providing ship handling knowledge data as prerequisite data to the language model, it becomes easier to output instruction data that conforms to ship handling knowledge from the learning model.

[0024] One aspect of the information processing apparatus of this disclosure enables other apparatuses to perform a portion of the computations in the learning model.

[0025] In one aspect, to use language models to output natural language as instruction data in response to instructions, large-scale learning models are required. Compared to processing solely within the information processing devices mounted on the watercraft, processing can be made more efficient by utilizing external server devices or services. Where the watercraft has difficulty communicating wirelessly with these server devices, a smaller language model learned only from natural language related to boat handling knowledge can be used to output instruction data.

[0026] One aspect of the information processing apparatus of this disclosure includes a processing unit that processes sea state data acquired by the current situation acquisition unit using an algorithm different from that of the learning model, and an instruction output unit that inputs the processed sea state data into the learning model.

[0027] In one aspect, before providing sea state data to the learning model, other learning models of a different kind or algorithms outside of the learning model can be used to provide sea state data, which includes the identification of objects captured in the image, depth distribution, or conditions inferred from sensor inputs.

[0028] In one aspect of the information processing apparatus of this disclosure, the sea state data includes data obtained from at least one of a camera, radar, sonar, GPS receiver, weather sensor, automatic identification device for ships, and fish detector.

[0029] On one hand, it outputs command data corresponding to the conditions shown by sea state data obtained from cameras, radar, sonar, GPS receivers, weather sensors, automatic identification devices for ships, and fish detectors.

[0030] In one aspect of the information processing apparatus of this disclosure, the command output unit inputs data obtained as sea state data from at least one of the radar, sonar, GPS receiver, weather sensor, automatic identification device for ships, and fish detector as image data into the learning model.

[0031] In one approach, the distribution of detection results obtained from radar or sonar, GPS receivers, weather sensors, automatic identification devices for ships, and fish detectors is input as image data into a learning model. By using a multimodal model that can translate the content of image data into language, the goal is to extract information from the distributed images with high recognition accuracy and reflect it in the instruction data.

[0032] In one aspect of the information processing apparatus of this disclosure, the instruction data includes at least one of the following: the rudder angle of the waterborne mobile body, the thrust, the output from the power source, and the instruction content for the equipment mounted on the waterborne mobile body.

[0033] In one aspect, the instructions, which are linguistic data, can control any one of the following: the rudder angle of the waterborne mobile body, the propulsion force, the output from the power source, and the instructions for the equipment mounted on the waterborne mobile body.

[0034] In one aspect of the information processing apparatus of this disclosure, the learning model outputs predicted data of the sea state data related to the input ship handling instructions together with the instruction data.

[0035] In one aspect, the learning model outputs predicted data based on the input sea state data. By using the predicted data as the basis for instruction data, the operator can identify the basis for the output instructions.

[0036] One aspect of the information processing apparatus of this disclosure includes a prediction accuracy determination unit, which determines the prediction accuracy of prediction data output by the learning model by comparing it with sea state data acquired successively by the current situation acquisition unit. The learning model receives feedback on the prediction accuracy and outputs prediction data or instruction data.

[0037] In one aspect, the predicted data based on the input sea state data is successively verified for accuracy. Therefore, it is hoped that the accuracy of the predicted conditions, which serve as the basis for the output of command data, can be improved.

[0038] One aspect of the information processing apparatus disclosed herein includes a policy control unit that generates a control policy for the stable navigation of the waterborne mobile body, and an instruction output unit that outputs instruction data following the control policy by inputting the control policy into the learning model.

[0039] On one hand, feedback is provided on whether the ship handling control output from the learning model conforms to the control strategy for safe navigation, and it is expected that the learning model will output instruction data in accordance with safe navigation.

[0040] In one aspect of the information processing apparatus disclosed herein, the waterborne mobile body is an unmanned operating vessel.

[0041] In one aspect, the automatic navigation of unmanned vessels can be achieved by operators using voice-based data remote communication to give instructions.

[0042] One aspect of the information processing system disclosed herein includes: a control device for controlling the movement of a waterborne mobile body as the object of operation; and an information processing device for outputting control data for the operation control of the waterborne mobile body to the control device, the information processing device including: an interface unit for inputting ship handling instructions from an operator as linguistic data; a status acquisition unit for acquiring sea state data showing the operating status of the waterborne mobile body via communication; an instruction output unit for outputting instruction data using a learning model, wherein the learning model outputs instruction data appropriate to the linguistic data indicating the status shown by the sea state data when the sea state data and the linguistic data are input; a control output unit for outputting control data for operating the waterborne mobile body based on the output of the learning model; and a communication unit for providing the control data to the control device via communication.

[0043] In one aspect, the information processing device that outputs control data using a learning model learned by inputting verbal instructions from the operator and sea state data can be a device mounted on land or another vessel relative to a waterborne mobile body. Thus, appropriate instruction data taking into account sea state predictions can be provided to the operator remotely.

[0044] One aspect of the information processing method disclosed herein is to input ship handling instructions from the operator as linguistic data, acquire sea state data showing the operating status of the waterborne mobile body as the object of operation, use a learning model that outputs instruction data suitable for the instructions of the linguistic data corresponding to the operating environment of the sea state data when the sea state data and the linguistic data are input, and output control data for operating and controlling the waterborne mobile body based on the output of the learning model.

[0045] One aspect of the computer program disclosed herein causes a computer to perform the following processing: inputting ship handling instructions from an operator as linguistic data, acquiring sea state data indicating the operating status of the waterborne mobile body being operated on, using a learning model that, when the sea state data and the linguistic data are input, outputs instruction data suitable for the instructions of the linguistic data corresponding to the operating environment of the sea state data, and outputting control data for operating and controlling the waterborne mobile body based on the output of the learning model. Attached Figure Description

[0046] [ Figure 1 [This is a schematic diagram of the navigation system disclosed herein.]

[0047] [ Figure 2 [Illustration 1] is a block diagram showing the structure of an information processing device.

[0048] [ Figure 3 [ ] is a block diagram showing the structure of the server device.

[0049] [ Figure 4 [ ] is an explanatory diagram based on the functions of an information processing program.

[0050] [ Figure 5 [ ] is a summary diagram of the learning model.

[0051] [ Figure 6 [This is an illustration of another example of the functionality of an information processing program.]

[0052] [ Figure 7 [ ] is a flowchart illustrating an example of the output processing steps of instruction data performed by an information processing device.

[0053] [ Figure 8 [This is an explanatory diagram showing an example of a display screen in the display unit.]

[0054] [ Figure 9 [This is an explanatory diagram showing another example of a display screen in the display section.]

[0055] [ Figure 10 [ ] is an explanatory diagram of the functions of the information processing program based on the second embodiment.

[0056] [ Figure 11 [ ] is a summary diagram of other forms of learning models.

[0057] [ Figure 12 [This is a flowchart illustrating an example of the processing steps performed by the information processing apparatus in the second embodiment.]

[0058] [ Figure 13 [ ] is an explanatory diagram of the functions of an information processing program based on the third implementation.

[0059] [ Figure 14 [This is a flowchart illustrating an example of the instruction data output processing steps performed by an information processing apparatus of a third embodiment.]

[0060] [ Figure 15 [This is an explanatory diagram showing an example of a display screen in the display unit of the fourth embodiment.]

[0061] [ Figure 16 [This is a schematic diagram of another form of navigation system.] Detailed Implementation

[0062] This disclosure will be specifically described with reference to the accompanying drawings illustrating its embodiments. In the following embodiments, the application of the information processing apparatus of this disclosure to an automatic control navigation system of a ship will be described.

[0063] The vessels in this disclosure include yachts, fishing boats, merchant ships, etc. A vessel is an example of a waterborne unmanned aerial vehicle (UAV) that navigates on water, a UAV that travels in water, or a flying body that moves within a specified height above the water surface, and may be replaced by any of these.

[0064] [First Implementation Form]

[0065] Figure 1 This is a schematic diagram of the navigation system 100 disclosed herein. The navigation system 100 is a system for realizing the automatic navigation of a vessel S. The vessel S can be a surface-to-water drone, a small vessel such as a yacht or fishing boat, or a large vessel such as a merchant ship. The navigation system 100 of this disclosure receives instructions based on natural language from an operator, and determines command data based on the received instructions and the sea conditions after the predicted time point, which can be obtained from sea state data obtained from inside and outside the vessel S, thereby realizing the automatic navigation of the vessel S.

[0066] In the navigation system 100, a learning model M1 is created and used. The learning model M1 learns by outputting command data of the vessel S in the presence of input natural language-based instructions and sea state data. The learning model M1 uses a large multimodal model (LMM) and learns by taking as input text or voice indicating instructions from the operator, numerical or graphical representations of sea state, and outputting command data.

[0067] The navigation system 100 includes: an information processing unit 1 that outputs control data for operating and controlling the vessel S; a control unit 2 that controls the navigation of the vessel S; and a sensor 3 that acquires navigation data for control purposes. The information processing unit 1 is a computer capable of communication and is connected to the control unit 2 and the sensor 3.

[0068] Control device 2 outputs control signals to the steering gear, propulsion generating device, and / or power source mounted on the vessel S. Control device 2 generates control signals corresponding to the command data provided by information processing device 1 and outputs them to the steering gear, propulsion generating device, and / or power source. Control device 2 outputs a signal corresponding to the steering angle to the steering gear. The propulsion generating device is, for example, a propeller or thruster. Control device 2 outputs a signal indicating whether the propulsion generating device is on or off, and a signal corresponding to the rotational speed when on. The power source is an engine or a secondary battery. Control device 2 outputs a signal indicating the output quantity to the power source.

[0069] Sensor 3 includes a camera 301 for capturing images at sea, a radar 302 mounted on the ship S, a sonar 303, a speedometer 304, a GPS receiver 305, etc. Sensor 3 also includes an Automatic Identification System (AIS) 306. Sensor 3 includes sensors that monitor the rudder angle in the steering gear, the propulsion force in the propulsion generating device, and the output from the power source. Sensor 3 may also include a fish detector and a weather sensor.

[0070] In the navigation system 100, the information processing device 1 receives instructions from the operator in natural language and inputs the instructions, along with images or numerical values ​​of the sea conditions obtained by the sensor 3, into the learning model M1 to obtain command data. The navigation system 100 automatically exports the command data and outputs it to the control device 2, thereby enabling the vessel S to navigate automatically.

[0071] In the navigation system 100 of this disclosure, the information processing device 1 performs the process of outputting instruction data obtained through the learning model M1 to the display unit 13 for operator confirmation. At this time, in response to received instructions, the navigation system 100 outputs text or images showing the predicted sea conditions and instruction data, along with images obtained based on the sea condition data, to the display unit 13 of the information processing device 1. Thus, not only can instruction data be automatically generated and navigation automatically performed, but the operator can also confirm the sea condition prediction on which the instruction data was derived, thereby understanding the causal relationship of the instruction data.

[0072] like Figure 1As shown, the navigation system 100 can communicate with the server device 4 or an external server via a network N, including communication media such as satellite communication and carrier networks. The server device 4 or the external server provides weather forecast services, or services based on information obtained from observation satellites, or information based on sea state prediction services. An external server refers to a group of servers that provide these services respectively. The server device 4 is a device that aggregates sea state data used in the navigation system 100 and provides information. The server device 4 can also be a device capable of collecting radar or sonar information provided by multiple vessels S using the navigation system 100 and providing it to other vessels S.

[0073] Figure 2 This is a block diagram showing the structure of the information processing device 1. The information processing device 1 uses a personal computer. The information processing device 1 includes a processing unit 10, a storage unit 11, a communication unit 12, a display unit 13, an operation unit 14, and a voice input / output unit 15.

[0074] The processing unit 10 includes one or more arithmetic processing devices such as a central processing unit (CPU), a microprocessor (MPU), and a graphics processing unit (GPU). The processing unit 10 includes temporary storage media such as static random access memory (SRAM) and dynamic random access memory (DRAM). The processing unit 10 reads the information processing program P1 stored in the storage unit 11 into the temporary storage medium and executes it, thereby enabling the general-purpose computer to perform various processes described later and function as the information processing device 1 of the navigation system 100 of this disclosure.

[0075] Storage unit 11 is a large-capacity non-volatile storage area such as a solid-state drive (SSD) or hard disk. Storage unit 11 stores the programs (program products) required for processing by processing unit 10 and reference setting data. The setting data may include identification information of the vessel S equipped with information processing device 1, or its type, model, and identification information of the operator (ship handler) of vessel S. The program product includes information processing program P1, or images and their design (configuration) for displaying on display unit 13. Storage unit 11 stores learning model M1. Part or all of learning model M1 may be stored in server device 4 or an external server. Storage unit 11 stores general knowledge about ship control (ship handling knowledge) and information about the operator operating vessel S. Storage unit 11 stores sea state data obtained from various sensors 3 (sea state data). Storage unit 11 may also store nautical chart data.

[0076] The information processing program (computer product) P1 and the learning model M1 stored in the storage unit 11 can be either an information processing program P9 or a learning model M9 that the processing unit 10 reads from the non-temporary storage medium 9 that can be read by a computer and stores it in the storage unit 11. Alternatively, the information processing program P1 and the learning model M1 can be either a program or a model that the processing unit 10 downloads from the server device 4 or other download server via the communication unit 12 and stores in the storage unit 11.

[0077] The communication unit 12 communicates with the server device 4 or an external server via network N. The communication unit 12 can use a communication module that communicates with land-based devices or other vessels via a dedicated AIS frequency. The processing unit 10 can use the communication unit 12 to notify the maritime traffic center on land via the land-based device, and simultaneously obtain navigation information such as time, speed, and navigation status from the server device. The processing unit 10 can also use the communication unit 12 to send command data to the control device 2 of the vessel S, which is the object of ship handling. The communication unit 12 can be a wireless communication module connected to an operator's network, a wireless communication module for Wireless Fidelity (WiFi), or a communication module that communicates with the server device 4 or an external server via satellite communication.

[0078] Display unit 13 is a display such as a liquid crystal display (LCD) or an organic electroluminescent (EL) display. Display unit 13 may be, for example, a display with a built-in touchscreen. Processing unit 10 can output instruction data to display unit 13 for operator confirmation.

[0079] The operation unit 14 is a user interface capable of input / output with the processing unit 10. It may be a touch screen built into the display unit 13, or it may be a structure including physical buttons, switches, and physical knobs. A part of the operation unit 14 may be a remote controller with a button that has the function of determining direction, and has the function of determining the direction that the remote controller itself is pointing at the time the button is pressed.

[0080] The voice input / output unit 15 includes a speaker and a microphone. The processing unit 10 or the voice input / output unit 15 itself has a voice recognition function that converts speech collected using the microphone into text data. The microphone can be equipped in the aforementioned remote controller. The voice input / output unit 15 can also output the speech data collected using the microphone to the storage unit 11. The processing unit 10 can output sound effects or speech corresponding to the processing using the speaker. Based on the information processing program P1, the processing unit 10 can output speech from the speaker of the voice input / output unit 15 explaining the instructions determined by the processing described later.

[0081] Figure 3 This is a block diagram illustrating the structure of server device 4. Server device 4 includes a processing unit 40, a storage unit 41, and a communication unit 42. Server device 4 may include a single server computer, or it may distribute processing across multiple server computers. Processing unit 40 includes one or more computing devices such as CPUs, MPUs, and GPUs. Processing unit 40 includes temporary storage media such as SRAM and DRAM.

[0082] Storage unit 41 is a large-capacity non-volatile storage area such as SSDs and hard drives. Storage unit 41 stores large-scale language models (LLMs).

[0083] The communication unit 42 enables communication with the information processing device 1 via network N. Specifically, the communication unit 42 is a network card. The communication unit 42 can be a wireless communication device connected to an operator's network, or it can be a wireless communication device for WiFi. The processing unit 40 can send and receive data with the information processing device 1 through the communication unit 42.

[0084] Server device 4 receives data from information processing device 1 and performs part of the processing by information processing device 1 to use learning model M1. Server device 4 is not necessary if a large-scale language model is included in learning model M1 in information processing device 1.

[0085] The method by which the information processing device 1 determines the command data in the navigation system 100 thus configured will be described. Figure 4 This is an explanatory diagram based on the functions of information processing program P1. Processing unit 10, based on information processing program P1 and learning model M1, performs... Figure 4 The various functions shown.

[0086] The processing unit 10 functions as an interface unit 101 that receives instructions from the operator as language data. As an interface unit 101, the processing unit 10 receives voice instructions from the operator as language data in the voice input / output unit 15. As an interface unit 101, the processing unit 10 can also receive instructions from the operator via text input in natural language in the operation unit 14. As an interface unit 101, the processing unit 10 can also receive instructions selected by the operator from options on the screen displayed on the display unit 13 in the operation unit 14. As an interface unit 101, the processing unit 10 shapes the instructions received as language data and temporarily stores them as instruction content. As an interface unit 101, the processing unit 10 adds text to instructions input as text to predict the overall time required until reaching the target position on the nautical chart based on the instructions. If the received instructions contain a statement indicating the target position, the processing unit 10 can also perform processing to convert the target position into latitude and longitude information as part of the shaping process.

[0087] The processing unit 10 functions as a current situation acquisition unit 102, which obtains sea state data of the current position of the vessel S from the sensor 3. The processing unit 10 acquires the current position of the vessel S via the GPS receiver 305. The processing unit 10 acquires the speed of the vessel S via the speedometer 304. As a current situation acquisition unit 102, the processing unit 10 acquires the rudder angle from the steering gear, the propulsion force from the propulsion generating device, and the output from the power source from the sensor 3.

[0088] The processing unit 10, which functions as the current situation acquisition unit 102, acquires image data relative to the direction of travel of the ship S, captured by the camera 301, which is one of the sensors 3. The camera 301 is positioned on the ship S to capture the front of the ship S in the direction of travel within the field of view, or multiple cameras can be set up in front, behind, left, and right, using cameras that capture a 360° field of view.

[0089] Processing unit 10, acting as a current situation acquisition unit 102, acquires data showing the surrounding conditions of ship S obtained from radar 302 and sonar 303, which are sensors 3. Based on the data obtained from radar 302, processing unit 10 can detect the presence of other ships, buoys, and birds in the sea surrounding ship S. Processing unit 10 can acquire image data depicting the distribution of detection results from radar 302. Based on the data obtained from sonar 303, processing unit 10 can detect objects containing fish in the sea surrounding ship S. Processing unit 10 can acquire the detection results of objects as image data. As a current situation acquisition unit 102, processing unit 10 can also acquire time information or current situation prediction results based on data from observation satellites from server device 4 or an external server via communication unit 12.

[0090] The processing unit 10, which is the current situation acquisition unit 102, reshapes the data acquired from the sensor 3, including position data, ship speed, and image data, according to the current situation data specifications input into the learning model M1 described later.

[0091] The processing unit 10, acting as the current situation acquisition unit 102, acquires sea state data from the camera 301, radar 302, sonar 303, ship speedometer 304, GPS receiver 305, and AIS 306 of the sensor 3 based on a predetermined update cycle (e.g., 1 second to tens of seconds), and stores it in the storage unit 11 in time sequence. The processing unit 10 can also acquire data such as the bearing of the ship S. During storage, the processing unit 10 can also acquire time information and establish a correspondence between the data.

[0092] Processing unit 10, acting as command output unit 103, uses learning model M1 to output command data based on the instruction content and the displayed status data. The command data includes rudder angle, propulsion force, and / or output from the power source. In command output unit 103, processing unit 10 provides ship handling knowledge as prerequisite knowledge to learning model M1. The command data may include predicted information on the direction of travel as a basis for controlling the ship's navigation S, and is output in natural language. Details regarding learning model M1 will be explained later.

[0093] The processing unit 10, acting as an instruction output unit 103, can output instruction data for any device of the sensor 3, or it can output instruction content for a device. For example, as an instruction output unit 103, the processing unit 10 can output a sensing execution instruction to the sonar 303, which is also a sensor 3. The processing unit 10 can also output instruction data that changes the detection range setting of the sonar 303, which is also a sensor 3.

[0094] The processing unit 10 functions as a control output unit 104, which shapes the rudder angle, propulsion force, and / or output from the power source output from the command output unit 103 into an output for the control device 2. The processing unit 10, as the control output unit 104, shapes the values, units, etc., in conjunction with the control device 2 mounted on the ship S.

[0095] Whenever the processing unit 10 receives an instruction from the interface unit 101, it inputs multiple data points spanning different time series or the most recent data from the sea state data acquired by the current situation acquisition unit 102 along with the instruction content into the learning model M1 and performs processing.

[0096] Figure 5This is a schematic diagram of the learning model M1. As mentioned above, the learning model M1 employs a multimodal model, receiving inputs of both text and images. For example, the learning model M1 learns to output text containing parameters included in the instruction data. The learning model M1 may use a Large Language-and-Vision Assistant (LLaVA). However, the learning model M1 is not limited to LLaVA and may employ other model architectures. Figure 5 In the example shown, the learning model M1 includes modules that perform operations such as tokenization and vectorization (embedding) on ​​the input text. The learning model M1 performs vector transformation on the input image, encodes (abstracts) the transformed data, and uses transformers to determine the data of interest. The learning model M1 concatenates the processed text data with the processed image data and inputs this concatenation into a large-scale language model (LLM). The large-scale language model can also run on server device 4 or an external server. Before and after inputting the concatenated data, the processing unit 10 provides the learning model M1 with information as prerequisites for ship handling knowledge, making the output command data relevant to the ship handling of the vessel S.

[0097] The learning model M1 may also not use a large-scale language model, but instead use a model learned from documents or images related to ship handling knowledge. The learning model M1 may also be configured as follows: a rudder angle model that learns by using past ship handling-related control experience as big data and outputting rudder angle values ​​based on input images and instruction data, or a propulsion model that learns by outputting propulsion force values ​​based on the same input images and instruction data.

[0098] The functions of the learning model M1 and the processing unit 10 are not limited to Figure 4 and Figure 5 The content shown. The processing unit 10 can also perform the function of the processing unit 107, which preprocesses the text of the image data and instruction data input to the learning model M1. Figure 6This is an explanatory diagram illustrating another example of the function of the information processing program P1. The processing unit 107 processes the data obtained from the current situation acquisition unit 102 and provides it to the instruction output unit 103. For example, the processing unit 107 may input image data into a segmentation model and identify objects captured in the image. The processing unit 107 may also use a model that converts to a depth distribution and converts it to a distance distribution of objects projected into the image data. The processing unit 107 may also perform hazard or hazard sign detection based on data that is not image data obtained from the sensor 30 and input the detection results into the learning model M1. For example, the processing unit may also input data obtained from predicting the next sea state based on time-series data obtained from the sensor 30 into the learning model M1.

[0099] By using the learning model M1 disclosed herein, unexpected scenarios can be handled compared to processing voice recognition of operator instructions and selecting from pre-defined navigation control modes based on the recognized text, thus contributing to safer navigation. Furthermore, in a structure where all modes are stored in the storage unit 11, a large amount of storage capacity is used. By using the learning model M1, instruction data is output within an analogous range even for unexpected scenarios, thereby reducing storage capacity usage.

[0100] The processing unit 10, acting as the command output unit 103, performs the following processing: It adds or partially modifies a statement requesting the issuance of commands containing specific values ​​regarding the steering gear, propulsion generating device, and / or power source to the language data of the natural language-based instruction text (or voice) received by the interface unit 101. The processing unit 10, acting as the command output unit 103, may also add information to the language data indicating which direction the one or more image data received by the interface unit 101 is for.

[0101] Thus, the processing of the ship handling command data using the learning model M1 which employs a multimodal model is explained. Figure 7 This is a flowchart illustrating an example of the instruction data output processing steps performed by the information processing device 1. The processing unit 10 displays a receiving screen of the receiving instruction on the display unit 13 and performs the following processing.

[0102] The processing unit 10 of the information processing device 1 receives instructions via voice, text, or image input in the interface unit 101 (step S101). The processing unit 10 outputs the received instructions as text or images in the display unit 13 (step S102). The processing unit 10 uses the functions of the interface unit 101 to convert the received instructions into speech data (step S103).

[0103] The processing unit 10 acquires sea state data from the sensor 3, including the camera, via the current situation acquisition unit 102 (step S104). The sea state data acquired in step S104 includes image data from the image acquired by the camera. The processing unit 10 can acquire sea state data as sensor data obtained from the radar 302, sonar 303, etc. The sensor data can also be acquired as numerical values ​​from the sensor data obtained from the radar 302, sonar 303, etc., or as image data showing a three-dimensional or two-dimensional distribution.

[0104] The processing unit 10, through the interface unit 101, further shapes the received instructional linguistic data to correspond with the image data, and adds text indicating the prediction to the linguistic data for the purpose of LLM included in the learning model M1 (step S105). In step S105, data obtained from radar 302 and data obtained from sonar 303, which are included in the acquired sea state data, can be added as numerical values ​​to the text.

[0105] Processing unit 10, acting as instruction output unit 103, generates text as prerequisite knowledge regarding the ship handling knowledge received in step S101 (step S106). In step S106, processing unit 10 extracts the text regarding the ship handling knowledge received from the ship handling knowledge stored in storage unit 11. Processing unit 10 provides the generated text to learning model M1 (LLM) (step S107). In step S107, if the LLM is running on server device 4 or an external server, processing unit 10 provides the text to the LLM via communication unit 12.

[0106] Processing unit 10, acting as command output unit 103, inputs the image data contained in the sea state data acquired in step S104 and the language data corrected in step S105 to learning model M1 (step S108). Processing unit 10 acquires command data output from learning model M1 (step S109). Command data includes text. Command data may also include image data. Processing unit 10 stores the output (command data) from learning model M1 acquired in step S109 (step S110).

[0107] The processing unit 10, acting as the control output unit 104, extracts the parameters to be provided to the control device 2 from the text of the acquired instruction data and shapes them for use by the control device 2 (step S111). The processing unit 10 outputs the shaped parameters to the control device 2 through the control output unit 104 (step S112).

[0108] The processing unit 10, acting as the command output unit 103, generates notification data (text, images, and / or voice) for output to the display unit 13 (step S113). In step S113, the processing unit 10 preferably includes explanatory text, explanatory images, etc., describing the predicted status of the destination from the current situation in the command output unit 103. The processing unit 10 outputs the generated text, images, and / or voice from the display unit 13 and / or the voice input / output unit 15 (step S114) and ends the processing.

[0109] pass Figure 7 The processing steps shown enable the vessel S to navigate automatically and appropriately based on instructions from the operator using natural language, taking into account sea state predictions. Alternatively, step S112 can be omitted, and step S114 can be used to process the instruction data. After the operator of vessel S confirms and approves the instruction or control data suggestion, the control data is then input to the control device 2. Stages or modes can be prepared such as: during testing until learning based on the learning model M1 is complete or trust is gained, the system may remain in a semi-automatic navigation mode, only prompting the operator with control instructions. The mode can be selected by the operator.

[0110] right Figure 4 The functions shown and Figure 7 The processing steps shown will be explained in more detail. The operator uses the operation unit 14 or the voice input / output unit 15 of the information processing device 1 to input instructions. For example, the operator inputs a voice command such as "depart for location at latitude and longitude ***, ***" into the microphone of the remote controller included in the operation unit 14. Figure 8 This is an explanatory diagram showing an example of the display screen 131 in the display unit 13. The processing unit 10, acting as the interface unit 101, performs recognition processing on the input voice and displays the recognition result on the display unit 13 (S102). Figure 8 As shown, image 132, captured by camera 301, is monitored and displayed on screen 131. Image 132 shows other ships in the area extending from the port towards the sea. Additionally, in... Figure 8 The text of the instructions received from the interface section 101 is displayed on the display screen 131.

[0111] Figure 9 This is an explanatory diagram showing another example of the display screen 131 in the display unit 13. Figure 9 The example shown is in Figure 8 The display shown is shown after screen 131, at the point in time when instruction data is output from the learning model M1. Screen 131 includes an instruction result area 133, which contains the text of the parameters extracted by the instruction output unit 103 from the output from the learning model M1. In the instruction result area 133, as shown... Figure 9As shown, the text displays the instruction data: "Departure from port by turning to the right based on the rules at the time of departure; keep to the right as other ships are returning." Thus, the information processing device 1 can display the situation predicted from the current conditions and simultaneously determine the instruction data based on the prediction.

[0112] The text in instruction result area 133 shows the ship handling knowledge used in the control of vessel S: "navigating on the routes prescribed for each port," and further shows the instruction data "move to starboard, turn to starboard" determined based on this ship handling knowledge. (The text then repeats itself, so the translation will only include the first instance.) Figure 9 As shown on display screen 131, the operator can identify the basis (departure rules) for the command data based on voice input and other inputs. Thus, the operator can understand why the steering gear, propulsion generator, and / or power source operate in the way they do.

[0113] As described above, according to the navigation system 100, the vessel S can navigate automatically and appropriately based on instructions from the operator in natural language, taking into account sea state predictions. By presenting the instructions and predicted sea states along with the automatically determined commands, the causal relationship for navigation control can also be appropriately clarified.

[0114] [Second Implementation Form]

[0115] In order to improve the reliability of the output of instruction data using the learning model M1 shown in the first embodiment, the information processing device 1 in the second embodiment further has the following functions. Except for some of the functions of the information processing device 1 described later, the structure of the navigation system 100 in the second embodiment is the same as that of the navigation system 100 in the first embodiment; therefore, common structures are marked with the same symbols and detailed descriptions are omitted.

[0116] Figure 10 This is an explanatory diagram based on the function of the information processing program P1 in the second embodiment. The processing unit 10, based on the information processing program P1 of the second embodiment, as follows... Figure 10 As shown, in addition to functioning as the interface unit 101, the current situation acquisition unit 102, the instruction output unit 103, and the control output unit 104, it also functions as the prediction accuracy determination unit 105 and the policy control unit 106.

[0117] The processing unit 10 functions as a prediction accuracy determination unit 105, which determines the accuracy of the prediction information contained in the output information from the learning model M1. At the time point of receiving the instruction, the processing unit 10 inputs the sea state data acquired by the current situation acquisition unit 102 into the learning model M1 and inputs the instruction data to output a situation prediction up to the target position. The processing unit 10, through the learning model M1, quantifies the accuracy based on the difference between the predicted situation at any time point up to the target position and the situation shown by the sea state data acquired by the current situation acquisition unit 102 after receiving the instruction. The processing unit 10 preferably performs this processing whenever the time point corresponding to the prediction data arrives between the time point from receiving the instruction to reaching the target position. As the prediction accuracy determination unit 105, the processing unit 10 feeds back the accuracy value calculated based on the larger the difference, the worse the prediction accuracy. The prediction accuracy is expressed, for example, by numerical values ​​or symbols. If the prediction accuracy is less than a specified value (or the difference is greater than a specified value) and the processing unit 10 can determine that the accuracy is poor, it explicitly instructs the learning model M1 to re-predict and can re-acquire instruction data (see reference). Figure 12 In this case, the processing unit 10 should notify the user to re-perform the prediction and re-output the instruction data via the display unit 13 and / or the voice input / output unit 15.

[0118] The processing unit 10 functions as a policy control unit 106, which generates medium- to long-term control policies to ensure stable ship handling of the vessel S. The processing unit 10 generates control policies based on instructions received from the interface unit 101 and sea state data acquired from the current situation acquisition unit 102, referencing ship handling knowledge. The processing unit 10 may also store templates of control policies corresponding to the instructions and sea state data as ship handling knowledge and refer to these templates. As a policy control unit 106, the processing unit 10 determines whether the command data (parameters) output from the command output unit 103 conforms to the control policies. If it determines that they do not conform, it may provide control policies to the learning model M1 and cause it to re-output command data for more stable ship handling. For example, the processing unit 10, as a policy control unit 106, generates a medium- to long-term policy that, after avoiding a collision to the right at the target position, returns to the initially determined route from the starting position to the target position. In this case, the processing unit 10, through the command output unit 103, determines at each change point in the command data up to the target position whether the command data output by the learning model M1 contradicts the medium- to long-term policy generated by the policy control unit 106. If there is a contradiction, the command data is re-output. The processing unit 10, acting as the policy control unit 106, can also use a learning model learned in a manner that outputs control policies when inputting instructions and sea state data to generate medium- to long-term control policies.

[0119] The processing unit 10 can also function as a policy control unit 106, which creates a current control policy to stabilize the ship S by referencing past control history. In this case, the processing unit 10 stores the rudder angle, thrust, and / or output from the power source as control history in the storage unit 11. As the policy control unit 106, the processing unit 10 creates a specific policy for stable ship handling based on the changes in rudder angle, thrust, and / or power source, according to the control history stored in the storage unit 11. For example, the processing unit 10 determines whether the change in rudder angle output from the control output unit 104 is within a specified range. If it determines that the change is outside the specified range, it creates a control policy for the learning model M1 to prevent a significant change in rudder angle. For example, the processing unit 10 determines whether the recent change in thrust output from the control output unit 104 is within a specified range. If it determines that the change is outside the specified range, it creates a control policy for the learning model M1 to prevent a significant change in thrust. The processing unit 10 inputs the determined control policy as a guideline into the learning model M1. The control policy can be described as text in natural language, or it can be shown as a value or symbol assigned to a pre-set guideline. If the control policy contradicts the parameters output from the control output unit 104, or its changes, the instruction data can be re-output and re-acquired.

[0120] The functions of the prediction accuracy determination unit 105 and / or the policy control unit 106 can also be implemented in the server device 4.

[0121] Even when the interface unit 101 does not receive an instruction, the processing unit 10 can still generate control policies based on the sea state data acquired by the current situation acquisition unit 102 whenever such data is acquired. The processing unit 10 can also generate control policies when it determines a change in the situation, such as when parameters contained in the sea state data change significantly.

[0122] In the second embodiment, the learning model M1 receives text indicating an instruction received at the interface unit 101, and inputs sea state data acquired at the current situation acquisition unit 102. In addition, such as... Figure 10 As shown, the learning model M1 also receives the accuracy output from the prediction accuracy determination unit 105 and the control policy output from the policy control unit 106. The processing unit 10, through the function of the instruction output unit 103, inputs the instruction content, the current status data, the accuracy, and the control policy data to the learning model M1. The processing unit 10 extracts parameters from the instruction data output by the self-learning model M1 and shapes them into control data as control output unit 104.

[0123] In the second embodiment, the learning model M1 can also be configured as a situation prediction model and an instruction output model. Figure 11 This is a schematic diagram of another form of the learning model M1. The learning model M1 can be divided into a situation prediction model M11 and a command output model M12. The situation prediction model M11 learns by taking sea state data obtained through the current situation acquisition unit 102 as input, predicting the situation, and outputting prediction data. The situation prediction model M11 is a multimodal model that receives inputs such as image data obtained from the camera 301, images of object detection obtained from the radar 302, etc., and ship speed values ​​obtained from the ship speedometer, and outputs prediction results of image data and / or numerical values. The prediction data is, for example, sea state data obtained by the sensor 3 after a predetermined time such as several minutes. The sea state data obtained as prediction data includes, for example, scenes captured by the camera 301, surrounding conditions captured by the radar 302, and / or detection results of objects in the sea detected by the sonar 303. The command output model M12 receives the prediction data output from the situation prediction model M11 and the input of instructions received from the operator, and outputs command data.

[0124] Referring to the flowchart, an example of the function of the processing unit 10 in the second embodiment as the prediction accuracy determination unit 105 will be explained. Figure 12 This is a flowchart illustrating an example of the processing steps performed by the information processing apparatus 1 in the second embodiment. The processing unit 10 of the information processing apparatus 1 in the second embodiment performs the same operations as in the first embodiment. Figure 7 The processing steps shown involve shaping and outputting control data for control device 2, followed by the following processing.

[0125] The processing unit 10 reads the stored instruction data acquired and output from the learning model M1 via the instruction output unit 103 according to the instruction (step S201). The processing unit 10 retrieves the predicted data based on the sea state data acquired at the time the instruction was received from the read instruction data (step S202). The processing unit 10 determines the measured sea state data corresponding to the time point of the predicted data from the data acquired by the current situation acquisition unit 102 (step S203). The time point corresponding to the predicted data refers to a predetermined time after receiving the instruction. The predetermined time is the time until the time point when the learning model M1 is instructed to predict the object. For example, if the processing unit 10 outputs the predicted sea state data for 10 minutes later in response to the instruction, the predetermined time is 10 minutes. In step S203, if the time point corresponding to the predicted data has not been reached, the processing unit waits until it can acquire the data.

[0126] The processing unit 10 compares the predicted data obtained in step S202 with the sea state data acquired in step S203 (step S204). The processing unit 10 determines the prediction accuracy based on the magnitude of the difference (discrepancy) between the predicted data and the actual sea state data (step S205). In step S205, the processing unit 10 determines the prediction accuracy to be worse when the difference is larger and better when the difference is smaller. The processing unit 10 can calculate the prediction accuracy numerically or determine it by sign.

[0127] The processing unit 10 determines the quality of the prediction accuracy by indicating whether the value of the prediction accuracy is above or within a specified range, or whether the sign of the prediction accuracy is a specific sign (step S206). If the prediction accuracy is determined to be good (S206: YES), it can be inferred that the instruction output using the prediction data based on the learning model M1 is reliable. Therefore, the processing unit 10 continues processing directly. The processing unit 10 determines whether there is other prediction data (step S207), and if it determines that there is no prediction data (S207: NO), the prediction accuracy determination process ends. In step S207, if the prediction data based on the sea state data acquired at the time of receiving the instruction contains predictions about multiple time points in the future from the time of receiving the instruction, the processing unit 10 determines that there is prediction data. If it determines that there is prediction data (S207: Yes), the processing unit 10 returns the processing to step S203. In this case, the processing unit 10 executes the processing of steps S204-S207 at the next prediction time point.

[0128] In step S206, if the prediction accuracy is determined to be poor (S206: No), the processing unit 10 re-inputs the shaped instruction text (language data) corresponding to the instruction data and the image data contained in the measured sea state data obtained in step S203 to the learning model M1 (step S208). The processing unit 10 obtains the instruction data re-output from the learning model M1 (step S209). The processing unit 10 stores the output (instruction data) from the learning model M1 obtained in step S209 (step S210).

[0129] The processing unit 10, acting as the control output unit 104, extracts the parameters to be provided to the control device 2 from the text of the acquired instruction data and shapes them for use by the control device 2 (step S211). The processing unit 10 outputs the shaped parameters (control data) to the control device 2 through the control output unit 104 (step S212).

[0130] The processing unit 10, acting as the command output unit 103, generates data for outputting text, images, and / or voice to the display unit 13 (step S213). In step S213, the processing unit 10 preferably includes explanatory text, explanatory images, etc., containing information about the predicted destination from the current situation in the command output unit 103. The processing unit 10 outputs the generated text, images, and / or voice from the display unit 13 and / or the voice input / output unit 15 (step S214), and proceeds to step S207. In step S214, since the prediction accuracy is not good, the processing unit 10 may also clearly indicate that command data has been output again and output it.

[0131] The processing unit 10 can also provide the prediction accuracy determined in step S205 to the learning model M1, so that it can relearn regardless of the prediction accuracy. Therefore, it is expected that the prediction accuracy of the learning model M1 can be further improved.

[0132] Because the processing steps performed by the instruction output unit 103 of the processing unit 10 are the same as those in the first embodiment... Figure 7 The processing steps shown are the same, so the description of the example display screen 131 is also omitted.

[0133] According to the navigation system 100 of the second embodiment, the vessel S can navigate automatically and appropriately based on instructions from the operator in natural language, taking into account sea state predictions. By presenting the instructions and predicted sea states along with the automatically determined command content, the causal relationship for navigation control can be appropriately clarified. Furthermore, it is expected that the accuracy of the sea state predictions used as the basis for the commands can be improved, and the navigation of the vessel S can be controlled based on more appropriate command content. In addition, it is expected that the control can obtain more appropriate command content based on the vessel S's past navigation performance.

[0134] [Third Implementation Form]

[0135] Figure 13 This is an explanatory diagram illustrating the functions of the information processing program P1 based on the third embodiment. The processing unit 10, based on the information processing program P1 and the learning model M1, performs… Figure 12 The various functions shown. Except for some functions of the information processing device 1 described later, the structure of the navigation system 100 in the third embodiment is the same as that of the navigation system 100 in the first and second embodiments. Therefore, the common structures are marked with the same symbols and detailed descriptions are omitted.

[0136] In the third implementation form, such as Figure 13As shown, the functions of the policy control unit 106 are different. The processing unit 10 of the information processing device 1 in the third embodiment performs the following functions: it receives the input of command data output by the command output unit 103 instead of the sea state data from the current situation acquisition unit 102 through the policy control unit 106, and determines the specific control data to be provided to the control device 2 based on the input command data.

[0137] Figure 14 This is a flowchart illustrating an example of the instruction data output processing steps performed by the information processing apparatus 1 in the third embodiment. For Figure 14 The processing steps shown are consistent with those in the first embodiment. Figure 7 The processing steps are common to each other, and the same step numbers are marked with detailed descriptions omitted.

[0138] In the third embodiment, if the instruction data acquired in step S109 is stored as a history record (S110), the processing unit 10, through the function of the policy control unit 106, determines parameters that correspond to the content of the acquired instruction data and provides them to the control device 2 based on past ship handling performance (step S131). In step S131, the processing unit 10, for example, confirms whether the parameters that can be extracted from the instruction data are within the data range required for stable ship handling of the vessel S before making a decision. If they are not within the data range, the processing unit 10 may also determine the parameters within the data range. The processing unit 10 may also extract ship handling performance data consistent with the predicted sea state data consistent with the instruction data from past performance data and determine parameters consistent with the extracted performance data.

[0139] The processing unit 10 shapes the determined parameters for use by the control device 2 (step S132) and outputs them to the control device 2 (S112).

[0140] According to the structure of the navigation system 100 in the third embodiment, the vessel S can be automatically navigated appropriately based on instructions from the operator in natural language, taking into account sea conditions. By presenting the instructions and predicted sea conditions along with the automatically determined commands, the causal relationship for navigation control can be appropriately clarified. Furthermore, it is expected that control decisions will be made and control data suitable for the past stable navigation performance of the vessel S will be output, thus obtaining appropriate commands for each vessel S.

[0141] [Fourth Implementation Form]

[0142] In the fourth embodiment, the processing unit 10, acting as the command output unit 103, generates output data that is output to the display unit 13 and / or the voice input / output unit 15 based on the command data obtained using the learning model M1. The output data includes numerical values ​​of the propulsion force and / or the output quantity from the power source contained in the command data, as well as images or voice based on the prediction information obtained from the learning model M1.

[0143] The structure of the navigation system 100 in the fourth embodiment is the same as that of the navigation system 100 in the first embodiment, except for some of the functions of the information processing device 1 described later. Therefore, the common structures are marked with the same symbols and detailed descriptions are omitted.

[0144] In the fourth embodiment, the processing unit 10 is manufactured in the same manner as in the first embodiment. Figure 7 When the data output as command content in step S114 is shown, an image showing the situation of the predicted destination or the navigation route is created and displayed by the display unit 13. For example, the processing unit 10 creates an image that overlays the navigation route controlled by the command data onto the image being captured by the camera 301.

[0145] Figure 15 This is an explanatory diagram showing an example of the display screen 131 in the display unit 13 of the fourth embodiment. Figure 15 The displayed screen 131 shown is the same as the one in the first embodiment. Figure 9 Similarly, the display screen 131 shown monitors the image 132 being captured by the camera 301. In the fourth embodiment, as... Figure 15 As shown, the processing unit 10 creates and displays a route image 134 overlaid on image 132, showing the navigation path of the ship S. The output of route image 134 is not mandatory and may only be command data or numerical values ​​of rudder angle, propulsion, and / or output from the power source.

[0146] Thus, the information processing device 1 in the navigation system 100 uses the learning model M1 to determine the control data for automatic navigation, and at the same time visualizes and prompts the automatically determined instructions, thereby clarifying the content of navigation control.

[0147] Navigation system 100 can also target Figure 1 The ship S shown is operated remotely via communication to control an unmanned vessel. Figure 16 This is a schematic diagram of another embodiment of the navigation system 100. In this other embodiment of the navigation system 100, the information processing unit 1 is mounted on the mothership MS, which is not the object of navigation. The information processing unit 1 may also exist on land, similar to the server unit 4. Figure 2In the navigation system 100 shown in the schematic diagram, the unmanned vessel S from the mother ship MS is used as the control target. In this case, the information processing unit 1 receives instructions from its operator and acquires sea state data from the sensor 3 via a communication device 5 mounted on the vessel S through remote communication. The communication device 5, like the communication unit 12 of the information processing unit 1, is a communication module that enables remote communication. The communication device 5 can be a communication module using a dedicated AIS frequency, a wireless communication module connected to an operator's network, a wireless communication module for WiFi, or a communication module that communicates with the information processing unit 1 via satellite communication. The communication device 5 can transmit sea state data obtained from the sensor 3 connected to the control unit 2 to the information processing unit 1. The communication device 5 stores identification data of the vessel S as the control target and establishes a correspondence with the identification data of the vessel S when transmitting and receiving data. The information processing device 1 can adopt any of the structures shown in the first to fourth embodiments described above, and provides the instruction data obtained by inputting operator instructions and sea state data into the learning model M1 to the control device 2 via the communication device 5 of the vessel S, which is the object of ship handling, through remote communication. The control device 2 controls the steering gear and the like based on the instruction data, and can also control any of the devices in the sensor 3 based on the instruction data.

[0148] The embodiments disclosed above are exemplary in all respects and not restrictive. The scope of the invention is set forth in the claims and includes all modifications within the meaning and scope equivalent to the claims. Configurations obtained by suitably combining the technical means disclosed in the various embodiments are also included within the technical scope of the invention.

[0149] Furthermore, the independent and dependent claims described in the claims statement can be combined with each other in any combination, regardless of the form of reference. Moreover, although the claims statement uses the form of a claim that references two or more other claims (multiple claim form), it is not limited to this. It can also use the form of a multiple claim that references at least one multiple claim (multiple dependent claim form).

[0150] Regarding the above implementation methods, the following notes are further disclosed.

[0151] (Note 1)

[0152] An information processing device, comprising:

[0153] The interface section accepts the operator's ship handling instructions as language-based data input;

[0154] The current situation acquisition unit acquires sea state data that shows the operational status of the mobile watercraft being operated on; and

[0155] The command output unit uses a learning model to output command data. When the sea state data and the language data are input, the learning model outputs command data that corresponds to the indication of the language data in relation to the state shown by the sea state data.

[0156] (Note 2)

[0157] The information processing apparatus according to Appendix 1 includes a control output unit.

[0158] The control output unit outputs control data for operating the moving body on the water, based on the output of the learning model.

[0159] (Note 3)

[0160] According to the information processing apparatus described in Appendix 1 or 2, wherein,

[0161] The instruction output unit outputs notification data, which includes at least one of the following: instruction content corresponding to the ship handling instruction, sea state data serving as the basis for the operation control, and instruction content corresponding to the instruction data.

[0162] (Note 4)

[0163] The information processing apparatus according to any one of Appendices 1 to 3, wherein the language data is speech or text of natural language.

[0164] (Note 5)

[0165] The information processing apparatus according to any one of Appendices 1 to 4, wherein,

[0166] The learning model includes a multimodal model and a natural language model.

[0167] (Note 6)

[0168] The information processing apparatus according to any one of Appendices 1 to 5, wherein,

[0169] The knowledge data on the operation of the waterborne mobile body is provided to the learning model as prerequisite data.

[0170] (Note 7)

[0171] According to the information processing device described in Appendix 5

[0172] This enables other devices to perform a portion of the computations in the learning model.

[0173] (Note 8)

[0174] The information processing apparatus according to any one of Appendices 1 to 7 includes a processing unit.

[0175] The processing unit processes the sea state data acquired by the current situation acquisition unit using an algorithm different from the learning model.

[0176] The instruction output unit inputs the sea state data, processed by the processing unit, into the learning model.

[0177] (Note 9)

[0178] The information processing apparatus according to any one of Appendices 1 to 8, wherein,

[0179] The sea state data includes data obtained from at least one of cameras, radar, sonar, GPS receivers, weather sensors, automatic identification devices for ships, and fish detectors.

[0180] (Postscript 10)

[0181] According to the information processing apparatus described in Appendix 9, wherein...

[0182] The command output unit inputs data obtained as sea state data from at least one of the radar, sonar, GPS receiver, weather sensor, automatic identification device for ships, and fish detector into the learning model as image data.

[0183] (Postscript 11)

[0184] The information processing apparatus according to any one of Appendices 1 to 10, wherein,

[0185] The command data includes at least one of the following: the rudder angle of the waterborne mobile body, the thrust, the output from the power source, and the command content for the equipment mounted on the waterborne mobile body.

[0186] (Postscript 12)

[0187] The information processing apparatus according to any one of Appendices 1 to 11, wherein,

[0188] The learning model outputs predicted data of the sea state data related to the input ship handling instructions along with the instruction data.

[0189] (Postscript 13)

[0190] The information processing apparatus according to Appendix 12 includes a prediction accuracy determination unit.

[0191] The prediction accuracy determination unit determines the prediction accuracy of the prediction data output by the learning model by comparing it with the sea state data acquired successively by the current situation acquisition unit.

[0192] The learning model receives feedback on the prediction accuracy and outputs prediction data or instruction data.

[0193] (Postscript 14)

[0194] The information processing apparatus according to any one of claims 1 to 13 includes a policy control unit.

[0195] The policy control unit generates control policies for the stable navigation of the waterborne mobile body.

[0196] The instruction output unit inputs the control policy into the learning model, thereby outputting instruction data that follows the control policy.

[0197] (Postscript 15)

[0198] The information processing apparatus according to any one of Appendices 1 to 14, wherein the waterborne mobile body is an unmanned operating vessel.

[0199] (Postscript 16)

[0200] An information processing system, comprising:

[0201] Control devices for controlling the navigation of a moving body on water as the object of ship handling; and

[0202] The information processing device outputs control data for the operation and control of the waterborne mobile body to the control device.

[0203] The information processing device includes:

[0204] The interface section accepts the operator's ship handling instructions as language-based data input;

[0205] The status acquisition unit acquires sea state data, which indicates the operational status of the mobile watercraft, via communication.

[0206] The instruction output unit uses a learning model to output instruction data. When the sea state data and the language data are input, the learning model outputs instruction data that is appropriate to the indication of the language data corresponding to the state shown by the sea state data.

[0207] The control output unit, based on the output of the learning model, outputs control data for operating the moving body on the water; and

[0208] The communication unit provides the control data to the control device via communication.

[0209] (Postscript 17)

[0210] An information processing method, wherein,

[0211] Input ship handling instructions from the operator as linguistic data;

[0212] Acquire sea state data that shows the operational status of the mobile watercraft being operated on;

[0213] Using a learning model, when the sea state data and the language data are input, the learning model outputs instruction data that is appropriate to the indication of the language data corresponding to the operating environment of the sea state data;

[0214] Based on the output of the learning model, control data for operating and controlling the moving body on the water is output.

[0215] (Postscript 18)

[0216] A computer program that causes a computer to perform the following processes:

[0217] Input ship handling instructions from the operator as linguistic data;

[0218] Acquire sea state data that shows the operational status of the mobile watercraft being operated on;

[0219] Using a learning model, when the sea state data and the language data are input, the learning model outputs instruction data that is appropriate to the indication of the language data corresponding to the operating environment of the sea state data;

[0220] Based on the output of the learning model, control data for operating and controlling the moving body on the water is output.

[0221] [the term]

[0222] Not all objectives, effects, or advantages can be achieved according to any particular embodiment described in this specification. Therefore, it will be apparent to those skilled in the art, for example, that a particular embodiment may be configured to operate in a manner that achieves or optimizes one or more effects or advantages taught in this specification, without necessarily achieving other objectives, effects, or advantages as taught or implied in this specification.

[0223] All processes described in this specification can be implemented by software code modules that execute through a computing system containing one or more computers or processors, and can be fully automated. The code modules can be stored on any type of non-transitory computer-readable medium or other computer storage device. Some or all of the methods can be implemented using dedicated computer hardware.

[0224] It will be apparent from this disclosure that many other variations exist besides those described herein. For example, depending on the embodiment, any particular action, event, or function of the algorithm described herein may be executed in different sequences and may be added, combined, or excluded entirely (e.g., not all described behaviors or events are necessary for executing the algorithm). Furthermore, in certain embodiments, actions or events may be executed in parallel rather than sequentially, such as through multithreading, interrupt handling, or via multiple processors or processor cores, or on other parallel architectures. Furthermore, different tasks or processes may also be executed by different machines and / or computing systems that can function together.

[0225] The various exemplary logic blocks and modules described in association with the embodiments disclosed in this specification can be implemented or executed by a machine such as a processor. The processor may be a microprocessor, but alternatively, it may be a controller, microcontroller, state machine, or a combination thereof. The processor may include circuitry configured to process computer-executable commands. In another embodiment, the processor includes an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable devices that perform logic operations without processing computer-executable commands. The processor may also be implemented as a combination of computing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor (DSP) core, or any other such configuration. While this specification is primarily directed towards digital technologies, the processor may also primarily comprise analog elements. For example, some or all of the signal processing algorithms described in this specification may be implemented using analog circuitry or mixed-signal circuitry. The computing environment includes, but is not limited to, computer systems based on computing engines within microprocessors, mainframe computers, digital signal processors, portable computing devices, device controllers, or apparatuses, and may include any type of computer system.

[0226] Unless otherwise stated, conditional terms such as “can,” “has been able,” “will,” or “possibly” are understood in their usual context and are used to convey that a particular implementation includes a specific feature, element, and / or step that other implementations do not. Therefore, such conditional terms generally do not imply that the feature, element, and / or step is any method necessary for one or more implementations, or that one or more implementations necessarily contain logic for determining whether such feature, element, and / or step is included in any particular implementation or whether it is performed.

[0227] Disjunctive phrases such as “at least one of X, Y, and Z” are understood in their usual context unless otherwise specified, and are used to indicate that an item, term, etc., can be any one of X, Y, and Z, or any combination thereof (e.g., X, Y, Z). Therefore, such disjunctive phrases generally do not imply that a particular embodiment requires at least one of X, at least one of Y, or at least one of Z to exist separately.

[0228] Any process description, element, or block in the flowcharts described in this specification and / or shown in the accompanying drawings should be understood as representing a module, segment, or code that potentially contains one or more executable commands for implementing a particular logical function or element in the process. Alternative embodiments are included within the scope of the embodiments described in this specification, in which elements or functions may, depending on their relevant functionality, be performed substantially simultaneously with or in reverse order of the illustrated or described content, or may be omitted or performed out of order.

[0229] Unless otherwise expressly stated, numerals such as “a” should generally be interpreted as including more than one described item. Therefore, phrases such as “a device configured to perform ~” are intended to include more than one listed device. Such a list of one or more listed devices can also be construed collectively as a reference to the execution of the description. For example, “a processor configured to perform A, B, and C” could include a first processor configured to perform A, and a second processor configured to perform B and C. Furthermore, even when a specific number of imported embodiments is explicitly listed, those skilled in the art should interpret such a list as generally meaning at least a number of listed items (e.g., a simple listing of “two items” without other modifiers generally means at least two items, or more than two items).

[0230] Generally, those skilled in the art will interpret the terms used in this specification as intended to be “non-limiting” terms (e.g., the term “comprising ~” should be interpreted as “not only that, but at least includes ~”, the term “having ~” should be interpreted as “at least has ~”, the term “including” should be interpreted as “including, but not limited to”, etc.).

[0231] For illustrative purposes, the term "horizontal" as used in this specification is not related to direction, but is defined as a plane parallel to the floor plane or surface of the area where the system described is used, or as the plane on which the method described is performed. The term "floor" may be used interchangeably with "ground" or "water surface." The term "vertical / plumb" refers to a direction perpendicular to / plumped from the defined horizontal line. Terms such as "upper side," "lower side," "below," "upper," "side," "higher," "lower," "above," "across," and "below" are defined relative to a horizontal plane.

[0232] The terms “attachment,” “connection,” “merging,” and other related terms used in this specification, unless otherwise noted, shall be construed as including detachable, movable, fixed, adjustable, and / or removable connections or links. Connections / links include direct connections and / or connections with intermediate structures between the two constituent elements described.

[0233] Unless otherwise expressly stated, the figures introduced by terms such as “about,” “approximately,” and “substantially” as used in this specification include the listed figures and also represent quantities that are close to the described quantities and perform the required function or achieve the desired result. For example, “about,” “approximately,” and “substantially”, unless otherwise expressly stated, refer to values ​​less than 10% of the described values. As used in this specification, features of disclosed embodiments introduced by terms such as “about,” “approximately,” and “substantially” represent features that have some variability and perform the required function or achieve the desired result with respect to said features.

[0234] Numerous variations and modifications may be incorporated into the above embodiments, and these elements should be understood as belonging to other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure and protected by the following claims.

[0235] Explanation of icon numbers

[0236] 100: Navigation System

[0237] 1: Information processing device

[0238] 10: Processing Department

[0239] 101: Interface Department

[0240] 102: Current Situation Acquisition Department

[0241] 103: Instruction Output Section

[0242] 104: Control Output Section

[0243] 105: Prediction Accuracy Judgment Department

[0244] 106: Strategy Control Department

[0245] 11: Storage Department

[0246] 12: Ministry of Communications

[0247] 13: Display Section

[0248] 131: Display screen

[0249] 14: Operations Department

[0250] 15: Voice Input / Output Section

[0251] P1: Information Processing Program

[0252] M1: Learning Model

[0253] M11: Situation Prediction Model

[0254] M12: Instruction Output Model

[0255] 2: Control device

[0256] 3: Equipment

[0257] 301: Camera

[0258] 302: Radar

[0259] 303: Sonar

[0260] 304: Ship speedometer

[0261] 305: GPS Receiver

[0262] 4: Server equipment

[0263] 42: Ministry of Communications

[0264] 5: Communication device

Claims

1. An information processing apparatus, comprising: The interface section accepts the operator's ship handling instructions as language-based data input; The current situation acquisition unit acquires sea state data that shows the operational status of the mobile watercraft being operated on; as well as The command output unit uses a learning model to output command data. When the sea state data and the language data are input, the learning model outputs command data that corresponds to the indication of the language data in relation to the state shown by the sea state data.

2. The information processing device according to claim 1, comprising a control output unit, The control output unit outputs control data for operating the moving body on the water, based on the output of the learning model.

3. The information processing apparatus according to claim 1, wherein, The instruction output unit outputs notification data, which includes at least one of the following: instruction content corresponding to the ship handling instruction, sea state data serving as the basis for the operation control, and instruction content corresponding to the instruction data.

4. The information processing apparatus according to claim 1, wherein, The linguistic data is speech or text in natural language.

5. The information processing apparatus according to any one of claims 1 to 4, wherein, The learning model includes a multimodal model and a natural language model.

6. The information processing apparatus according to claim 5, wherein, The knowledge data on the operation of the waterborne mobile body is provided to the learning model as prerequisite data.

7. The information processing apparatus according to claim 5, wherein, This enables other devices to perform a portion of the computations in the learning model.

8. The information processing apparatus according to claim 1, comprising a processing unit, The processing unit processes the sea state data acquired by the current situation acquisition unit using an algorithm different from the learning model. The instruction output unit inputs the sea state data, processed by the processing unit, into the learning model.

9. The information processing apparatus according to claim 1, wherein, The sea state data includes data obtained from at least one of cameras, radar, sonar, GPS receivers, weather sensors, automatic identification devices (AIDs), and fish detectors.

10. The information processing apparatus according to claim 9, wherein, The command output unit inputs data obtained as sea state data from at least one of the radar, sonar, GPS receiver, weather sensor, AIS and fish detector into the learning model as image data.

11. The information processing apparatus according to claim 1, wherein, The command data includes at least one of the following: the rudder angle of the waterborne mobile body, the thrust, the output from the power source, and the command content for the equipment mounted on the waterborne mobile body.

12. The information processing apparatus according to claim 1, wherein, The learning model outputs predicted data of the sea state data related to the input ship handling instructions along with the instruction data.

13. The information processing apparatus according to claim 12, comprising a prediction accuracy determination unit, The prediction accuracy determination unit determines the prediction accuracy of the prediction data output by the learning model by comparing it with the sea state data acquired successively by the current situation acquisition unit. The learning model receives feedback on the prediction accuracy and outputs prediction data or instruction data.

14. The information processing apparatus according to claim 1, comprising a policy control unit, The policy control unit generates control policies for the stable navigation of the waterborne mobile body. The instruction output unit inputs the control policy into the learning model, thereby outputting instruction data that follows the control policy.

15. The information processing apparatus according to claim 1, wherein, The waterborne mobile body is an unmanned vessel.

16. An information processing system, comprising: Control device, used to control the movement of a moving body on water that is the object of operation; as well as The information processing device outputs control data for the operation and control of the waterborne mobile body to the control device. The information processing device includes: The interface section accepts the operator's ship handling instructions as language-based data input; The status acquisition unit acquires sea state data, which indicates the operational status of the mobile watercraft, via communication. The instruction output unit uses a learning model to output instruction data. When the sea state data and the language data are input, the learning model outputs instruction data that is appropriate to the indication of the language data corresponding to the state shown by the sea state data. The control output unit, based on the output of the learning model, outputs control data for operating the moving body on the water; and The communication unit provides the control data to the control device via communication.

17. An information processing method, wherein, Input ship handling instructions from the operator as linguistic data; Acquire sea state data that shows the operational status of the mobile watercraft being operated on; Using a learning model, when the sea state data and the language data are input, the learning model outputs instruction data that is appropriate to the indication of the language data corresponding to the operating environment of the sea state data; Based on the output of the learning model, control data for operating and controlling the moving body on the water is output.

18. A computer program that causes a computer to perform the following processes: Input ship handling instructions from the operator as linguistic data; Acquire sea state data that shows the operational status of the mobile watercraft being operated on; Using a learning model, when the sea state data and the language data are input, the learning model outputs instruction data that is appropriate to the indication of the language data corresponding to the operating environment of the sea state data; Based on the output of the learning model, control data for operating and controlling the moving body on the water is output.

Images