System for implementing a digital voice-managed navigation logbook integrated into the data acquisition system of a watercraft and application method

Abstract

The present invention relates to a system that allows natural language processing (NLP) to be used to enable the bridge crew of a ship to verbally update logbook entries in real time and to request information regarding the logbook, natural language being used at all times for human-machine interaction. The entries may also relate to data extracted from ship sensors. The system comprises: an audio acquisition system for capturing user requests; a processing device for processing the audio by means of artificial intelligence (AI) in combination with NLP; a memory for storing the data generated by the system; a speaker that allows the verbal response of the system; a plurality of common peripherals such as a mouse and computer keyboard; a display device; and network connection means that adjust to the synchronisation of the local system on the ship with the global system.

Description

[0001] SYSTEM FOR IMPLEMENTING A DIGITAL NAVIGATION LOGBOOK MANAGED BY VOICE AND INTEGRATED WITH THE DATA ACQUISITION SYSTEM OF A VESSEL AND APPLICATION METHOD OBJECT OF THE INVENTION

[0002] The present invention reveals a system that allows the implementation of a digital, voice-managed logbook, integrated with a vessel's data acquisition system, based on IoT, AI, and natural language processing (NLP) techniques. The invention aims to use natural language processing to allow the bridge crew to verbally update logbook entries in real time and request information from it. Human-machine interaction is carried out at all times through the use of natural language. These entries would also be related to data extracted from the ship's sensors.

[0003] TECHNICAL SECTOR

[0004] Digital logbook. Electronic logbook. Logbook. BACKGROUND OF THE INVENTION

[0005] A voyage log is a document located on the bridge of a vessel. This logbook, whether paper or electronic, can record a large amount of information of various types. It generally records voyage parameters and / or data (e.g., voyage logs, fuel quantities, de-icing data, various reports) as well as maintenance parameters and data (e.g., management of defects observed on the aircraft by the crew or maintenance operators, vessel maintenance history, etc.). This particular document is subject to several naval regulations. In particular, it constitutes a single reference, due to the regulation. Today, the logbook, whether paper or electronic, is physically located on board the vessel. Its physical location requires sequential updates (i.e., one user after another).A logbook, for example, made of paper, generally comprises several sheets, with several carbon copies for each sheet. These copies allow different users to have read access to the aircraft's technical status. In order to plan maintenance, which requires a consolidated technical status, each sheet used in the logbook is manually copied into the vessel's maintenance system. The paper original of the logbook is kept on board. It is this original that is updated by the crew and certain operators. In reality, the logbook records the vessel's technical condition, which the ship's master takes into account when preparing for the voyage, accepts before departure, and updates as necessary during the voyage.Following these updates, and in accordance with the vessel's scheduled maintenance, maintenance operators report on the maintenance operations performed to ensure the vessel's seaworthiness. The circulation of the printed document is therefore limited in space and / or time, following a series of predefined stages. The electronic versions of the logbook directly transpose the paper version. Updates are performed on board in a sequential manner, with the only difference being that the maintenance system is updated electronically. In certain cases, these electronic versions allow for remote consultation.

[0006] This information management (paper logbook or its transposition into electronic format) has limitations. The prior art describes various approaches to facilitate the manipulation of a logbook (particularly in its electronic version). These approaches resolve certain technical problems, but have residual limitations. Sequential write access may require users to wait for their volume before being able to make an update. As for ground crews, they only have access to potentially obsolete and / or incomplete information (and in particular without having been warned), for example, due to particular timing constraints and limitations (e.g., manual copies, etc.). There is a need for systems and methods to manage the data recorded in this regulated logbook.

[0007] In the technical state of the art, document US11276254 discloses a system for implementing the management of an electronic logbook of an aircraft, said system comprising: one or more computers implementing a server hosting at least a first database, said at least one first database creating an instance of said electronic logbook, said one or more computers being configured to perform the steps of: rejecting a read and / or write request by a customer from among a plurality of said electronic logs, based on predefined criteria comprising: criteria associated with predefined time intervals determined between time milestones governing read and / or write access rights to the customer's publications during a flight;and network connectivity criteria between one or more client stations and a database associated with the electronic logbook, wherein said client station is associated with one of a plurality of user categories, and wherein each user category is assigned reader and / or write privileges with respect to each predefined time interval;and said system further comprises one or more additional computers implementing a server hosting at least a second database and located in a cockpit of an aircraft, said at least one second database instantiating said electronic logbook, said at least one second database being synchronized with at least a first database, wherein the time milestones comprise at least one of the following: an acceptance by a pilot of a technical status of the aircraft, a review of said acceptance, an approval for recommissioning, a time from which the pilot becomes active on a mission, signature-entry by a pilot, and / or signature-out of the technical status by the pilot upon completion of the mission;

[0008] US8396958 discloses an automated record keeping system for machinery on board a vessel, said automated record keeping system comprising a central computer on board a vessel and at least one local computer on board the vessel, each of said local computers including an interface for a user and being connected to said central computer and having programmed in memory a computer program product including a computer usable medium having computer program logic recorded thereon, said computer program logic being embodied in computer code to enable said corresponding local computer to perform record keeping pertaining to said machinery, said record keeping including maintaining and updating multiple log books, said maintaining and updating including editing existing log books,the addition of new logbooks and the deletion of existing logbooks, each such logbook belonging to said machine.,

[0009] None of the prior art documents indicate or suggest a sound acquisition system of sufficient quality to capture user requests in combination with information processing equipment using AI, which allows the audio to be processed. This system allows the structured processing and storage of the recorded information through the use of natural language processing techniques to understand the recorded voice messages and carry out the following sub-stages: i) Enter the extracted information into the logbook appropriately when the message requires it. ii) Understand possible questions and prepare a response; and techniques for translating and adapting the extracted sensor data and injecting it into the logbook database.

[0010] The advantages of the present invention over the prior art are evident, as the invention revealed a system that allows for the implementation of a digital, voice-managed logbook, integrated with a vessel's data acquisition system. This allows for the use of natural language processing to enable the bridge crew to automatically update logbook entries in real time and request information about them. Human-machine interaction is maintained at all times through the use of natural language. These entries are also related to data extracted from the vessel's sensors.The result would be a logbook that allows for much simpler and more precise writing, as the manager could write entries by voice while performing other tasks. Allow the user to verbally consult information contained in the logbook or available on the system's sensor network. Synchronize each voice entry with the relevant sensor data recorded by the ship. Store all the ship's sensor data as a time series through a periodic routine. Be fully digital, allowing for more secure and efficient data storage, transfer, and exploitation, while also giving much greater value to the extracted data. Be able to synchronize its data with a global system that can access all the client's vessels.

[0011] Perfect time traceability of events. This will not only facilitate the extraction of knowledge from the collected data, but also simplify insurance claims in the event of a breakdown or accident. DESCRIPTION

[0012] DESCRIPTION OF THE DRAWINGS

[0013] To complement the description being made and in order to help better understand the characteristics of the invention, according to a preferred example of its practical implementation, a set of drawings is attached as an integral part of said description, in which the following has been represented for illustrative and non-limiting purposes:

[0014] FIG 1.- A general diagram of the concept that allows implementing a digital logbook is shown, where the steps of information acquisition (1) are shown, which includes a) Recording and transcription of verbal information and b) Acquisition of information from sensors available on the bridge; data processing (2) of the recorded information with Artificial Intelligence (AI) through the use of Natural Language Processing (NLP) techniques, to understand the recorded voice messages; data storage (3) and downloading or printing of logbook data; human-machine interface (4) that allows user interaction visually and through peripherals (an arrow is shown that relates the sailor and the peripheral with the machine), such as the review and modification of logbook entries (5); the graphical representation of time series data and certain important metrics for the user (6);audios (7) and uploading data securely to a global platform or a Data Warehouse (platform used to collect and analyze data from multiple heterogeneous sources) in an automated manner, where all navigation logs are stored;

[0015] PREFERRED EMBODIMENT OF THE INVENTION

[0016] The present invention reveals a system that allows the implementation of a digital, voice-managed logbook integrated with the vessel's data acquisition system, based on IoT techniques, AI, and natural language processing. The project aims to use natural language processing to allow bridge crew members to verbally update logbook entries in real time and request information from it. Human-machine interaction is carried out at all times through the use of natural language. These entries would also be related to data extracted from the vessel's sensors. To achieve this objective, the present invention incorporates various technologies currently under development and that are part of Industry 5.0.

[0017] For the development of the invention, a concept known as Data Science is required, which allows for the unification of statistics, data analysis, machine learning, and their related methods, in order to understand and analyze real-life phenomena. Data Science is essential for the present invention, as it is present in different phases of the invention, from the use of neural networks for noise cancellation in data collection and data processing, to classification systems to extract relevant information (Information Retrieval Systems) or question and answer systems (Q&A Systems) to obtain information requested by the user. This technology encompasses other, more specific technologies such as:

[0018] Machine Learning and Deep Learning (data transformation and use): Once you have a sufficient amount of data for analysis, it's essential to have tools for processing it. This way, you can extract all the value it contains.

[0019] Machine Learning: This uses algorithms to predict future results based on historical data and data received by the system in real time. These algorithms, which use advanced mathematical functions, will be adjusted as they receive more data, allowing the system to learn automatically.

[0020] Deep Learning: Broadly speaking, it's a specific branch of machine learning that uses neural networks, a particular family of algorithms containing a series of nodes arranged in layers. Each node has specific weight functions that decide what action to take on the input. The weights contained in the neural network represent its knowledge.

[0021] Natural Language Processing, or NLP, is a field of computer science, artificial intelligence (AI), and linguistics that studies the interactions between computers and human language. In other words, it investigates how machines communicate with people through the use of natural languages, such as Spanish or English.

[0022] Natural Language Understanding (NLU) is a subcomponent of NLP (natural language processing) that deals with machine reading comprehension. NLP literally interprets what is said or written, while NLU identifies intent and deeper meaning. NLU is considered a complex problem in artificial intelligence (AI).

[0023] Classification systems allow texts to be classified according to their subject matter, relevance, positivity, etc., using artificial intelligence algorithms. Basically, they involve assigning a document to one or more categories. This is a key element in the present invention, as it requires distinguishing between what information should be incorporated and what should be discarded.

[0024] Information Retrieval Systems (IR Systems) are architectures designed to extract the relevant information required for a specific task. An IR System consists of a series of artificial intelligence algorithms and probabilistic calculations that, working together in an orderly manner, extract the necessary information from all available information.

[0025] Question answering systems (QA systems) is a discipline within the fields of information extraction and natural language processing. Q&A systems use artificial intelligence techniques to provide answers to questions posed in natural language. This is important for intelligent user interaction and information retrieval.

[0026] The system requires technologies used to process, analyze, and interpret data from the information sources provided by vessels, as well as sound data provided by system users. This requires:

[0027] Human-Computer Interaction (HCI) is the discipline that studies how people interact with computers and the extent to which computers are developed to interact with people. These include:

[0028] Speech recognition (audio-to-text transcription). Speech recognition is a technology that allows the user's voice to be received and interpreted as input data for applications. The sound produced by the interlocutor is collected through an audio input device and processed by artificial intelligence noise cancellation using neural networks, capable of differentiating between the sound produced by the interlocutor and the ambient sound (noise), eliminating the latter. Once the audio has been "cleaned," it is transcribed and stored in the vessel's internal database for subsequent processing and analysis using machine learning techniques.

[0029] Speech synthesis (text-to-audio). Another necessary process in the system is the effective communication of analysis results to the user (the Captain or pilots on watch). This process can be carried out through various means, from dashboards to voice system communication. In this case, we would be using speech synthesis, the inverse of speech recognition. This is achieved using audio output devices and text-to-speech software.

[0030] Voice recognition: It is a biometric technology for recognizing an individual's voice.

[0031] Data processing requires an ETL pipeline (Extract, Transform, and Load), which is a set of processes used to move data from one or more sources to a database or data warehouse in an automated manner. The main objective of an ETL pipeline is to collect raw data, perform the necessary transformations, and transfer it to the storage location, leaving it ready for subsequent analysis. An ETL pipeline can be planned and automated with a wide variety of tools, some more basic (crontab), others that require no code but are more rigid (Apache Nifi), and others that are more complex and modern and allow for high customization and efficiency (Apache Airflow).

[0032] Apache Airflow: It is a batch-oriented framework for orchestrating, scheduling, and automating workflows, that is, for building data pipelines. "Data pipeline" is a general term referring to the set of processes carried out to move data from a source system to a destination system. An ETL pipeline is a data pipeline that uses a data extraction, transformation, and loading strategy.

[0033] Apache Kafka: This is an event streaming platform that enables real-time data transmission. Kafka's architecture allows for low latency, horizontal scalability, and the absorption of load spikes that may occur in the system.

[0034] On-vessel storage system: Since the vessel will not have an internet connection at sea for security reasons, the data will be stored on local servers. Once in port, it will be uploaded to a cloud-based data warehouse that aggregates and optimizes the data as appropriate. The following options are being chosen to meet the vessel's storage needs, with the possibility of adding others as needed:

[0035] MySQL: It is an open source relational database management system with a client-server model.

[0036] InfluxDB: A database management system designed for working with time-series data. InfluxDB is part of a comprehensive platform that supports the collection, monitoring, visualization, and alerting of time-series data.

[0037] Data Warehouse: It is a platform used to collect and analyze data from multiple heterogeneous sources.

[0038] Data Lake: It is a centralized repository for storing large amounts of raw data, which uses a flat architecture to store it.

[0039] The system for implementing a digital logbook, managed by voice and integrated with a vessel's data acquisition system, comprises the following elements:

[0040] A sound acquisition system of sufficient quality to capture the user's requests, this system can be made up of headphones with a microphone, preferably with high-quality audio capture, along with a sound card;

[0041] A processing system powerful enough to process audio using Artificial Intelligence (AI) in combination with NLP (Natural Language Processing). It must have a high-speed processor, at least 16GB of RAM, and high-capacity memory to ensure a smooth response. If it is not possible to use a system already integrated into the vessel, and depending on available space, a barebone device (frame) could be used, a type of computer case with small dimensions and a specific design, such as a NUC (Computing Unit). Ideally, it should have industrial features (high IP protection and high heat dissipation).

[0042] Sufficient memory to store the data generated by the system. However, storage capacity should be assessed based on the mature needs of the system.

[0043] A speaker that allows for verbal feedback from the system. Depending on the case, the one included with the aforementioned headset may be sufficient.

[0044] Common peripherals such as a computer mouse and keyboard

[0045] A visualization system consisting of a monitor, where the screen could alternatively be touch-sensitive, to facilitate integration into a vessel's dashboard.

[0046] Network connection media that adapt to the synchronization needs of the system located on the ship with the global system.

[0047] Therefore, the system that allows the implementation of a voice-managed digital logbook integrated with a vessel's data acquisition system comprises: a sound acquisition system to capture user requests; processing equipment to process voice audios from the sound acquisition system using Artificial Intelligence (AI) in combination with NLP (natural language processing); a memory to store the data generated by the system; a speaker that allows the system to respond verbally; common peripherals such as a computer mouse and keyboard; visualization equipment; and network connection means that adapt to the synchronization of the system located on the vessel with the global one.

[0048] The system recognizes as input any message preceded by a specific stimulus, for example, a keyword or the pressing of a button. A fingerprint sensor or camera would be used to update the identity of the commanding user. This way, at the beginning of each shift, the officer would be identified and would be the only one authorized to add logbook entries.

[0049] Along with each update to the notebook, the AI ​​processing the voice input will be able to determine which sensor measurements relate to the input, and through this routine, it will be able to retrieve and attach them along with it.

[0050] Periodically and for a configurable period, all available sensor measurements on the vessel will be collected and stored in the logbook database, along with a time stamp.

[0051] Based on the transcription of the voice message captured by the microphone, the AI ​​will have a series of models capable of processing the text and successfully carrying out the task required.

[0052] After determining that the message is a logbook update, the AI ​​will determine how to fill out which part of the logbook structure, also considering whether it may need to update an already entered field. During this process, it will also infer which ship's sensors may be related to the logbook entry, request them, and add them to the database along with the audio and text.

[0053] To support the user, the system will be able to identify and understand questions, searching for the appropriate answer and synthesizing audio to transmit it to the user. It is understood that the system will primarily be able to:

[0054] Return current or past vessel sensor values;

[0055] Recall old logbook entries that answer the question if they exist;

[0056] Notify which logbook fields remain to be entered;

[0057] Extract key information, classify the message, determine its typology (log entry or system query).

[0058] The Human-Machine Interface (HMI) comprises the following functions: visual interaction and interaction via computer peripherals; review and modification of logbook entries; graphical representation of time-series data and certain metrics important to the user; downloading logbook entries and associated or time-series data in Excel, CSV, XML, JSON, or PDF formats; and securely uploading data to a global database.

[0059] The procedure for implementing a digital, voice-managed logbook integrated with a vessel's data acquisition system comprises the following operational steps:

[0060] 1- Acquisition of information that includes: a) Recording and transcription of verbal information b) Acquisition of information from sensors available on the bridge

[0061] 2- Structured processing and storage of the recorded information with Artificial Intelligence (AI) through the use of Natural Language Processing (NLP) techniques, to understand the recorded voice messages and carry out the following sub-stages: i) Enter the extracted information into the logbook (shipping log) appropriately, when the message requires it. ii) Understand possible questions and prepare a response; translation and adaptation techniques for the extracted sensor data, and its injection into the logbook database.

[0062] 3- User interaction visually and through peripherals that allows: a) The review and modification of logbook entries b) The graphical representation of time series data and certain important metrics for the user c) The download or printing of logbook data d) The secure upload of data to a global platform where all navigation logbooks are stored (logbook).

Claims

Periodic data acquisition Periodically and for a configurable period, all available sensor measurements on the vessel will be collected and stored in the logbook database, along with a time stamp. INFORMATION ACQUISITION Recording and transcribing verbal information The sailor responsible for supervising the voyage and preparing the logbook will have the necessary hardware to interact with the system verbally. Depending on the maturity of the product, this voice recording system will vary in complexity. Initially, a personal microphone installed in a headset or on a panel on the bridge itself will be used. To avoid errors, the system would recognize as input any message preceded by a specific stimulus, for example, a keyword or the pressing of a button. A fingerprint sensor or camera would be used to update the identity of the commanding officer. This way, at the beginning of each shift, the officer would identify themselves and would be the only one authorized to add logbook entries. At that point, the system would request permission to save each recorded voice entry along with the transcription in the digital logbook. This way, the veracity of any entries the user considers inappropriate can be corroborated. Furthermore, correcting an inappropriate entry will allow the system to continue learning in a supervised manner. Several ambient microphones could be installed on the bridge, continuously listening and recording information deemed valuable. User recognition could also be achieved through voice. Acquisition of information from sensors available on the bridge Depending on the hardware and communication protocol present on the vessel, a data acquisition routine will be programmed with two purposes: Acquisition of data related to voice input Along with each update to the notebook, the AI ​​processing the voice input will be able to determine which sensor measurements relate to the input, and through this routine, it will be able to retrieve and attach them along with it. Periodic data acquisition Periodically and for a configurable period, all available sensor measurements on the vessel will be collected and stored in the logbook database, along with a time stamp. Artificial intelligence (AI) provides databases that can be synchronized with ground stations when network conditions permit. Natural language processing Based on the transcription of the voice message captured with the microphone, the AI ​​will have a series of models capable of processing the text and successfully carrying out the task required: Entering entries in the digital logbook After determining that the message is a logbook update, the intelligence will determine how to fill out which part of the logbook structure, also considering whether it may need to update an already entered field. During this process, it will also infer which ship's sensors may be related to the logbook entry, request them, and add them to the database along with the audio and text. Question Answering Systems (Q&A Systems) To support the user, the system will be able to identify and understand questions, searching for the appropriate answer and synthesizing audio to transmit it to the user. It is understood that the system will primarily be able to: Return current or past vessel sensor values • Recall old logbook entries that answer the question if they exist • Notify which logbook fields remain to be entered • Extract key information, classify the message, determine its typology (log entry or system query). Sensor data This system model generates the system's data entries in the logbook database. It will process, transform, and adapt the extracted data each period, ensuring that: Have a structure that is easy to exploit later: If there is a lot of data that is currently useless, but could be exploited on a large scale and in conjunction with data from other vessels, a Data Lake would be used. On the other hand, if the utility, format and meaning of the data are very closed, as it seems a priori, it will make more sense to use a Data Warehouse for each vessel, and another general one that includes the small ones. • Are in the appropriate format (Sl) • They are introduced already interpreted if necessary • Filter out those whose values ​​are classified as erroneous or null. Artificial intelligence (AI) uses a wired IoT communication system to collect data from the ship's sensors and store it in databases. HUMAN-MACHINE INTERFACE Review and modification of logbook entries This module will allow you to view entire logbooks and manually modify entries that are considered incorrect or that you want to complete. It will also allow you to listen to the saved audio recordings of each entry if the user has given the system permission to store them.

Images