Underwater control system based on multi-modal neural network large model

Abstract

The invention discloses an underwater control system based on a multi-modal neural network large model, the system comprises an underwater robot and the multi-modal neural network large model, the underwater robot is used for collecting environmental data, and is also used for decomposing a received functional control instruction into an action execution signal, and sending the action execution signal to the multi-modal neural network large model; completing a set task target based on the action execution signal; the multi-modal neural network large model is used for generating prompt information based on received environment data, underwater domain knowledge and a task target sent by the underwater robot, and extracting prediction performance function sequence information capable of realizing the task target from a robot performance function library through the prompt information, and converting the performance function sequence information into a functional control instruction. According to the system, the underwater robot has high sensing ability, and a large model has understanding and reasoning ability facing the underwater environment, so that autonomous decision and accurate control of the intelligent agent with the body are realized.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and robotics technology, specifically relating to an underwater control system based on a large multimodal neural network model. Background Technology

[0002] Underwater embodied agents are intelligent robots capable of autonomously performing diverse tasks in underwater environments. These systems have significant applications in deep-sea scientific research, seabed resource exploration, environmental monitoring, and underwater engineering. However, existing control methods for underwater agents typically rely on predefined rules and traditional algorithms, which have significant limitations in terms of perception capabilities and environmental understanding.

[0003] Patent application CN114104230A discloses an underwater robot, including a support assembly, a main control cabin assembly, a lens cabin assembly, a battery cabin assembly, a supplementary lighting assembly, and multiple thrusters. The support assembly includes multiple support plates spaced apart, and multiple buoyancy tubes and support rods passing through the support plates. This structural design allows for the assembly of the entire robot using different lens cabin, main control cabin, and battery cabin assemblies according to actual needs. The completely open layout provides users with more space for expanding and developing mounted equipment. Furthermore, this invention enables the underwater robot to achieve full-degree-of-freedom attitude navigation and underwater attitude stability control, and allows for adjustment of the relative positions of the supplementary lighting and lens according to actual needs, thereby improving underwater imaging quality and stability, and providing convenient conditions for underwater vessel hull inspection.

[0004] Patent application CN108196445A discloses an underwater robot attitude and position control method based on dual closed-loop active disturbance rejection. First, a dynamic model is established for the underwater robot using the momentum and moment theorems and the transformation relationship between the geodetic coordinate system and the body coordinate system. A velocity loop expansion state observer is designed to observe unmodeled and external disturbances and compensate for and suppress them in real time, greatly increasing the velocity loop's disturbance rejection performance and improving the system's robustness. Addressing the high-frequency vibration characteristics of the velocity loop, an active disturbance rejection controller based on a nonlinear function feedback velocity loop is constructed using disturbance observations. Based on the nonlinear function fhan, a tracking differentiator for the target position signal is designed, providing the transition process of the position signal and the target velocity estimate, avoiding abrupt changes in system response and resolving the contradiction between speed and overshoot.

[0005] When faced with complex and ever-changing underwater environments, the methods described in the two patents struggle to handle dynamic obstacles and unstructured scenarios. Their autonomy and flexibility in task execution are insufficient, often requiring human intervention to complete complex tasks. Furthermore, underwater operations require a broad range of knowledge, which traditional methods cannot effectively integrate and utilize, thus limiting the system's adaptability and the reliability of task completion. Summary of the Invention

[0006] This invention provides an underwater control system based on a large multimodal neural network model. This system enables underwater robots to have high perception capabilities and allows the large model to have the ability to understand and reason about the underwater environment, thereby realizing autonomous decision-making and precise control of the embodied intelligent agent.

[0007] In a specific embodiment of the invention, the underwater robot first collects environmental data using devices such as sonar, optical cameras, and hydrological sensors. After preprocessing and feature extraction, the data is input into a multimodal large model. While receiving input from multiple sources, the model, combined with fused underwater domain knowledge, can perform semantic understanding and reasoning about the environmental state and task objectives. Based on this semantic-level analysis, the model generates a complete task decision sequence, including path planning, obstacle avoidance strategies, operational actions, and task decomposition. Subsequently, the decision sequence is further mapped into low-level executable control commands, which drive the thrusters, servos, or manipulators to complete specific actions, thereby ensuring that the underwater robot can autonomously complete tasks such as cruising, detection, grasping, and returning.

[0008] Compared with existing technologies, this invention significantly enhances the environmental perception and task understanding capabilities of underwater agents by introducing a large multimodal neural network model, enabling them to independently complete diverse tasks in complex and dynamic underwater environments. This method not only improves the autonomy and robustness of the agent but also reduces the need for human intervention, increasing task execution efficiency and accuracy while enhancing the system's adaptability to real-world marine environments. Attached Figure Description

[0009] Figure 1 This is a schematic diagram illustrating the application of the underwater embodied intelligent control method based on a large multimodal neural network model in an intelligent agent control system, as provided in a specific embodiment of the present invention. Detailed Implementation

[0010] With the development of artificial intelligence, large-scale language models and multimodal neural networks have made significant progress in natural language processing and intelligent decision-making. These models can not only understand and generate natural language, but also integrate visual, acoustic, and multi-source sensor information to achieve complex reasoning and control. Therefore, introducing large-scale multimodal neural network models into the control process of underwater embodied intelligent agents can effectively overcome the shortcomings of traditional methods and provide new solutions for improving the autonomy, intelligence, and execution efficiency of underwater missions.

[0011] The purpose of this invention is to propose an underwater embodied intelligent control method based on a large multimodal neural network model, in order to solve the technical problems of existing underwater robots, such as insufficient perception, strong environmental uncertainty, high task complexity, and limited knowledge utilization. The method of this invention introduces a multimodal perception data processing and domain knowledge fusion mechanism, enabling the large model to possess the ability to understand and reason about the underwater environment, thereby achieving autonomous decision-making and precise control of the embodied intelligent agent.

[0012] This invention provides an underwater control system based on a large multimodal neural network model, such as... Figure 1 As shown, it includes underwater robots and a large model of multimodal neural networks.

[0013] The underwater robot provided in this specific embodiment of the invention is used to collect environmental data and also to decompose received functional control commands into action execution signals, and to complete the set task objectives based on the action execution signals. In one specific embodiment, such as... Figure 1 As shown, this functional control instruction is... Figure 1 The robot commands in the system cause the robot to move from the starting point to the target object at a speed v and along a set route, thus completing the target task.

[0014] In one specific embodiment, the underwater robot provided in this embodiment includes a multimodal sensor, a control system, a drive thruster, a servo motor, a manipulator, and a preprocessing module.

[0015] In this embodiment, the multimodal sensor is used to acquire environmental information such as sonar images, water flow velocity, water pressure, water temperature, and terrain structure. This information first enters a preprocessing module, where it is transformed into a feature vector of a unified format through filtering, denoising, and normalization steps. This feature vector can simultaneously characterize temporal variations and spatial distribution features, facilitating subsequent processing and analysis of large-scale models.

[0016] Furthermore, the multimodal sensors provided in this embodiment include devices such as sonar, optical cameras, and hydrological sensors.

[0017] The control system provided in this embodiment decomposes the received functional control commands into action execution signals, which include motor speed signals, servo motor angle signals, and robot arm grasping posture signals.

[0018] The drive thruster, servo motor, and manipulator provided in this embodiment are used to receive motor speed signals, servo motor angle signals, and manipulator grasping posture signals, respectively, in order to complete movement, obstacle avoidance, detection, and sample collection operations.

[0019] The multimodal neural network large model provided in the specific embodiments of the present invention is used to generate prompt information based on the environmental data, underwater domain knowledge and task objectives sent by the underwater robot. The prompt information is used to extract the predictive function sequence information that can achieve the task objectives from the robot function library, and the function sequence information is converted into functional control instructions.

[0020] The prompts provided in the specific embodiments of this invention are dynamically generated and can constrain the direction of model inference. These prompts include not only descriptions in natural language but also symbolic labels and conditional constraints to ensure that the model output matches the actual task. The construction of these prompts follows a method driven by both static underwater domain knowledge and dynamic real-time environmental data, enabling the large model to fully consider real-time environmental constraints and domain safety standards during inference and generation, thereby outputting high-level decision-making results that meet the requirements of underwater operations, i.e., predictive function sequence information that achieves the specified task objectives.

[0021] In one specific embodiment, static underwater domain knowledge includes information such as basic underwater navigation knowledge, marine operation standards, common terrain risks, and operational experience; dynamic real-time environmental data includes obstacles identified by the current sonar image, real-time water flow velocity, and water pressure values.

[0022] In one embodiment, the multimodal neural network large model provided in this embodiment can generate corresponding prompts based on the input dynamic real-time environmental data. When the robot performs grasping or sample collection tasks, the prompts will automatically incorporate constraints such as stable pose, water flow threshold, and safe distance. When performing complex path planning, the prompts will inject constraint logic to avoid excessively steep terrain slopes and to bypass predictable obstacles in advance.

[0023] The multimodal neural network model provided in this specific embodiment of the invention, upon receiving environmental data or pre-processed environmental data, generates prompting information. This prompting information transforms the multimodal input into a unified semantic representation, identifying obstacles, water flow distribution, and terrain features. Guided by the prompting information, the representation generates a high-level decision-making scheme. Subsequently, the high-level decision is converted into low-level functional control commands through an instruction translation module and transmitted to the robot control system. The control system further decomposes these functional commands into specific execution signals such as motor speed, servo motor angle, and manipulator grasping posture, driving the thrusters, servos, and manipulator to complete operations such as movement, obstacle avoidance, detection, and sample collection. Through this process, the underwater robot can achieve autonomous perception, semantic understanding, task planning, and action execution in complex environments.

[0024] This invention defines a high-level robot function library. This library can be tailored to specific dimensions or scenarios of interest and should match the actual implementation on an underwater robot platform. Furthermore, the function names should be sufficiently descriptive so that large language models can follow them. Robotics, as a mature field, already has a large number of open-source libraries available for basic functions in perception and motion domains (such as object detection and segmentation, mapping, motion planning, control, and grasping).

[0025] Specific embodiment 2 of the present invention creates a hint for a large language model, describing the objective while also identifying the high-level function libraries allowed in the library. This hint may also include constraint information or how the large language model should organize its response. If properly specified in the hint, the large language model can use predefined functions for robot reasoning and execution. An important hint design requirement is that the names of all application programming interfaces (APIs) must describe the overall functional behavior. Clear interface names are essential for the large language model to infer the functional connections between APIs and generate the required results for the problem.

[0026] In one embodiment, high-level functions can be defined as wrappers for the actual implementation. For example, a function named `detect_object(object_name)` can be internally linked to an OpenCV function or computer vision model, while functions like `move_to(x,y,z)` can internally call motion planning and obstacle avoidance pipelines and provide the underwater robot with the corresponding low-level motor instructions. Depending on the specific circumstances, we recommend interpreting the functionality of the application programming interface and, where necessary, decomposing it into sub-components with explicit inputs and outputs.

[0027] The prompts provided in specific embodiments of the present invention also include symbolic labels and contextual descriptions, which include conditional constraints, the environment in which the task objective occurs, the robot's current state, and solution examples.

[0028] Larger language models can generate more accurate responses. In addition to the robot API, good contextual descriptions should also include: Constraints and Requirements: Specify constraints or requirements related to the task. If the task involves moving objects, you can specify the weight, size, and shape of the objects to be moved.

[0029] Environment: Describe the environment in which the robot's mission takes place. For example, if the mission is underwater cruising, you can describe any obstacles or hazards that need to be avoided.

[0030] Current state: Describes the current state of the robot system. For example, if the task is to spot an object, you can describe the current position and orientation of the robot and the object.

[0031] Purpose and Objectives: State the purpose and objectives of the mission. If the mission is an underwater patrol, you can specify the target location and the required arrival time.

[0032] Solution examples: Demonstrate how to solve similar tasks as a means of guiding strategies for large-scale language model solutions.

[0033] Users can evaluate the code output of a large language model at any time through direct analysis or simulation, as provided in specific embodiments of the present invention, and provide feedback to the large language model regarding the quality and safety of the output code. Even well-designed prompts may not contain all the necessary information to solve the problem, or in some cases the large language model may be unable to generate the correct response with zero samples. In these cases, we have found that a simple and effective strategy for users is to send additional instructions to the large language model in a chat format describing the problem and let it correct itself.

[0034] On the other hand, in terms of system structure, the device of the present invention comprises three parts: an underwater sensor module, a computation and inference module, and an execution control module. The underwater sensor module is used to acquire environmental information; the computation and inference module runs a multimodal large model through an embedded computing platform or cloud computing resources to complete environmental understanding and task decision-making; the execution control module is responsible for converting the instructions generated by the model into physical actions to drive the robot to complete the predetermined task. Through this method and device, the underwater embodied intelligent agent can maintain a high degree of autonomy and adaptability in complex underwater environments, significantly improving the efficiency and accuracy of task execution.

Claims

1. An underwater control system based on a large multimodal neural network model, characterized in that, include: Underwater robots are used to collect environmental data and decompose received functional control commands into action execution signals, and complete the set task objectives based on the action execution signals; A large multimodal neural network model is used to generate prompt information based on the environmental data, underwater domain knowledge and task objectives sent by the underwater robot. The prompt information is used to extract the predictive function sequence information that can achieve the task objectives from the robot's function library, and the function sequence information is converted into functional control instructions.

2. The underwater control system based on a large multimodal neural network model according to claim 1, characterized in that, The method for obtaining the large model of the multimodal neural network includes: The large model is trained based on general corpora and multimodal data, enabling it to have semantic understanding capabilities for natural language, images, and time-series signals. Based on the augmented dataset, the large model trained on the base model is retrained to obtain a large multimodal neural network model, including: An enhanced dataset is constructed based on corresponding samples of sonar images and manually annotated data, simulation-generated datasets, and historical logs of underwater navigation and operation tasks; After multimodal alignment of the augmented dataset, training samples are obtained. The training samples are then input into the large model after basic training to obtain the predicted function sequence information. A loss function is constructed based on the predicted function sequence information and the true function sequence information. Based on different task objectives, a contrastive learning method is used to retrain the large model based on the task-oriented loss function to obtain a multimodal neural network large model.

3. The underwater control system based on a large multimodal neural network model according to claim 1, characterized in that, The underwater robot includes: A multimodal sensor is used to acquire environmental data, including sonar images, water flow velocity, water pressure, water temperature, and terrain structure. The control system decomposes the received functional control commands into action execution signals, which include motor speed signals, servo motor angle signals, and robotic arm grasping posture signals. The system includes a drive thruster, a servo motor, and a robotic arm. These components are used to receive motor speed signals, servo motor angle signals, and robotic arm grasping posture signals, respectively, to facilitate movement, obstacle avoidance, detection, and sample collection operations.

4. The underwater control system based on a large multimodal neural network model according to claim 3, characterized in that, The underwater robot also includes a preprocessing module, which converts the feature vector into a uniform format through filtering, denoising, and normalization steps.

5. The underwater control system based on a large multimodal neural network model according to claim 1, characterized in that, The underwater domain knowledge includes basic underwater navigation knowledge, marine operation standards, common terrain risks, and operational experience information.

6. The underwater control system based on a large multimodal neural network model according to claim 1, characterized in that, The prompts also include symbolic labels, conditional constraints, the environment in which the task objective occurs, the robot's current state, and solution examples.

7. The underwater control system based on a large multimodal neural network model according to claim 1, characterized in that, The robot function library includes multiple application programming interfaces (APIs) built from function functions. Each API is given a name that describes the overall functional behavior, enabling the multimodal neural network model to infer the functional connections between APIs and thus obtain the predicted function sequence information.

8. The underwater control system based on a large multimodal neural network model according to claim 1, characterized in that, The robot function library includes multiple function functions that can realize basic functions in the fields of perception and motion. The function names of the function functions need to be able to describe the corresponding functions.

Images