Unmanned ship six-degree-of-freedom prediction method and system based on visual analysis

Abstract

The invention provides an unmanned ship six-degree-of-freedom prediction method and system based on visual analysis, and relates to the field of automatic control of unmanned ships, and the method comprises the steps: obtaining the multi-modal data of an unmanned ship through collecting the visual data, inertial measurement unit data and historical navigation data of the unmanned ship; fusing and preprocessing the obtained multi-modal data, performing feature extraction on visual data, performing time sequence analysis on inertial measurement unit data and historical navigation data, performing position and attitude prediction by using a multi-task learning framework, and constructing a six-degree-of-freedom prediction model of the unmanned ship; adjusting parameters and a data fusion strategy of the six-degree-of-freedom prediction model according to real-time environment data, and integrating multi-modal data processing, the six-degree-of-freedom prediction model and a self-adaptive real-time correction mechanism into an unmanned ship control system so as to predict the six-degree-of-freedom state of the unmanned ship and perform accurate navigation control. And high-precision, real-time and intelligent attitude and position prediction of the unmanned ship is realized.

Description

Technical Field

[0001] The present invention relates to the field of automated control of unmanned ships, and in particular to a six-degree-of-freedom prediction method and system for unmanned ships based on visual analysis. Background Art

[0002] In the actual application scenarios of unmanned ships, the water environment is usually highly uncertain (such as random changes in waves, wind speed and obstacles), which puts higher requirements on the six-degree-of-freedom (6DOF) prediction method of unmanned ships. Most of the existing six-degree-of-freedom prediction methods of unmanned ships rely on single sensor data (such as visual or inertial sensors), and the navigation accuracy is easily affected in complex water environments.

[0003] Although some systems have introduced multi-sensor fusion technology, there are still deficiencies in the processing and real-time performance of multi-modal data. In addition, resource-constrained unmanned ship equipment has difficulty running complex high-precision prediction algorithms, which limits its promotion in practical applications. Therefore, an innovative method based on multi-modal data fusion and AI prediction model is urgently needed to improve the accuracy and adaptability of six-degree-of-freedom state prediction. Summary of the invention

[0004] In view of this, the purpose of the present invention is to propose a six-degree-of-freedom prediction method and system for unmanned ships based on visual analysis, aiming to improve the navigation performance of unmanned ships in complex water environments, and to achieve high-precision, real-time and intelligent posture and position prediction of unmanned ships through multimodal data fusion, AI-based six-degree-of-freedom prediction model and adaptive real-time correction strategy.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] Based on the above objectives, in a first aspect, the present invention provides a six-degree-of-freedom prediction method for an unmanned ship based on visual analysis, comprising the following steps:

[0007] Collect visual data, inertial measurement unit (IMU) data and historical navigation data of the unmanned ship to obtain multimodal data of the unmanned ship;

[0008] The acquired multimodal data is fused and preprocessed, and features are extracted from the visual data. Time series analysis is performed on the IMU data and historical navigation data. Position and attitude prediction are performed using a multi-task learning framework, and a six-degree-of-freedom prediction model for the unmanned ship is constructed.

[0009] Adjust the parameters and data fusion strategy of the six-degree-of-freedom prediction model according to real-time environmental data, and correct the prediction results through an adaptive real-time correction mechanism to optimize the navigation accuracy and path planning accuracy of the unmanned ship;

[0010] Multimodal data processing, six-degree-of-freedom prediction model and adaptive real-time correction mechanism are integrated into the unmanned ship control system to predict the six-degree-of-freedom state of the unmanned ship and perform precise navigation control.

[0011] As a further solution of the present invention, the visual data collects images or point cloud data of the external environment in real time through the visual sensor carried on the unmanned ship, the IMU data collects dynamic information of the acceleration, angular velocity and direction of the hull through the inertial sensor, and the historical navigation data is used to provide prior knowledge support by collecting historical navigation data of the unmanned ship.

[0012] As a further solution of the present invention, the visual data includes images or point cloud data acquired by a camera, lidar or other visual sensors, the IMU data includes acceleration, angular velocity and direction data of the unmanned ship; the historical navigation data includes the navigation track, speed, heading, wind speed and wave height information of the unmanned ship.

[0013] As a further solution of the present invention, a convolutional neural network (CNN) is used to extract features from the visual data to identify and capture key features of the environment including obstacles, tracks and water surface information; a long short-term memory network (LSTM) is used to perform time series analysis on the IMU data and historical navigation data to capture the dynamic changes of the unmanned ship and trends in future states.

[0014] As a further solution of the present invention, when the acquired multimodal data is fused and preprocessed, the visual features, IMU features and historical data features are spliced ​​to generate a multimodal feature vector. Different weights are assigned to each data source through weighted fusion. The weights are dynamically adjusted according to the real-time environmental data. The multimodal fusion feature vector F fuse The calculation method is as follows, using the weight α v , α i and α h Weighted superposition visual feature F vis 、IMU feature F imu and historical characteristics F hist :

[0015] F fuse =α v ·F vis +α i ·F imu +α h ·F hist

[0016] Among them, α v , α i and α h Represent the weights of vision, IMU and historical data respectively.

[0017] As a further solution of the present invention, the current environmental state is classified based on the environmental data collected in real time, and the weight of the IMU data is increased according to a preset rule to maintain posture stability; wherein:

[0018] When the light intensity is lower than the threshold L min When α is reduced, the weight of visual data is reduced. v , the calculation formula is:

[0019]

[0020] When the wave height exceeds the threshold H max When , increase the weight of IMU data α i :

[0021] α i =α i +β·(HH max )

[0022] When the wind speed exceeds the set value W max When , reduce the historical data weight α h , enhance the impact of real-time data on predictions:

[0023]

[0024] As a further solution of the present invention, the multi-task learning framework is also used to divide the six-degree-of-freedom prediction task into two parts: position prediction and attitude prediction, predict the six-degree-of-freedom state of the unmanned ship in real time, and output estimated values ​​of position (x, y, z) and attitude (roll angle, pitch angle, yaw angle) respectively; including:

[0025] Position prediction: Estimate the current position of the unmanned ship in the water through the x, y, z coordinates output by the model;

[0026] Attitude prediction: Estimate the attitude state of the unmanned ship through the roll angle, pitch angle, and yaw angle output by the model.

[0027] As a further solution of the present invention, the six-degree-of-freedom prediction method for unmanned ships based on visual analysis also includes synchronously marking the six-degree-of-freedom label of each data through GPS and attitude sensor, including position (x, y, z) and attitude (roll angle, pitch angle, yaw angle), and each data records a set of position and attitude parameters, wherein obstacles in the visual data are marked with the position, shape and size information of the obstacles to identify surface obstacles; historical navigation data are segmented by trajectory and marked with path labels to identify the target position, path curvature and navigation difficulty of each trajectory.

[0028] As a further scheme of the present invention, in the constructed six-degree-of-freedom prediction model of the unmanned ship, the visual data is input in the form of RGB images, each frame of the image is 224x224 pixels in size, the data format is a four-dimensional tensor, the data type is float32, and all pixel values ​​are normalized to the range of [0,1]; the IMU data contains acceleration and angular velocity, which are three-dimensional vectors respectively, representing dynamic changes on the x, y, and z axes. The data format of the IMU data is a two-dimensional tensor, and the data type is float32; the data input format of the historical data is a two-dimensional tensor, and the data type is float32.

[0029] As a further scheme of the present invention, when the six-degree-of-freedom prediction model of the unmanned ship is constructed to perform position prediction output, the six-degree-of-freedom position prediction output is a three-dimensional vector, representing the coordinate prediction values ​​of the unmanned ship in the x, y, and z directions; the data format is a two-dimensional tensor, defined as [batch_size,3], where each sample contains three coordinate parameters, the data type is float32, and the output value is usually not normalized.

[0030] As a further solution of the present invention, when the six-degree-of-freedom prediction model of the unmanned ship is constructed to perform attitude prediction output, the six-degree-of-freedom attitude prediction output is a three-dimensional vector, representing the roll angle, pitch angle and yaw angle of the unmanned ship, defined as [roll, pitch, yaw]; the data format is a two-dimensional tensor, defined as [batch_size, 3], each sample contains three angle parameters, the data type is float32, and the output angle value is usually expressed in radians.

[0031] As a further solution of the present invention, the six-degree-of-freedom prediction model of the unmanned ship is constructed. In the six-degree-of-freedom prediction stage, when predicting the position and attitude, the six-degree-of-freedom prediction model outputs the position and attitude, that is, [x, y, z, roll, pitch, yaw], and adopts a multi-task loss function. The position prediction uses the mean square error (MSE) loss L position , pose prediction uses cosine similarity loss L attitude ,in:

[0032]

[0033] The total loss function L total for:

[0034] L total =λ1·L position +λ2·L attitude

[0035] Among them, λ1 and λ2 are loss weights.

[0036] As a further solution of the present invention, by weighted fusion, each data source is assigned different weights to perform environmental perception and weight adjustment, collect environmental information including light intensity, wave height, obstacle distance in real time, and calculate weight adjustment parameters; set the environmental impact factor E to dynamically adjust the fusion weight of vision, IMU and historical data. The formula is as follows

[0037] E=α·L+β·W+γ·D

[0038] Among them, L is the light intensity, W is the wave height, D is the distance to the nearest obstacle, and α, β, and γ are the corresponding adjustment coefficients, which are determined based on model experiments;

[0039] The weight update mechanism is to update the visual data weight α according to the value of the environmental factor E. v 、IMU data weight α i and historical data weight α h Dynamic update, if the visual data has a greater impact under low light conditions, the weight adjustment formula is:

[0040]

[0041] When the light intensity L is lower than the threshold L threshold When the visual data weight α v Reduce, while the IMU data weight α i Increase to ensure overall prediction accuracy.

[0042] In a second aspect, the present invention provides a six-degree-of-freedom prediction system for an unmanned ship based on visual analysis, comprising:

[0043] Data acquisition module: used to collect the visual data, inertial measurement unit (IMU) data and historical navigation data of the unmanned ship to obtain the multi-modal data of the unmanned ship;

[0044] A model building module is used to fuse and preprocess the acquired multimodal data, extract features from visual data, perform time series analysis on the IMU data and historical navigation data, use a multi-task learning framework to predict position and attitude, and build a six-degree-of-freedom prediction model for the unmanned ship;

[0045] The model optimization module is used to adjust the parameters and data fusion strategy of the six-degree-of-freedom prediction model according to real-time environmental data, and to correct the prediction results through an adaptive real-time correction mechanism to optimize the navigation accuracy and path planning accuracy of the unmanned ship;

[0046] The integrated deployment module is used to integrate multimodal data processing, six-degree-of-freedom prediction model and adaptive real-time correction mechanism into the unmanned ship control system to predict the six-degree-of-freedom state of the unmanned ship and perform precise navigation control.

[0047] In another aspect of the present invention, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, any one of the above-mentioned six-degree-of-freedom prediction methods for unmanned ships based on visual analysis according to the present invention is executed.

[0048] In another aspect of the present invention, a computer-readable storage medium is provided, storing computer program instructions, which, when executed, implement any of the above-mentioned six-degree-of-freedom prediction methods for unmanned ships based on visual analysis according to the present invention.

[0049] Compared with the prior art, the six-degree-of-freedom prediction method and system for unmanned ships based on visual analysis proposed in the present invention have the following beneficial effects:

[0050] 1. Improved the accuracy and safety of unmanned ship navigation. By fusing visual data with inertial measurement unit (IMU) data and historical navigation data, the six degrees of freedom (position, velocity, acceleration, pitch angle, roll angle, yaw angle) of the unmanned ship can be estimated more accurately. This provides accurate attitude and motion state prediction for unmanned ships in complex waters, effectively improving the navigation accuracy of unmanned ships, thereby greatly improving navigation safety.

[0051] 2. Enhanced adaptability and robustness. The method of the present invention can cope with complex and dynamic water environments, and realize dynamic prediction of six degrees of freedom through real-time visual analysis and data fusion under different sea conditions, weather, waterways, etc. Especially when encountering interference from external factors such as water surface fluctuations and wind speed changes, the model based on deep learning can adaptively adjust to maintain the accuracy and stability of the prediction, ensuring the robustness of the unmanned ship in an uncertain environment.

[0052] 3. Improved autonomous navigation capabilities of unmanned ships. By acquiring and processing visual data, IMU data, and historical navigation data in real time, the method of the present invention can achieve autonomous navigation and decision support for unmanned ships in different environments. Especially in remote or signal-free areas, unmanned ships can independently complete navigation path planning and obstacle avoidance, reducing dependence on manual operations and external navigation systems, and significantly improving the autonomy of unmanned ships.

[0053] 4. Improved prediction accuracy through data fusion. The method of the present invention maximizes the complementarity of various sensor data through multimodal data fusion, combined with visual perception and inertial navigation information. Visual data provides rich environmental information, while IMU data supplements the motion information of the hull, and historical navigation data further provides a time series pattern of navigation status. The combination of the three significantly improves the accuracy of six-degree-of-freedom prediction.

[0054] 5. Optimized the performance of the unmanned ship control system. By accurately predicting the six-degree-of-freedom state of the unmanned ship, the method of the present invention can provide accurate attitude and motion feedback for the control system of the unmanned ship, optimize the control strategy, and improve the control accuracy. Real-time six-degree-of-freedom prediction can provide data support for path planning, heading adjustment, speed control, etc., ensuring that the unmanned ship travels efficiently according to the predetermined target.

[0055] 6. Enhanced adaptability to complex water environments: The method of the present invention is particularly suitable for applications in complex water environments, such as narrow waterways, large water surface fluctuations, or dense obstacles. Visual analysis can identify factors such as obstacles on the water surface and changing water flow directions, while the deep learning model can adaptively adjust the motion prediction of the hull according to changes in the environment, helping unmanned ships to better cope with complex marine or inland water environments.

[0056] In summary, the unmanned ship six-degree-of-freedom prediction method and system based on visual analysis of the present invention, through multimodal data fusion and advanced deep learning technology, not only improves the navigation accuracy and safety of unmanned ships in complex environments, but also promotes the development of unmanned ship technology towards a more intelligent, autonomous and reliable direction, and has significant application value and broad market prospects.

[0057] These and other aspects of the present application will be more concise and understandable in the following description of the embodiments. It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the exemplary embodiments or related technical descriptions. The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0059] Figure 1 The present invention is a flowchart of a six-degree-of-freedom prediction method for an unmanned ship based on visual analysis according to an embodiment of the present invention.

[0060] Figure 2 It is a structural block diagram of an unmanned ship six-degree-of-freedom prediction system based on visual analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are intended to distinguish two non-identical entities or non-identical parameters with the same name. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, other steps or units inherent to a process, method, system, product or device that includes a series of steps or units.

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0066] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0067] In view of the fact that the navigation accuracy of the existing six-degree-of-freedom prediction method for unmanned ships is easily affected in complex water environments, multi-sensor fusion technology has been introduced, but there are still deficiencies in the processing and real-time performance of multi-modal data. The present invention proposes a six-degree-of-freedom prediction method and system for unmanned ships based on visual analysis, aiming to improve the navigation performance of unmanned ships in complex water environments, and realizes high-precision, real-time and intelligent posture and position prediction of unmanned ships through multi-modal data fusion, AI-based six-degree-of-freedom prediction model and adaptive real-time correction strategy.

[0068] See also Figure 1 As shown, an embodiment of the present invention provides a six-degree-of-freedom prediction method for an unmanned ship based on visual analysis, the method comprising the following steps:

[0069] Step S10: collect visual data, inertial measurement unit (IMU) data and historical navigation data of the unmanned ship to obtain multimodal data of the unmanned ship.

[0070] In this step, the visual data collects images or point cloud data of the external environment in real time through the visual sensors carried on the unmanned ship, the IMU data collects dynamic information of the acceleration, angular velocity and direction of the hull through inertial sensors, and the historical navigation data is used to provide prior knowledge support by collecting historical navigation data of the unmanned ship.

[0071] Among them, the visual data includes images or point cloud data obtained by cameras, lidar or other visual sensors, the IMU data includes the acceleration, angular velocity and direction data of the unmanned ship; the historical navigation data includes the navigation track, speed, heading, wind speed and wave height information of the unmanned ship.

[0072] In this embodiment, data collected by various sensors on the unmanned ship are integrated, including visual data, inertial measurement unit (IMU) data, and historical navigation data.

[0073] Step S20: fuse and preprocess the acquired multimodal data, extract features from the visual data, perform time series analysis on the IMU data and historical navigation data, use a multi-task learning framework to perform position and attitude prediction, and build a six-degree-of-freedom prediction model for the unmanned ship.

[0074] In this embodiment, during multimodal data collection and preprocessing, the prediction accuracy and robustness of the six-degree-of-freedom state are improved by combining multiple sensor data on the unmanned ship (such as vision, IMU, historical data, etc.). The first step is data collection and preprocessing. Among them:

[0075] Visual data acquisition: The unmanned ship is equipped with a camera to collect images and videos of the surrounding environment. These visual data are used to capture obstacles, waterways, and surface features in the external environment. The processing steps include denoising, grayscale, scale normalization, and other operations to enhance the quality and consistency of the image.

[0076] IMU data acquisition: The inertial measurement unit (IMU) captures the acceleration and angular velocity of the ship in real time. These data are used to reflect the attitude changes of the unmanned ship, such as roll, pitch, yaw, etc. The IMU data is filtered (such as Kalman filtering or low-pass filtering) to remove high-frequency noise and improve data stability.

[0077] Historical data collection: The unmanned ship’s past navigation data, such as historical navigation tracks, attitude angles, speeds, etc., are used as references for model training and calibration. The key features of the navigation track are extracted through cluster analysis or regression analysis.

[0078] Data preprocessing: Vision, IMU, and historical data need to be uniformly normalized and filtered to ensure data quality and format consistency. In particular, in visual data processing, convolutional neural networks (CNNs) process image data through feature extraction, while IMU and historical data are processed as time series.

[0079] In this step, a convolutional neural network (CNN) is used to extract features from the visual data to identify and capture key features of the environment, including obstacles, tracks, and water surface information. A long short-term memory network (LSTM) is used to perform time series analysis on the IMU data and historical navigation data to capture the dynamic changes of the unmanned ship and trends in its future state.

[0080] During feature extraction, visual feature extraction uses convolutional neural networks (CNNs) to process image data and extract high-dimensional features in the environment, such as the location, shape, texture, and channel information of obstacles. CNNs can automatically learn these features and convert them into feature vectors. MU data feature extraction uses long short-term memory networks (LSTMs) or recurrent neural networks (RNNs) to process IMU data and extract the dynamic motion characteristics of the hull. Historical data feature extraction uses clustering or regression analysis to extract the motion patterns in past navigation data and provide prior knowledge support for the model.

[0081] In this embodiment, a multimodal data fusion strategy is proposed. The fusion process not only utilizes the advantages of various sensors, but also dynamically adjusts different data sources through weighted averaging, adaptive updating and other methods to make the predicted input data more robust and accurate. In the fusion process, the visual data provides the external environment information of the unmanned ship, the IMU data captures the dynamic motion characteristics of the hull, and the historical navigation data provides prior knowledge support for the model. Through the joint use of multimodal data, the present invention can effectively improve the prediction accuracy of the six-degree-of-freedom state of the unmanned ship in a complex dynamic environment.

[0082] Among them, visual features, IMU features and historical data features are spliced ​​to generate a multimodal feature vector. These spliced ​​feature vectors are further processed through a fully connected layer (multi-layer perceptron, MLP) to capture the correlation between different data sources. When fusing and preprocessing the acquired multimodal data, visual features, IMU features and historical data features are spliced ​​to generate a multimodal feature vector. Different weights are given to each data source through weighted fusion. The weights are dynamically adjusted according to real-time environmental data. The multimodal fusion feature vector F fuse The calculation method is as follows, using the weight α v , α i and α h Weighted superposition visual feature F vis 、IMU feature Fimu and historical characteristics F h ist :

[0083] F fuse =α v ·F vis +α i ·F imu +α h ·F h ist

[0084] Among them, α v , α i and α h Represent the weights of vision, IMU and historical data respectively.

[0085] Based on the real-time collected environmental data, the current environmental state is classified and the weight of the IMU data is increased according to the preset rules to maintain the posture stability; among which:

[0086] When the light intensity is lower than the threshold L min When α is reduced, the weight of visual data is reduced. v , the calculation formula is:

[0087]

[0088] When the wave height exceeds the threshold H max When , increase the weight of IMU data α i :

[0089] α i =α i +β·(HH max )

[0090] When the wind speed exceeds the set value W max When , reduce the historical data weight α h , enhance the impact of real-time data on predictions:

[0091]

[0092] In this embodiment, the multi-task learning framework is also used to divide the six-degree-of-freedom prediction task into two parts: position prediction and attitude prediction, predict the six-degree-of-freedom state of the unmanned ship in real time, and output the estimated values ​​of position (x, y, z) and attitude (roll angle, pitch angle, yaw angle) respectively; including:

[0093] Position prediction: Estimate the current position of the unmanned ship in the water through the x, y, z coordinates output by the model;

[0094] Attitude prediction: Estimate the attitude state of the unmanned ship through the roll angle, pitch angle, and yaw angle output by the model.

[0095] In this embodiment, the unmanned ship six-degree-of-freedom prediction method based on visual analysis also includes synchronously marking the six-degree-of-freedom label of each data through GPS and attitude sensor, including position (x, y, z) and attitude (roll angle, pitch angle, yaw angle), and each data records a set of position and attitude parameters, wherein the obstacles in the visual data are marked with the position, shape and size information of the obstacles to identify surface obstacles; the historical navigation data is segmented by trajectory and marked with path labels to identify the target position, path curvature and navigation difficulty of each trajectory.

[0096] Among them, the visual data in the constructed six-degree-of-freedom prediction model of the unmanned ship is input in the form of RGB images, each frame of the image is 224x224 pixels in size, the data format is a four-dimensional tensor, the data type is float32, and all pixel values ​​are normalized to the range of [0,1]; the IMU data contains acceleration and angular velocity, which are three-dimensional vectors respectively, representing dynamic changes on the x, y, and z axes. The data format of the IMU data is a two-dimensional tensor with a data type of float32; the data input format of the historical data is a two-dimensional tensor with a data type of float32.

[0097] Among them, when the six-degree-of-freedom prediction model of the constructed unmanned ship performs position prediction output, the six-degree-of-freedom position prediction output is a three-dimensional vector, which represents the coordinate prediction value of the unmanned ship in the x, y, and z directions; the data format is a two-dimensional tensor, defined as [batch_size,3], where each sample contains three coordinate parameters, the data type is float32, and the output value is usually not normalized.

[0098] In this embodiment, when the six-degree-of-freedom prediction model of the constructed unmanned ship performs attitude prediction output, the six-degree-of-freedom attitude prediction output is a three-dimensional vector, representing the roll angle, pitch angle and yaw angle of the unmanned ship, defined as [roll, pitch, yaw]; the data format is a two-dimensional tensor, defined as [batch_size, 3], each sample contains three angle parameters, the data type is float32, and the output angle value is usually expressed in radians.

[0099] The six-degree-of-freedom prediction model of the unmanned ship is constructed. In the six-degree-of-freedom prediction stage, when predicting the position and attitude, the six-degree-of-freedom prediction model outputs the position and attitude, that is, [x, y, z, roll, pitch, yaw]. The multi-task loss function is used, and the position prediction uses the mean square error (MSE) loss L position , pose prediction uses cosine similarity loss L attitude ,in:

[0100]

[0101] The total loss function L total for:

[0102] L total =λ1·L position +λ2·L attitude

[0103] Among them, λ1 and λ2 are loss weights.

[0104] Step S30: adjust the parameters and data fusion strategy of the six-degree-of-freedom prediction model according to the real-time environmental data, correct the prediction results through an adaptive real-time correction mechanism, and optimize the navigation accuracy and path planning accuracy of the unmanned ship.

[0105] In this step, the environment is perceived and the weight is adjusted by assigning different weights to each data source through weighted fusion. Environmental information including light intensity, wave height, and obstacle distance is collected in real time, and the weight adjustment parameters are calculated. The environmental impact factor E is set to dynamically adjust the fusion weight of vision, IMU, and historical data. The formula is as follows

[0106] E=α·L+β·W+γ·D

[0107] Among them, L is the light intensity, W is the wave height, D is the distance to the nearest obstacle, and α, β, and γ are the corresponding adjustment coefficients, which are determined based on model experiments;

[0108] The weight update mechanism is to update the visual data weight α according to the value of the environmental factor E. v 、IMU data weight α i and historical data weight α h Dynamic update, if the visual data has a greater impact under low light conditions, the weight adjustment formula is:

[0109]

[0110] When the light intensity L is lower than the threshold L threshold When the visual data weight α v Reduce, while the IMU data weight α i Increase to ensure overall prediction accuracy.

[0111] The dynamic weight adjustment mechanism includes:

[0112] Environmental perception and weight adjustment: Collect environmental information in real time (such as light intensity, wave height, obstacle distance, etc.) and calculate weight adjustment parameters. Set the environmental impact factor E to dynamically adjust the fusion weight of vision, IMU and historical data. The formula is as follows:

[0113] E=α·L+β·W+γ·D

[0114] Among them, L is the light intensity, W is the wave height, D is the distance to the nearest obstacle, and α, β, and γ are the corresponding adjustment coefficients, which are determined based on model experiments.

[0115] Weight update mechanism: According to the value of the environmental factor E, the visual data weight α v 、IMU data weight α i and historical data weight α h Dynamic update. Assuming that visual data has a greater impact in low-light conditions, the weight adjustment formula is:

[0116]

[0117] When the light intensity L is lower than the threshold L threshold When the visual data weight α v Reduce, while the IMU data weight α i Increase to ensure overall prediction accuracy.

[0118] When calculating weight update, initialize the weight α v =0.4, α i =0.3, α h =0.3; when the environmental factor E changes, the weight is dynamically updated by the following formula:

[0119]

[0120] Where ΔE=E current -E base , E base This is the baseline value under normal conditions.

[0121] The updated weights need to be normalized to ensure that the sum is 1, that is:

[0122] α v +α i +α h =1.

[0123] By introducing an adaptive real-time correction mechanism, the system continuously collects environmental data (such as wind speed, wave height, obstacle location, etc.), adjusts model parameters and data fusion strategies in real time, and enhances the navigation performance of the system in complex environments. The correction process uses a combination of rule-driven and AI. First, a preliminary correction is made based on real-time environmental data, and then the correction result is further optimized using the AI ​​algorithm, so that the unmanned ship can respond quickly when the environment changes drastically and dynamically adjust the six-degree-of-freedom state. This adaptive correction mechanism significantly improves the navigation accuracy and path planning capabilities of the unmanned ship, especially in the case of sudden environmental changes (such as sudden obstacles or bad weather).

[0124] Step S40: Integrate multimodal data processing, six-degree-of-freedom prediction model and adaptive real-time correction mechanism into the unmanned ship control system to predict the six-degree-of-freedom state of the unmanned ship and perform precise navigation control.

[0125] In this step, when the weather changes suddenly, the adjustment strategy is implemented. Weather monitoring and early warning mechanism: weather parameters are monitored in real time through onboard meteorological sensors (such as temperature, humidity, air pressure and wind speed sensors) and water environment sensors (such as water temperature and wave height sensors). Once signs of drastic weather changes (such as a sharp increase in wind speed and a sudden drop in air pressure) are detected, the system automatically enters early warning mode.

[0126] Weight adjustment strategy under sudden weather changes:

[0127] Rule 1: When the wind speed exceeds the preset threshold (such as 15 m / s), increase the weight of IMU data and reduce the weight of visual data to ensure more stable attitude and speed adjustments. The specific adjustment formula is:

[0128] α i =α i +δ·(WW threshold )

[0129]

[0130] Where W is the current wind speed, W threshold is the set wind speed threshold.

[0131] Rule 2: When the air pressure drops by more than 20hPa, the system determines that rain or storm may occur, and reduces the weight of historical data, enhancing the responsiveness of real-time data to the current environment to cope with sudden changes in the environment:

[0132]

[0133] Where P is the current air pressure, P normal It is the reference value of normal air pressure.

[0134] Rule 3: When the wave height exceeds the set value (such as 3 meters), increase the weight of IMU data and introduce multi-frame image weighted processing to weighted average the frame information in the visual data to reduce the impact of single-frame errors on judgment.

[0135] In the six-degree-of-freedom prediction method of an unmanned ship based on visual analysis of the present invention, the preprocessing of image data includes:

[0136] Denoising: First, apply a Gaussian filter to the collected image data for denoising, using a Gaussian kernel (such as a 3x3 or 5x5 matrix) with a standard deviation of 1. The specific operation is to convolve the image under the Gaussian kernel to remove noise interference while retaining the main edge features of the image to reduce interference caused by changes in ambient lighting or water surface reflections.

[0137] Normalization: The image data is normalized to the range of [0,1]. The normalization steps are as follows: subtract the average grayscale value of the entire image from each pixel value, and then divide it by the standard deviation of the image pixel value to ensure that the values ​​are evenly distributed when the data is input into the model, reducing the impact of inconsistent brightness.

[0138] Scaling: Use bilinear interpolation to resize images to the standard size required by the model (such as 224x224 pixels) to ensure that images of different scenes have the same resolution. By maintaining a uniform resolution, it helps the stability of the model during the feature extraction stage.

[0139] Time series smoothing: Time series smoothing is performed on continuous frame images. The specific method is to fuse the pixel values ​​of each frame image with the pixel values ​​of the previous and next frames in a weighted average manner (for example, the weight of the previous frame is 0.2, the weight of the current frame is 0.6, and the weight of the next frame is 0.2) to reduce the inter-frame mutations caused by device jitter and improve the accuracy of feature extraction.

[0140] The preprocessing of IMU data includes applying a Kalman filter to remove random noise in the IMU data. Assuming that the acceleration and angular velocity changes are state variables, the processing is performed in two steps: prediction and update:

[0141] Prediction step: Predict the current state based on the state estimate and control input at the previous moment.

[0142] Update step: Adjust the state prediction value in combination with the actual measurement value at the current moment to obtain the filtered acceleration and angular velocity values. Kalman filtering can significantly reduce noise interference and make IMU data smoother, which is particularly suitable for attitude estimation.

[0143] Smooth the high-frequency noise of IMU data. Use a Butterworth low-pass filter with a cutoff frequency of 5Hz to filter out the high-frequency components in the acceleration and angular velocity data, retaining only the low-frequency part to ensure the stability of the data. The specific method is to determine the noise frequency range through frequency domain transformation, and then use the filtering formula to reduce the high-frequency components. For IMU data, perform sensor zero bias calibration and temperature compensation to avoid errors caused by temperature changes or sensor drift. For example, by regularly collecting zero input data, calculating the bias value and subtracting the bias from the actual data, the accuracy of the data can be improved.

[0144] For the preprocessing of historical navigation data, the key feature extraction uses K-means clustering to extract features from historical navigation trajectory data. First, the historical data is clustered according to features such as speed, attitude angle, and path curvature, and then the features of each cluster center point are extracted as the prior input of the model. This method can effectively summarize the laws of historical navigation and reduce data redundancy. For numerical features such as speed and attitude, they are scaled to the range of [0,1]. The normalization formula is: Among them, x represents the data feature value, ensuring that the scales of different data sources are consistent when fused to avoid a certain feature occupying too much weight. The autoregressive moving average model (ARIMA) is used to model and analyze the time series characteristics of historical trajectories, from which the changing trends of speed and posture are extracted to predict the movement law of the unmanned ship in the current environment and provide a reference for real-time navigation.

[0145] When processing data consistency, timestamp alignment is performed. In multi-sensor data synchronization, bilinear interpolation is used to align timestamps of different data sources to ensure the time consistency of visual, IMU and historical data. When converting formats, the processed data is uniformly converted into the tensor format required for model input. For example, visual data is a four-dimensional tensor of [batch, height, width, channels], and IMU and historical data are two-dimensional tensors of [batch, features], ensuring that the data can be seamlessly connected when input into the fusion model.

[0146] In this embodiment, when acquiring model training data and constructing a data set, the data sources include:

[0147] Visual data: collected by the high-definition camera on the unmanned ship in different environments, including open waters, dense obstacle areas, strong light and low light conditions, to cover various situations that the unmanned ship may encounter.

[0148] IMU data: The acceleration and angular velocity data collected in real time by the inertial measurement unit (IMU) on the unmanned ship ensures that the data can accurately reflect the motion posture of the hull.

[0149] Historical navigation data: The trajectory, speed, posture and other data of unmanned ships sailing in different waters are selected to construct a representative historical data set.

[0150] The data collection methods are:

[0151] Acquisition frequency: The acquisition frequency of visual data and IMU data is set to 10 Hz, that is, 10 frames per second, to ensure that the model can respond to water surface changes and attitude adjustments in real time.

[0152] Environmental conditions: Data collection covers weather conditions such as sunny days, cloudy days, and haze, and is collected under conditions of different wind speeds and wave heights to ensure that the data is widely representative.

[0153] Sample size: At least 5,000 data records are collected for model training, each of which contains a visual image frame, the corresponding IMU acceleration and angular velocity data, and the historical navigation trajectory. These data samples ensure that the training set has sufficient diversity to avoid model overfitting.

[0154] When labeling data and using annotation strategies, the position and attitude tags use GPS and attitude sensors to synchronously label each piece of data with six degrees of freedom (6DOF) tags, including position (x, y, z) and attitude (roll angle, pitch angle, yaw angle). Each piece of data records a set of position and attitude parameters. Obstacle labels are manually or automatically labeled based on object detection models such as YOLO to label obstacles in visual data, with information such as the position, shape, and size of the obstacles to help the model accurately identify surface obstacles. Path labels segment historical navigation data by trajectory and add path labels to identify the target position, path curvature, and navigation difficulty of each trajectory, so that the model can learn navigation modes in different environments.

[0155] The present invention also includes data enhancement, wherein visual data enhancement includes:

[0156] Brightness change: Adjust the brightness of the image (within ±30%) to simulate different lighting conditions to improve the model's adaptability to lighting changes.

[0157] Rotation and Cropping: Randomly rotate the image (±10°) and perform random cropping to increase the model’s perspective diversity and anti-interference ability.

[0158] Adding noise: Add random Gaussian noise to the image to simulate factors such as water surface reflection and sensor noise to enhance the model's anti-noise ability.

[0159] IMU data enhancement includes:

[0160] Data perturbation: Add a small Gaussian perturbation (mean 0, standard deviation 0.1) to the IMU data to simulate the jitter and measurement error in the real environment and ensure that the model is robust in different devices and environments.

[0161] Time offset: Apply a time series offset (±1 to 2 time steps) to the IMU data to simulate the effects of sensor delays or data synchronization errors and improve the temporal robustness of the model.

[0162] Historical data enhancements include:

[0163] Trajectory smoothing: Smoothing the original trajectory data through interpolation enhances the continuity of the data and enables the model to learn historical navigation patterns more smoothly.

[0164] Random path deformation: Randomly deform the trajectory data to make the navigation path slightly deviate from the original route, simulate the small route adjustments in real navigation, and improve the model's ability to cope with trajectory changes.

[0165] In this embodiment, in the output data format and content of the six-degree-of-freedom prediction model, the six-degree-of-freedom position prediction output is:

[0166] The position output is a three-dimensional vector, which represents the predicted coordinates of the unmanned ship in the x, y, and z directions.

[0167] The data format is a two-dimensional tensor, defined as [batch_size,3], where each sample contains three coordinate parameters.

[0168] The data type is float32 and the output value is usually unnormalized so that it can be directly used for subsequent navigation and control.

[0169] For example, the tensor format for a single position prediction output is [1,3], containing [x,y,z].

[0170] The six-degree-of-freedom posture prediction output is:

[0171] The attitude output is a three-dimensional vector, representing the roll angle, pitch angle, and yaw angle of the unmanned ship, defined as [roll, pitch, yaw].

[0172] The data format is a two-dimensional tensor, defined as [batch_size,3], and each sample contains three angle parameters.

[0173] The data type is float32, and the output angle value is usually expressed in radians to be consistent with the sensor data.

[0174] For example, the tensor format of a single pose prediction output is [1,3], containing [roll, pitch, yaw].

[0175] For example, the input data of the model consists of [visual data (224, 224, 3), IMU data (10, 6), historical data (50, 5)], and the total input format is [1, 224, 224, 3], [1, 10, 6], and [1, 50, 5]. The output data format of the model is position output [1, 3] and attitude output [1, 3].

[0176] In this embodiment, the six-degree-of-freedom prediction model also includes model pruning processing, which uses structured pruning to reduce model parameters by removing convolution kernels or neurons. The main steps are as follows:

[0177] Initial model training: First train the model on the complete dataset until convergence to obtain a baseline model.

[0178] Evaluate weight importance: Calculate the L1 norm of each convolution kernel. Assuming the weight matrix is ​​W, its L1 norm is:

[0179]

[0180] Convolution kernels whose weight norm is less than the threshold are considered unimportant and can be removed.

[0181] Stepwise pruning: Remove unimportant convolution kernels layer by layer according to the set pruning rate (such as 30%). For example, suppose a convolution layer initially has 64 convolution kernels, and only 45 convolution kernels are retained after pruning.

[0182] Fine-tuning after pruning: Fine-tune the model again after pruning to restore accuracy.

[0183] Effect and parameter settings:

[0184] When the pruning rate is set to 30%, the convolution kernel parameters are reduced by about 30% and the memory usage is saved by about 40%.

[0185] The fine-tuned model keeps the accuracy loss within 1% and maintains close prediction performance.

[0186] The unmanned ship six-degree-of-freedom prediction method based on visual analysis of the present invention can more accurately estimate the six degrees of freedom (position, speed, acceleration, pitch angle, roll angle, yaw angle) of the unmanned ship by fusing visual data with inertial measurement unit (IMU) data and historical navigation data. This provides accurate attitude and motion state prediction for the unmanned ship's navigation in complex waters, effectively improves the navigation accuracy of the unmanned ship, and thus greatly improves navigation safety.

[0187] The method of the present invention can cope with complex and dynamic water environments, and realize dynamic prediction of six degrees of freedom through real-time visual analysis and data fusion under different sea conditions, weather, waterways and other conditions. Especially when encountering interference from external factors such as water surface fluctuations and wind speed changes, the model based on deep learning can adaptively adjust to maintain the accuracy and stability of the prediction, and ensure the robustness of the unmanned ship in an uncertain environment. By acquiring and processing visual data, IMU data and historical navigation data in real time, the method of the present invention can realize autonomous navigation and decision support of unmanned ships in different environments. Especially in remote or signal-free areas, unmanned ships can independently complete navigation path planning and obstacle avoidance, reduce dependence on manual operations and external navigation systems, and significantly improve the autonomy of unmanned ships.

[0188] The method of the present invention utilizes the complementarity of various sensor data to the greatest extent by combining multimodal data fusion, visual perception and inertial navigation information. Visual data provides rich environmental information, while IMU data supplements the motion information of the hull, and historical navigation data further provides a time series pattern of the navigation state. The combination of the three significantly improves the accuracy of the six-degree-of-freedom prediction. By accurately predicting the six-degree-of-freedom state of the unmanned ship, the method of the present invention can provide accurate attitude and motion feedback for the control system of the unmanned ship, optimize the control strategy, and improve the control accuracy. Real-time six-degree-of-freedom prediction can provide data support for path planning, heading adjustment, speed control, etc., to ensure that the unmanned ship travels efficiently according to the predetermined goals.

[0189] The method of the present invention is particularly suitable for applications in complex water environments, such as narrow waterways, large water surface fluctuations, or dense obstacles. Visual analysis can identify factors such as obstacles on the water surface and changing water flow directions, while the deep learning model can adaptively adjust the motion prediction of the hull according to environmental changes, helping unmanned ships better cope with complex marine or inland water environments.

[0190] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0191] It should be understood that, although described in a certain order, these steps are not necessarily performed in sequence in the above order. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, a part of the steps of the present embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a part of the steps or stages in other steps or other steps.

[0192] In the second aspect of the embodiment of the present invention, see Figure 2 As shown, the present invention also provides a six-degree-of-freedom prediction system for an unmanned ship based on visual analysis, comprising:

[0193] Data acquisition module 100: used to collect visual data, inertial measurement unit (IMU) data and historical navigation data of the unmanned ship to obtain multi-modal data of the unmanned ship;

[0194] A model building module 200 is used to fuse and preprocess the acquired multimodal data, extract features from the visual data, perform time series analysis on the IMU data and historical navigation data, use a multi-task learning framework to predict the position and attitude respectively, and build a six-degree-of-freedom prediction model for the unmanned ship;

[0195] The model optimization module 300 is used to adjust the parameters and data fusion strategy of the six-degree-of-freedom prediction model according to the real-time environmental data, correct the prediction results through an adaptive real-time correction mechanism, and optimize the navigation accuracy and path planning accuracy of the unmanned ship;

[0196] The integrated deployment module 400 is used to integrate multimodal data processing, six-degree-of-freedom prediction model and adaptive real-time correction mechanism into the unmanned ship control system to predict the six-degree-of-freedom state of the unmanned ship and perform precise navigation control.

[0197] Through the above detailed steps, the unmanned ship six-degree-of-freedom prediction system based on visual analysis of the present invention is used to execute the steps of the unmanned ship six-degree-of-freedom prediction method based on visual analysis in the above embodiment, which will not be repeated here.

[0198] The unmanned ship six-degree-of-freedom prediction system based on visual analysis of the present invention not only improves the navigation accuracy and safety of unmanned ships in complex environments through multimodal data fusion and advanced deep learning technology, but also promotes the development of unmanned ship technology towards a more intelligent, autonomous and reliable direction. It has significant application value and broad market prospects.

[0199] The application scenarios of the unmanned ship six-degree-of-freedom prediction method and system based on visual analysis of the present invention are as follows:

[0200] Application scenario 1: Inspection tasks in marine aquaculture areas

[0201] Scenario description: Unmanned boats are used for automatic inspections in marine aquaculture areas. In this area, the water surface is relatively calm, but due to the dense aquaculture facilities, the obstacles in the environment are complex and dense. The unmanned boat needs to accurately predict its own position and attitude to achieve stable navigation and obstacle avoidance.

[0202] 6DOF prediction effect: The 6DOF model based on multimodal fusion controls the errors of position and posture within 0.2 meters and 2 degrees during the inspection process, ensuring high-precision navigation of the unmanned ship during inspections in densely populated areas and avoiding collisions.

[0203] Experimental results: In actual tasks, after using the 6DOF model, the inspection time of the unmanned boat in the aquaculture area was reduced by 15%, and the probability of obstacle collision was reduced to less than 0.1%.

[0204] Application scenario 2: Garbage cleaning in inland waters

[0205] Scenario description: Unmanned boats are used to clean up garbage in inland waters. The water flow in these waters varies greatly and is affected by wind speed and obstacles (such as bridge piers and ships). Unmanned boats need to adjust their posture and speed in real time to travel stably and clean up floating garbage.

[0206] 6DOF prediction effect: The real-time attitude error predicted by the model is kept within 1.5 degrees, and the position deviation is within 0.1 meter, ensuring the cleaning accuracy and stability of the unmanned boat in unstable water flow.

[0207] Simulation data: The simulation results in this environment show that after applying the 6DOF prediction model, the cleaning path deviation is reduced to within 0.05 meters and the cleaning efficiency is improved by 20%.

[0208] Experimental results or simulation data are being supplemented. Experimental settings: three environments, inland river, marine aquaculture area and open water, are selected to conduct unmanned ship navigation tests respectively. Test indicators include position error, attitude error, path deviation and cleaning / inspection efficiency.

[0209] Experimental data:

[0210] Position prediction error: In inland waters, the average position error of the 6DOF model is 0.15 meters; in marine aquaculture areas, it is 0.18 meters; and in open waters, it is 0.12 meters.

[0211] Attitude prediction error: Under different environments, the average attitude error of the 6DOF model is 1.2 degrees, and the maximum error is 2.5 degrees.

[0212] Improved efficiency: Compared with the traditional single-sensor model, the multimodal 6DOF model improves inspection efficiency by about 15%-20% and cleaning efficiency by 18%-22%.

[0213] The performance in actual application scenarios is as follows: In actual applications, the adaptability of the 6DOF model in complex environments is particularly outstanding. For example, in high-wave environments, the unmanned ship can smoothly adjust its attitude and maintain its course thanks to the adaptive correction strategy. Test results show that the 6DOF model effectively reduces the yaw caused by sudden changes in water flow and wind speed, allowing the unmanned ship to achieve a mission completion rate of more than 98% in complex environments.

[0214] According to a third aspect of an embodiment of the present invention, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method of any one of the above embodiments is implemented.

[0215] The computer device includes a processor and a memory, and may also include: an input system and an output system. The processor, memory, input system and output system may be connected via a bus or other means, and the input system may receive input digital or character information, and generate signal input related to the migration predicted by the six degrees of freedom of the unmanned ship based on visual analysis. The output system may include display devices such as display screens.

[0216] As a non-volatile computer-readable storage medium, the memory can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the program instructions / modules corresponding to the unmanned ship six-degree-of-freedom prediction method based on visual analysis in the embodiment of the present application. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of the unmanned ship six-degree-of-freedom prediction method based on visual analysis, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the local module via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0217] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data. The processors of the multiple computer devices of the computer device of this embodiment execute various functional applications and data processing of the server by running non-volatile software programs, instructions and modules stored in the memory, that is, implementing the steps of the six-degree-of-freedom prediction method of the unmanned ship based on visual analysis of the above method embodiment.

[0218] It should be understood that, without conflicting with each other, all the embodiments, features and advantages described above for the unmanned ship six-degree-of-freedom prediction method based on visual analysis according to the present invention are also applicable to the unmanned ship six-degree-of-freedom prediction and storage medium based on visual analysis according to the present invention.

[0219] It will also be appreciated by those skilled in the art that various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given to the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or hardware depends on specific applications and the design constraints imposed on the entire system. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the disclosed scope of the embodiments of the present invention.

[0220] Finally, it should be noted that the computer-readable storage medium (e.g., memory) herein may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. As an example and not by way of limitation, a non-volatile memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. A volatile memory may include a random access memory (RAM), which may act as an external cache memory. As an example and not by way of limitation, RAM may be obtained in a variety of forms, such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to include, but are not limited to, these and other suitable types of memory.

[0221] The various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure herein may be implemented or executed using the following components designed to perform the functions herein: a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and / or any other such configuration.

[0222] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope disclosed in the embodiments of the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0223] It should be understood that, as used herein, the singular form "a" or "an" is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations of one or more of the items listed in association. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments.

[0224] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the embodiments of the present invention as above, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A six-degree-of-freedom prediction method for an unmanned ship based on visual analysis, characterized in that: The method comprises the following steps: Collect visual data, inertial measurement unit data and historical navigation data of the unmanned ship to obtain multimodal data of the unmanned ship; The acquired multimodal data are fused and preprocessed, and features are extracted from the visual data. Time series analysis is performed on the inertial measurement unit data and historical navigation data. Position and attitude prediction are performed using a multi-task learning framework, and a six-degree-of-freedom prediction model for the unmanned ship is constructed. Adjust the parameters and data fusion strategy of the six-degree-of-freedom prediction model according to real-time environmental data, and correct the prediction results through an adaptive real-time correction mechanism to optimize the navigation accuracy and path planning accuracy of the unmanned ship; Multimodal data processing, six-degree-of-freedom prediction model and adaptive real-time correction mechanism are integrated into the unmanned ship control system to predict the six-degree-of-freedom state of the unmanned ship and perform precise navigation control.

2. The six-degree-of-freedom prediction method for an unmanned ship based on visual analysis as claimed in claim 1, characterized in that: The visual data collects images or point cloud data of the external environment in real time through the visual sensors carried on the unmanned ship, the inertial measurement unit data collects dynamic information of the acceleration, angular velocity and direction of the hull through the inertial sensors, and the historical navigation data is used to provide prior knowledge support by collecting the historical navigation data of the unmanned ship.

3. The six-degree-of-freedom prediction method for an unmanned ship based on visual analysis as claimed in claim 2, characterized in that: The visual data includes images or point cloud data obtained by a camera, lidar or visual sensor, and the inertial measurement unit data includes the acceleration, angular velocity and direction data of the unmanned ship; the historical navigation data includes the navigation track, speed, heading, wind speed and wave height information of the unmanned ship.

4. The six-degree-of-freedom prediction method for an unmanned ship based on visual analysis as claimed in claim 1, characterized in that: A convolutional neural network is used to extract features from the visual data to identify and capture key features of the environment, including obstacles, tracks, and water surface information. A long short-term memory network is used to perform time series analysis on the inertial measurement unit data and historical navigation data to capture the dynamic changes of the unmanned ship and trends in its future state.

5. The six-degree-of-freedom prediction method for an unmanned ship based on visual analysis as claimed in claim 4, characterized in that: When fusing and preprocessing the acquired multimodal data, the visual features, inertial measurement unit features and historical data features are spliced ​​to generate a multimodal feature vector. Different weights are assigned to each data source through weighted fusion. The weights are dynamically adjusted according to the real-time environmental data. The multimodal fusion feature vector F fuse The calculation method is as follows, using the weight α v , α i and α h Weighted superposition visual feature F vis 、Inertial measurement unit characteristics F imu and historical characteristics F h ist : F fuse =a v ·F vis +a i ·F imu +a h ·F hist Among them, α v , α i and α h Represent the weights of visual data, inertial measurement unit data and historical navigation data respectively.

6. The six-degree-of-freedom prediction method for an unmanned ship based on visual analysis as claimed in claim 5, characterized in that: Based on the real-time collected environmental data, the current environmental status is classified, and the weight of the inertial measurement unit data is increased according to the preset rules to maintain the attitude stability; among which: When the light intensity is lower than the threshold L min When α is reduced, the weight of visual data is reduced. v , the calculation formula is: When the wave height exceeds the threshold H max When α is increased, the weight of the inertial measurement unit data is increased. i : a i =a i +β·(HH max ) When the wind speed exceeds the set value W max When , reduce the historical data weight α h , enhance the impact of real-time data on predictions:

7. The six-degree-of-freedom prediction method for an unmanned ship based on visual analysis as claimed in claim 1, characterized in that: The multi-task learning framework is also used to divide the six-degree-of-freedom prediction task into two parts: position prediction and attitude prediction, predict the six-degree-of-freedom state of the unmanned ship in real time, and output the estimated values ​​of position and attitude respectively; including: Position prediction: Estimate the current position of the unmanned ship in the water through the x, y, z coordinates output by the model; Attitude prediction: Estimate the attitude state of the unmanned ship through the roll angle, pitch angle, and yaw angle output by the model.

8. The six-degree-of-freedom prediction method for an unmanned ship based on visual analysis as claimed in claim 7, characterized in that: The six-degree-of-freedom prediction method for unmanned ships based on visual analysis also includes synchronously marking the six-degree-of-freedom label of each data through GPS and attitude sensors, including position and attitude. Each data records a set of position and attitude parameters, wherein obstacles in the visual data are marked with the position, shape and size information of the obstacles to identify surface obstacles; historical navigation data are segmented by trajectory and marked with path labels to identify the target position, path curvature and navigation difficulty of each trajectory.

9. The six-degree-of-freedom prediction method for an unmanned ship based on visual analysis as claimed in claim 8, characterized in that: The six-degree-of-freedom prediction model of the unmanned ship is constructed. In the six-degree-of-freedom prediction stage, when predicting the position and attitude, the six-degree-of-freedom prediction model outputs the position and attitude, that is, [x, y, z, roll, pitch, yaw]. The multi-task loss function is used, and the position prediction uses the mean square error (MSE) loss L position , pose prediction uses cosine similarity loss L attitude ,in: The total loss function L total for: L total =λ1·L position +λ2·L attitude Among them, λ1 and λ2 are loss weights.

10. A six-degree-of-freedom prediction system for unmanned ships based on visual analysis, characterized in that: The system is used to execute the six-degree-of-freedom prediction method of an unmanned ship based on visual analysis as described in any one of claims 1 to 9, comprising: Data acquisition module: used to collect the visual data, inertial measurement unit data and historical navigation data of the unmanned ship to obtain the multimodal data of the unmanned ship; A model building module, used to fuse and preprocess the acquired multimodal data, extract features from the visual data, perform time series analysis on the inertial measurement unit data and historical navigation data, use a multi-task learning framework to predict the position and attitude respectively, and build a six-degree-of-freedom prediction model for the unmanned ship; The model optimization module is used to adjust the parameters and data fusion strategy of the six-degree-of-freedom prediction model according to real-time environmental data, and to correct the prediction results through an adaptive real-time correction mechanism to optimize the navigation accuracy and path planning accuracy of the unmanned ship; The integrated deployment module is used to integrate multimodal data processing, six-degree-of-freedom prediction model and adaptive real-time correction mechanism into the unmanned ship control system to predict the six-degree-of-freedom state of the unmanned ship and perform precise navigation control.

Images