Multi-mode sensing fusion ship navigation method and system

Abstract

The invention relates to the technical field of ship navigation, in particular to a multi-mode sensing fusion ship navigation method and system. According to the method, synchronous signal calibration is carried out on multi-path radar equipment, the position and the motion state of a ship are obtained in real time in combination with a quantum positioning system and ship sensor equipment, radar and sonar signals are processed through a radar data processing unit and a sonar detection transducer respectively, and after fusion is carried out through a data fusion processing unit, the ship position and the motion state are obtained. Target data and position information which are updated in real time are generated, gestures and voice commands are recognized through a man-machine interaction system, and finally fused data and routes are displayed on a display module. According to the invention, the precision and reliability of ship navigation are improved, and the robustness of the system and the user experience are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of ship navigation, and in particular to a multi-modal perception fusion ship navigation method and system. Background Art

[0002] Ship navigation technology is an important part of the modern navigation field. With the progress of technology, ship navigation technology has made remarkable development. Traditional ship navigation mainly relies on a single radar or GPS positioning system, and these systems have certain limitations in complex sea area environments. In order to improve the accuracy and reliability of ship navigation, multi-modal perception technology has gradually become a research hotspot. Multi-modal perception technology combines the information of multiple sensors, such as radar, sonar, satellite positioning, etc. By fusing and processing data from different sources, it can comprehensively grasp the environmental information around the ship, thereby enhancing the safety and efficiency of navigation.

[0003] In the existing ship navigation technology, common solutions include using a single radar system for environmental monitoring, using GPS or other satellite positioning systems to obtain the ship's position information, and detecting underwater obstacles through sonar equipment. In addition, some more advanced systems have begun to attempt to integrate multiple sensors, such as the joint use of radar and sonar, and introducing an on-board sensor network to obtain richer motion state information. However, the data processing and fusion of these systems are usually relatively primitive, lacking an effective data fusion mechanism, resulting in low information utilization rate and unable to fully utilize the advantages of each sensor.

[0004] The main defects of the existing technology are that although multi-modal perception technology has been widely recognized in theory, in practical applications, the data collection and processing of various sensors often lack an effective synchronization mechanism, making it difficult to ensure the consistency and accuracy of data. At the same time, the fusion processing ability of multi-source data is limited. Especially in complex marine environments, how to efficiently integrate and analyze information from different sensors is still an urgent problem to be solved. In addition, the existing systems are relatively simple in terms of human-computer interaction, lacking effective recognition and support for gestures and voice commands, which limits the convenience and flexibility of operations. In view of the above problems, the existing technology urgently needs to be improved. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-modal perception fusion ship navigation method, system, computer device and storage medium, which have the advantages of improving the synchronization of multi-sensor data and positioning accuracy, enhancing the convenience of human-computer interaction, and improving the navigation reliability in complex sea area environments.

[0006] The present invention provides a multimodal perception fusion ship navigation method, and the technical solution is as follows: A multimodal perception fusion ship navigation method includes the following steps: S1. Synchronize the signal calibration of multiple ship radar devices so that the multiple radars are synchronized at the same moment; S2. Obtain the ship position information in real time through the positioning system, locate the ship with the help of the quantum positioning system, and obtain the ship motion state with the help of the on-board sensor device; S3. The radar data processing unit calculates the azimuth and distance information of the surrounding environment targets and positioning targets through the echo processing algorithm, and transmits the obtained data to the radar data fusion module; S4. The transducer of sonar detection receives the signal formed by underwater acoustic propagation and the background noise signal, and outputs them to the ship's sonar integrated processing unit for processing; S5. Identify gestures and voice commands through the human-computer interaction unit, decompose the commands into text after identification, and execute the corresponding task calls according to the text; S6. The data fusion processing unit updates the radar target data, position information data, and graphic display data in real time after fusion, and transmits them to the display module; S7. Call the corresponding data fusion display screen according to the current task, obtain the corresponding data from the data fusion unit, call the graphic display library according to the screen setting parameters to draw the route, and present it on the display through the display module.

[0007] Further, the present invention also proposes that the step of obtaining the ship position information in real time through the positioning system is: S21. Measure the distance between the sending end and the receiving end of the quantum positioning system according to actual needs; S22. Send the quantum signal in the form of an optical pulse sequence; S23. Receive the signal according to the receiver carried by the crew; S24. Determine the quantum position information according to the received optical pulse and transmit it in real time.

[0008] Further, the present invention also proposes that the steps of identifying gestures and voice commands through the human-computer interaction system are: S51. Obtain the image or audio signal from the human-computer interaction device; S52. The image processing module performs gesture recognition through feature extraction and recognition classification; S53. The voice recognition engine recognizes the received voice signal to obtain the corresponding text.

[0009] Furthermore, the present invention also proposes that the processing steps for the transducer receiving signal in sonar detection are as follows: S41. Perform filtering preprocessing to filter out echo interference and reduce signal distortion. The filtering function is a frequency-related function, where f is the frequency of the signal to be filtered, f0 is the filtering center frequency, and A and μ are filtering function parameters. For the signal with a frequency f = 0 to be filtered, the filtering function is Calculate the signal angle and distance information as: θ = arctan(In / Im + π / 2), where In is the signal received inside the radar and Im is the signal received by the radar; S42. Amplify the signal. Through signal amplification, the amplification of the echo signal is achieved. The echo signal X is related to the amplification gain K and the noise N. The amplified signal is Y = KX + N, where the amplification gain K can be set by controlling the amplifier parameters, and the noise N is the noise generated by the amplifier; S43. According to the received echo signal, identify the distance and azimuth information of the positioning target and the surrounding environmental targets.

[0010] Furthermore, the present invention also proposes that after preprocessing, for the echo signal, it is received by an echo transducer. The output voltage of the transducer is Ui = 2πfC0Ii, where C0 is the capacitance characteristic and Ii is the output current of the echo transducer. The echo signal is processed and amplified by an amplifier, and the output signal is Ua.

[0011] Furthermore, the present invention also proposes that the echo transducer and the azimuth scanning mechanism are installed on the detection head. The azimuth scanning mechanism can rotate, and the azimuth of the target sound wave emission is calculated by rotating the echo transducer. The azimuth scanning mechanism is driven by a stepper motor, and the azimuth angle step value is calculated using the following formula: θmin = arcsin(1 / 2XmaxD / F), sθ = D / F - θmin, θmax = D / F - θmin, where θ is the echo angle, D is the maximum distance of the radar echo, Xmax is the maximum amplitude of the echo, F is the transmission frequency, θmin is the minimum value of the azimuth change, s is the step amount of the stepper motor, and θmax is the maximum value of the azimuth change.

[0012] Furthermore, the present invention also proposes that the radar data fusion module matches the distance and azimuth information of the target obtained by radar scanning with the coordinates on the electronic chart to obtain the coordinates of the target on the electronic chart: Ri = λ + (Xi - X1) / (X2 - X1), θi = Yi - Y1 + 90°, where R refers to the navigation ship as the coordinate origin, X and Y are the Cartesian coordinates of the electronic chart. Taking the navigation ship as the coordinate origin, calculate the azimuth and distance coordinates of the target on the electronic chart. When Xi = X1 and Yi = Y1, the target coordinates are the radar coordinate origin, and λ is the laser coordinate rotation parameter.

[0013] Furthermore, the present invention also provides a multimodal perception fusion ship navigation system, which includes a radar data processing unit, a radar data fusion module, a radar sensor, a radar data fusion unit, a positioning system, a controller, a graphics display library, a comprehensive display module, a human-computer interaction system, a voice recognition engine, a sonar sensor, and a display module. The output of the radar sensor is connected to the radar data processing unit, the output of the radar data processing unit is connected to the radar data fusion module, the positioning system is connected to the controller, the output of the controller is connected to the radar data fusion module, the radar data fusion module is connected to the radar data fusion unit, the radar data fusion unit is connected to the comprehensive display module, the comprehensive display module is also connected to the graphics display library, the display module is connected to the human-computer interaction system, the human-computer interaction system includes a video recognition module and a voice recognition engine, both the video recognition module and the voice recognition engine are connected to the controller, the voice recognition engine is connected to the comprehensive display module, the output of the sonar sensor is connected to the controller, and the output of the human-computer interaction system is connected to the comprehensive display module.

[0014] Furthermore, the present invention also provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above method is implemented.

[0015] Furthermore, the present invention also provides a computer-readable storage medium, which stores a computer program for executing the above method.

[0016] As can be seen from the above, a multimodal perception fusion ship navigation method, system, computer device, and storage medium provided by the present invention achieve efficient synchronous acquisition and fusion processing of multimodal sensor data through technical means such as multi-channel radar synchronous calibration, quantum positioning and sensor data fusion, sonar signal processing, and human-computer interaction instruction decomposition, improve the navigation positioning accuracy and interaction convenience in complex environments, and have the advantages of improving data synchronization and positioning accuracy, enhancing human-computer interaction convenience, and improving navigation reliability in complex sea areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] The technical solutions in the present application will be clearly and completely described below with reference to the accompanying drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the present application usually described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application claimed, but merely represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application. It should be noted that: similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, the terms "first", "second", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0019] In traditional ship navigation systems, the application of multi-modal perception technology faces core problems such as the lack of a multi-source heterogeneous data synchronization mechanism, inefficient data fusion algorithms, and a single human-computer interaction method. The multi-channel radar equipment has a time deviation in the collected data due to the lack of an accurate clock synchronization mechanism. The data interface protocol between the quantum positioning system and conventional sensors is not unified, resulting in a coordinate reference difference. The parameter setting of the filtering function in the sonar signal processing process has insufficient matching with the echo characteristics, and a unified spatial coordinate system conversion relationship has not been established for the azimuth calculation models output by different sensors. These problems directly lead to the fracture of the target tracking trajectory, the coordinate mapping error of the electronic nautical chart exceeding the tolerance threshold, and the environmental obstacle recognition delay reaching 15%-20% of the system response period.

[0020] For example, when performing a night navigation mission in a sea area with strong electromagnetic interference, the multi-channel X-band radars of a certain type of 10,000-ton cargo ship have a time difference of 32 ms due to the crystal oscillator frequency drift, resulting in an angular deviation of 0.8° in the AIS target azimuth information obtained in adjacent scan cycles. The reverberation signal received by the sonar array has a center frequency offset of 200 Hz in the band-stop filter, causing the signal-to-noise ratio of the underwater reef echo to drop below 3 dB. The operator needs to operate 7 physical control panels simultaneously to input steering commands, resulting in an emergency collision avoidance operation response time exceeding the 3-second standard stipulated by the International Maritime Organization.

[0021] If the above problems are not solved, the non-uniform spatio-temporal reference of multi-sensors will lead to the confidence level of the fused track being lower than the convergence condition of the Kalman filter. The cumulative error of the dynamic update of the electronic chart may exceed the 50% threshold of the ECDIS safety boundary layer. The difference in the processing delay of heterogeneous data will cause the rank deficiency phenomenon in the multi-target tracking association matrix, and the decision tree of the ship's automatic collision avoidance system will not be able to complete the optimal path search within a limited time. The operation complexity of the traditional interaction method will make the workload index of the duty officers exceed the ergonomic safety standard specified by the IMO, significantly increasing the probability of human operation errors.

[0022] Facing the above problems, the present application first considers establishing a synchronization mechanism for multiple radar devices to eliminate time deviation. However, the traditional clock synchronization scheme requires additional hardware support and is too costly. In this regard, the present application proposes to achieve device-level synchronization through signal calibration, which directly acts on the radar signal transmission link. To solve the coordinate reference difference between quantum positioning and traditional sensors, the present application chooses to establish a unified spatio-temporal reference system before data fusion and realizes the spatial alignment of multi-source data through a dynamic coordinate transformation model. Aiming at the defects of sonar signal processing, the present application adopts an adaptive filtering algorithm based on echo characteristics to complete noise suppression and feature enhancement in the preprocessing stage. In addition, the present application finds that the traditional human-computer interaction interface has a problem of a single input channel, and optimizes the operation process design by integrating multi-modal input methods of gesture recognition and voice commands.

[0023] In this regard, the present application proposes a multi-modal perception fusion ship navigation method, including the following steps: performing synchronous signal calibration on multiple ship radar devices to keep the multiple radars synchronized at the same moment; obtaining the ship position information in real time through a positioning system, positioning the ship with the help of a quantum positioning system, and obtaining the ship motion state with the help of on-board sensor devices; the radar data processing unit calculates the azimuth and distance information of the surrounding environmental targets and positioning targets through an echo processing algorithm, and transmits the obtained data to the radar data fusion module; the transducer of sonar detection receives the signal formed by underwater sound propagation and the background noise signal, and outputs it to the ship's sonar integrated processing unit for processing; the human-computer interaction unit recognizes gestures and voice commands, decomposes the commands into text after recognition, and executes the corresponding task calls according to the text; the data fusion processing unit updates the radar target data, position information data, and graphic display data in real time after fusion, and transmits them to the display module; according to the current task, call the corresponding data fusion display screen, obtain the corresponding data from the data fusion unit, and call the graphic display library according to the screen setting parameters to draw the route, and present it on the display through the display module.

[0024] Among them, multimodal perception fusion refers to integrating and processing data from different sensors. Specifically, it can be achieved by using multi-source data fusion algorithms for radar, sonar, and quantum positioning systems. By eliminating data differences between sensors, it solves the problem of incomplete information of traditional single sensors.

[0025] Among them, synchronous signal calibration refers to uniformly calibrating the time reference of multiple radar devices. Specifically, it can be achieved by using timestamp synchronization technology or hardware trigger mechanisms to ensure the timing consistency of data collected by multiple radars and solve the positioning errors caused by asynchronous multi-sensor data.

[0026] Among them, a quantum positioning system refers to a device that uses quantum entanglement effects for position measurement. Specifically, it can be achieved by encoding quantum state information with optical pulse sequences to solve the signal interference problem of traditional satellite positioning in complex electromagnetic environments.

[0027] Among them, the echo processing algorithm refers to a mathematical method for analyzing radar reflection signals. Specifically, it can be achieved by using fast Fourier transform or matched filtering algorithms to extract target distance and azimuth information and solve the problem of inaccurate target recognition caused by noise interference.

[0028] Among them, gesture and voice command recognition of the human-computer interaction unit refers to parsing user instructions through image and audio signals. Specifically, it can be achieved by using convolutional neural network or hidden Markov model algorithms to reduce the response delay of manual operations in complex environments.

[0029] Among them, the real-time update of the data fusion processing unit refers to dynamically integrating and correcting multi-source data. Specifically, it can be achieved by using Kalman filtering or Bayesian estimation methods to solve the problem of decision-making lag caused by redundant or conflicting multi-sensor data.

[0030] Among them, the route drawing of the graphics display library refers to converting fusion data into visual navigation information. Specifically, it can be achieved by using a vector map rendering engine or dynamic path planning algorithms to solve the problem of insufficient update rate of traditional electronic nautical charts.

[0031] The core innovation of this application lies in constructing a complete ship navigation closed-loop system through multi-modal data synchronization calibration, quantum positioning, and multi-source sensor fusion processing, combined with real-time human-computer interaction and dynamic graphics display technology, to solve the technical bottlenecks of isolated traditional single-sensor data, incomplete environmental perception, and low interaction efficiency.

[0032] The working process and principle of this application are as follows: First, synchronous signal calibration is performed on multiple ship radar devices to ensure that multiple radars are synchronized at the same time. This step eliminates the time deviation between different radars by adjusting the clock synchronization mechanism of the radar devices, laying a foundation for subsequent data fusion.

[0033] Next, the ship's position information is obtained in real time through a positioning system. Specifically, a quantum positioning system is used to accurately locate the ship, and at the same time, on-board sensor devices are used to obtain the ship's motion state. The quantum positioning system provides high-precision position data, while the on-board sensors provide dynamic information such as speed and attitude.

[0034] The radar data processing unit calculates the azimuth and distance information of the surrounding environmental targets and positioning targets through an echo processing algorithm. This algorithm filters, amplifies, and analyzes the radar echo signal to extract target features. The processed data is transmitted to the radar data fusion module to prepare for subsequent fusion analysis.

[0035] The transducer of sonar detection receives the signals formed by underwater acoustic propagation and background noise signals. These signals are output to the ship's sonar integrated processing unit for processing. Sonar signal processing includes steps such as noise suppression, signal enhancement, and target recognition to obtain underwater environmental information.

[0036] The human-computer interaction unit recognizes gesture and voice commands. After recognition, the commands are decomposed into text, and corresponding task calls are executed according to the text. This multi-modal interaction method improves the flexibility and efficiency of operation.

[0037] The data fusion processing unit updates the fused radar target data, position information data, and graphic display data in real time and transmits them to the display module. The fusion processing involves spatio-temporal alignment, feature extraction, and information integration of multi-source data.

[0038] Finally, the corresponding data fusion display screen is called according to the current task. The corresponding data is obtained from the data fusion unit, and the graphic display library is called according to the screen setting parameters to draw the route. The final result is presented on the display through the display module, providing intuitive visual support for ship navigation decision-making.

[0039] As a preferred embodiment, the solution of the present application is specifically implemented as follows:

[0040] When synchronizing the signal calibration of the ship's multi-channel radar equipment, a high-precision clock source is used as the reference, and the time synchronization of each radar equipment is carried out through the Network Time Protocol (NTP). The synchronization accuracy can reach the microsecond level, ensuring the time consistency of multi-channel radar data.

[0041] The ship positioning system adopts a combined navigation solution of a dual-frequency GPS receiver combined with an Inertial Navigation System (INS). The GPS receiver provides absolute position information, and the INS provides high-frequency attitude and speed data. The two perform data fusion through the Kalman filtering algorithm to achieve centimeter-level positioning accuracy.

[0042] The radar data processing unit uses pulse compression technology to improve range resolution and adopts the Constant False Alarm Rate (CFAR) detection algorithm for target detection. For the multi-target scenario, the Nearest Neighbor algorithm is applied for data association to form target tracks.

[0043] Sonar signal processing uses adaptive beamforming technology to improve the signal-to-noise ratio and direction resolution. A matched filter is used for target detection, and time-frequency analysis methods are combined to identify target features.

[0044] The human-computer interaction unit integrates deep learning-based gesture recognition algorithms and natural language processing technologies. 3D convolutional neural network models are used for gesture recognition, and end-to-end models based on transformers are used for speech recognition. The recognition results are converted into system instructions through rule-based parsers.

[0045] The data fusion processing unit adopts a distributed information fusion framework. The Federated Kalman Filter algorithm is used for state estimation of multi-source sensor data, and the evidence theory is applied to process uncertain information, finally generating a unified situation awareness result.

[0046] The display module uses the OpenGL graphics library for 3D scene rendering, supporting multi-level electronic chart display. Route drawing and target tracking visualization are realized through vector graphics technology, providing interactive functions such as zooming and panning.

[0047] Through the above solutions, this application realizes the efficient fusion and processing of multi-modal perception data. Synchronization signal calibration eliminates the time deviation of multi-channel radar data and improves the accuracy of target tracking. The combination of the quantum positioning system and traditional sensors enhances the reliability of ship position information. The adaptive filtering algorithm improves the processing effect of sonar signals and enhances the underwater environment perception ability. The multi-modal human-computer interaction method simplifies the operation process and reduces human operation errors. The data fusion processing unit realizes the unified processing of heterogeneous data, providing a comprehensive information basis for situation awareness and decision support. The comprehensive application of these technical means significantly improves the performance and reliability of the ship navigation system in complex sea area environments, providing a strong guarantee for safe and efficient navigation.

[0048] In some of the above solutions of this application, when obtaining the ship position information in real time through the positioning system, there is a problem of asynchronous transmission and reception of quantum signals, resulting in delayed update or increased error of the position information.

[0049] This application further proposes a method for obtaining the ship position information in real time through the positioning system, including the following steps: measuring the distance between the sending end and the receiving end of the quantum positioning system according to actual needs; sending the quantum signal in the form of an optical pulse sequence; receiving the signal according to the receiver carried by the crew; determining the quantum position information based on the received optical pulses and transmitting it in real time.

[0050] Among them, the optical pulse sequence form of the quantum signal can be realized by single - photon pulses or entangled photon pairs. At the sending end, a modulator is used to encode the quantum state information into the optical pulses, and at the receiving end, the signal is captured by a single - photon detector or a coincidence counting device. The distance measurement between the sending end and the receiving end can be completed based on the quantum key distribution protocol or the quantum time synchronization protocol. The receiving end carried by the crew can be integrated into a portable device or a wearable terminal and connected to the ship's main control system through a wireless communication module. The transmission path of the optical pulses needs to consider the attenuation and interference of the ocean environment, and wavelength - division multiplexing or polarization - encoding technology is used to improve the signal stability.

[0051] Specifically, the sending end generates an optical pulse sequence according to a preset quantum state. For example, a single - photon pulse is emitted by a laser, and a position - encoding information is loaded by a modulator. After the receiving end captures the optical pulse through a detector, the signal is decoded using a quantum measurement basis, and the propagation delay is calculated by combining the timestamp information, thereby determining the distance parameter. The portable design of the receiving end enables it to be flexibly deployed at different positions on the ship, such as the deck or the cabin, and the errors caused by signal occlusion are eliminated through multi - node cooperation. During the transmission of the optical pulse, an adaptive gain control technology is used to compensate for the signal attenuation caused by the ocean atmosphere or water body, and error - correcting coding is used to reduce the influence of noise. In the real - time transmission link, the position data is encrypted and uploaded to the ship's navigation system through a low - latency communication protocol to ensure the synchronous fusion with radar and sonar data. Thus, the measurement accuracy and real - time performance of the quantum positioning system are significantly improved, solving the problem of signal drift of traditional satellite positioning in complex sea conditions.

[0052] As a preferred embodiment, the solution of the present application is specifically implemented as follows:

[0053] The distance is measured between the sending end and the receiving end of the quantum positioning system. Specifically, the sending end is installed on the ship, and the receiving end is carried by the crew. The sending end includes a quantum signal generator and an optical pulse transmitter, and the receiving end includes an optical pulse receiver and a quantum information processing unit.

[0054] The quantum signal is sent in the form of an optical pulse sequence. Further, the quantum signal generator generates quantum - bit information, and the optical pulse transmitter encodes the quantum - bit information into an optical pulse sequence. The wavelength of the optical pulse sequence is 1550 nm, the pulse width is 100 ps, and the repetition frequency is 10 MHz.

[0055] The receiving end carried by the crew receives the signal. Thus, the optical pulse receiver captures the sent optical pulse sequence and converts it into an electrical signal.

[0056] Determine quantum position information based on the received optical pulses and transmit it in real time. Specifically, the quantum information processing unit decodes the received electrical signals to extract qubit information. The quantum state is reconstructed through quantum state tomography, and the precision of position estimation is calculated using the quantum Fisher information theory. Finally, the calculated precise position information is transmitted in real time to the navigation system of the ship through the wireless communication module.

[0057] Through the above technical solution, the present application achieves high-precision ship positioning. The quantum positioning system utilizes the characteristics of quantum states to overcome the problems of the traditional GPS positioning system affected by the ionosphere and multipath effects. At the same time, the real-time transmitted position information provides accurate and reliable data input for the ship navigation system, improving the precision and reliability of navigation. In addition, the design of the crew carrying the receiving end enhances the flexibility and practicality of the system, enabling the crew to obtain precise position information at different positions on the ship.

[0058] In some of the above solutions of the present application, the human-computer interaction system needs to process gestures and voice commands simultaneously. However, in actual applications, the existing interaction methods often only support single-modal input, resulting in insufficient operation flexibility and unable to effectively integrate multi-modal information to improve the interaction efficiency and accuracy.

[0059] The present application further proposes steps for the human-computer interaction system to recognize gestures and voice commands, including: obtaining image or audio signals from the human-computer interaction device; the image processing module performs gesture recognition through feature extraction and recognition classification; the speech recognition engine recognizes the received speech signal to obtain the corresponding text.

[0060] Among them, the image processing module uses a convolutional neural network model based on deep learning for gesture feature extraction. After the input image is preprocessed, spatial features are extracted through multiple convolutional layers, and then classification output is performed through a fully connected layer. The speech recognition engine uses an end-to-end acoustic model, combined with a hidden Markov model and a language model for decoding, to convert the time-domain speech signal into a text sequence. The results of gesture recognition and speech recognition are aligned through a time synchronization mechanism to ensure the collaborative parsing of multi-modal instructions.

[0061] Specifically, when the user inputs a gesture image through the camera, the image processing module first normalizes the image to eliminate illumination and background interference, and then extracts the hand contour and key point features, and judges the gesture type through a classifier. At the same time, after the voice signal is denoised and framed, the Mel Frequency Cepstral Coefficients (MFCCs) are extracted as features by the speech recognition engine, and the phoneme probabilities are calculated by inputting them into the acoustic model, and the most likely text sequence is generated in combination with the language model. The gesture recognition result is matched with the speech recognition text through timestamps. If both are recognized within the preset time window, the fused instruction is executed. For example, when the user makes a "swipe right" gesture and says the "zoom in" instruction, the system combines the interface swipe operation corresponding to the gesture with the zoom operation corresponding to the speech to generate a composite control signal to drive the display module to adjust the view. This process reduces the misjudgment rate of a single modality by processing image and voice signals in parallel, and improves the execution accuracy of complex instructions through semantic fusion.

[0062] As a preferred embodiment, the solution of the present application is specifically implemented as follows:

[0063] The human-computer interaction system includes an image acquisition device and an audio acquisition device. The image acquisition device acquires the gesture images of the crew members, and the audio acquisition device acquires the voice commands of the crew members.

[0064] The image processing module first preprocesses the acquired gesture images, including image denoising, enhancement, and segmentation. Then, the features of the gestures, such as contours, shapes, and directions, are extracted. Finally, the pre-trained classifier is used to classify the extracted features to identify specific gesture commands.

[0065] The speech recognition engine preprocesses the acquired speech signals, including denoising and framing. Then, the acoustic features of the speech, such as Mel Frequency Cepstral Coefficients (MFCCs), are extracted. Finally, the deep neural network model is used to convert the acoustic features into text.

[0066] The recognition results are transmitted to the control system in text form, and the control system performs corresponding ship operation tasks according to the recognized gestures and voice commands.

[0067] Through the above technical solution, the present application realizes the accurate recognition of the crew members' gestures and voice commands, improves the efficiency and convenience of human-computer interaction. This solution can work stably in a complex ship environment, provides an intuitive operation method for the crew members, and reduces the possibility of misoperation. At the same time, the multi-modal interaction method combining gestures and voice enhances the robustness of the system and improves the flexibility and safety of ship navigation control.

[0068] In some of the above solutions of the present application, during sonar detection, the echo signal is vulnerable to background noise and interference, resulting in signal distortion, which in turn affects the accuracy of target recognition. Especially in complex sea area environments, traditional filtering methods are difficult to effectively separate the effective signal from the noise.

[0069] The present application further proposes processing steps for the signals received by the transducer of sonar detection, including: performing filtering preprocessing to filter out echo interference and reduce signal distortion; amplifying the signal to amplify the echo signal through signal amplification; and identifying and positioning the distance and azimuth information of the target and surrounding environmental targets based on the received echo signal.

[0070] Among them, the filtering preprocessing uses a frequency-related function. By setting the filtering center frequency f0 and parameters A, μ, a filter with specific frequency band suppression ability is constructed. When it is necessary to filter out the signal with frequency f = 0, the corresponding filtering function is selected to eliminate baseline drift. In the signal amplification process, a gain coefficient K is introduced, and the signal intensity is controlled by adjusting the amplifier parameters. At the same time, the influence of the amplifier's own noise N on the output signal Y = KX + N is considered. The angle calculation uses the formula θ = arctan(In / Im + π / 2), and the ratio of the signals received inside and outside the radar is used to determine the target azimuth.

[0071] Specifically, the original signal received by the transducer first undergoes filtering function processing to eliminate low-frequency interference and high-frequency noise and retain the effective echo frequency band. The preprocessed signal is sent to a variable gain amplifier, and the gain value K is dynamically adjusted according to the sea area environment, so that the weak echo signal reaches a processable level while ensuring the signal-to-noise ratio. The amplified signal drives a stepper motor through an azimuth scanning mechanism, and the minimum azimuth angle is calculated in combination with the formula θ = arcsin(1 / 2XmaxD / F). The target azimuth is accurately calculated using the parameters of the maximum echo amplitude Xmax and the transmission frequency F. Finally, the accurate azimuth angle θ is obtained by calculating the ratio of In to Im, and the three-dimensional coordinate data of the target is output in cooperation with the distance calculation algorithm. This processing flow effectively improves the target recognition accuracy in complex noise environments through cascaded filtering, amplification, and calculation modules.

[0072] As a preferred embodiment, the solution of the present application is specifically implemented as follows:

[0073] Process the signals received by the transducer of sonar detection. First, perform filtering preprocessing to filter out echo interference and reduce signal distortion. The filtering function is a frequency-related function, f is the frequency of the signal to be filtered, f0 is the filtering center frequency, where A and μ are the parameters of the filtering function. For the signal with frequency f = 0 to be filtered, the filtering function is to calculate the signal angle and distance information: θ = arctan(In / Im + π / 2), where In is the signal received inside the radar and Im is the signal received by the radar.

[0074] Next, the signal is amplified. Through signal amplification, the amplification of the echo signal is achieved. The echo signal X is related to the amplification gain K and the noise N. The amplified signal is Y = KX + N, where the amplification gain K can be set by controlling the amplifier parameters, and the noise N is the noise generated by the amplifier.

[0075] Finally, based on the received echo signal, the distance and azimuth information of the positioning target and the surrounding environmental targets are identified.

[0076] In specific implementation, a digital signal processor (DSP) can be used to perform filtering and signal amplification operations. For the filtering preprocessing, a band-pass filter can be used, with the filtering center frequency f0 set to 50 kHz and the bandwidth to 10 kHz. For signal amplification, a variable gain amplifier can be used, with the initial gain set to 20 dB, and the gain is automatically adjusted according to the received signal strength. The calculation of the distance and azimuth information can be achieved through the FFT transform and the CFAR detection algorithm.

[0077] Through the above technical solutions, this application can effectively improve the quality and reliability of sonar signals. The filtering preprocessing can remove environmental noise and interference signals, improving the signal-to-noise ratio of the effective signals. Signal amplification can enhance weak echo signals and expand the detection range. By analyzing the processed signals, the surrounding environmental targets can be accurately identified, providing reliable underwater environment information for ship navigation, thereby improving navigation safety. In addition, this solution can adapt to the sonar detection requirements under different sea conditions, with strong practicality and adaptability.

[0078] In some of the above solutions of this application, there are signal conversion errors and noise interference when the sonar transducer receives the echo signal, resulting in insufficient accuracy of the subsequent processed data and affecting the recognition accuracy of the target azimuth and distance.

[0079] This application further proposes that the output voltage Ui of the echo transducer is calculated by the formula Ui = 2πfC0Ii, where C0 is the capacitance characteristic parameter and Ii is the transducer output current. The echo signal is processed by the amplifier and the output signal is Ua.

[0080] Among them, the calculation formula of the transducer output voltage clearly stipulates the product relationship between the capacitance characteristic parameter and the frequency, quantifying the conversion ratio of the current and the voltage. The processing process of the amplifier includes a gain adjustment function, which can dynamically adjust the amplification multiple according to the signal strength. The capacitance parameter C0 is directly related to the physical structure of the transducer, the frequency f is determined by the characteristics of the sonar transmitted signal, and the current Ii reflects the original strength of the echo signal.

[0081] Specifically, when the transducer converts the received acoustic wave signal into an electrical signal, a linear relationship between voltage and current is established through the formula Ui = 2πfC0Ii, where the capacitance parameter C0 is pre-calibrated according to the material characteristics of the transducer, and the frequency f is synchronized with the sonar transmission frequency. When the amplifier amplifies Ui, it adjusts the amplitude of the output signal Ua according to the preset gain coefficient. For example, when the echo signal strength is weak, the gain coefficient is increased to improve the signal-to-noise ratio; when the signal is saturated, the gain is reduced to avoid waveform distortion. The calibration of the capacitance parameter C0 is obtained through experimental measurement to ensure the accuracy of voltage conversion. Through this formula and amplification processing, the weak current signal output by the transducer is accurately converted into a processable voltage signal, and at the same time, the amplifier effectively suppresses circuit noise, enabling the subsequent azimuth scanning mechanism to calculate the target azimuth based on the clear signal.

[0082] As a preferred embodiment, the solution of the present application is specifically implemented as follows:

[0083] After preprocessing, for the echo signal, it is received by the echo transducer. The echo transducer is made of piezoelectric ceramic material and has good acoustic-electric conversion characteristics. The transducer outputs a voltage Ui = 2πfC0Ii, where C0 is the capacitance characteristic and Ii is the output current of the echo transducer. The echo signal is processed and amplified by the amplifier, and the output signal is Ua. The amplifier uses a low-noise operational amplifier, which has the characteristics of high input impedance and low output impedance. The gain of the amplifier can be controlled by adjusting the feedback resistor, and is usually set to 20 - 40 dB. The amplified signal Ua enters the analog-to-digital converter to convert the analog signal into a digital signal for subsequent digital signal processing. The sampling rate of the analog-to-digital converter is set to 1 MHz, and the resolution is 16 bits to ensure high-precision acquisition of the signal.

[0084] Through the above technical solution, the present application realizes the efficient reception and processing of echo signals. The echo transducer converts the acoustic wave signal into an electrical signal, and the amplifier amplifies the weak signal, improving the signal-to-noise ratio of the signal. By precisely controlling the amplifier gain, it can adapt to echo signals of different intensities, expanding the dynamic range of the system. High-precision analog-to-digital conversion ensures the accuracy of digital signals, providing a reliable data basis for subsequent signal analysis and target recognition. This signal processing method improves the system's detection ability for weak signals and enhances the target detection performance in complex marine environments.

[0085] In some of the above solutions of the present application, there are problems with insufficient accuracy in the installation method of the echo transducer and the azimuth scanning mechanism on the detection head and the calculation method of the azimuth angle. Since the rotation accuracy of the azimuth scanning mechanism directly affects the accuracy of target azimuth recognition, the relationship between the control parameters of the stepping motor and the calculation of the azimuth angle is not clear in the traditional method, resulting in limited real-time performance and accuracy of target azimuth detection.

[0086] The present application further proposes that the echo transducer and the azimuth scanning mechanism are installed in the detection head. The azimuth scanning mechanism can rotate, and the azimuth of the target sound wave emission is calculated by rotating the echo transducer. The azimuth scanning mechanism is driven by a stepper motor, and the azimuth angle step value is calculated using the following formula: θmin = arcsin(1 / 2XmaxD / F) s = D / F - θmin θmax = D / F - θmin where θ is the echo angle, D is the maximum distance of the radar echo, Xmax is the maximum amplitude of the echo, F is the emission frequency, θmin is the minimum value of azimuth change, s is the step amount of the stepper motor, and θmax is the maximum value of azimuth change.

[0087] Among them, the azimuth scanning mechanism and the echo transducer are integrated inside the detection head, and the detection head is configured as a rotatable structure; the step amount s of the stepper motor, the azimuth angles θmin and θmax are mathematically related through the formula; the parameter Xmax in the formula is dynamically determined by the maximum amplitude of the radar echo, and F is automatically adjusted according to the sound wave emission frequency; D is the target distance corresponding to the maximum value of the echo signal intensity within the radar detection range.

[0088] Specifically, when the detection head rotates, the stepper motor drives the azimuth scanning mechanism to adjust the angle with the step amount s, and the azimuth angle change corresponding to each step is calculated through the difference between θmin and θmax. When the maximum amplitude Xmax of the radar echo changes, θmin is dynamically adjusted according to the formula arcsin(1 / 2XmaxD / F) to ensure that the calculation of the azimuth angle matches the characteristics of the current echo signal. The change in the emission frequency F realizes the adaptive update of the angle calculation by adjusting the denominator term, so that the step amount s of the stepper motor is synchronized with the current working frequency. This calculation method makes the resolution of the azimuth angle form a quantitative relationship with the radar detection distance D and the signal intensity, avoiding azimuth recognition errors caused by parameter changes. For example, when the detection distance D increases, the calculation result of θmin automatically decreases, and the step amount s is adjusted accordingly to improve the azimuth detection accuracy of distant targets.

[0089] As a preferred embodiment, the solution of the present application is specifically implemented as follows:

[0090] The echo transducer and the azimuth scanning mechanism are installed in the detection head. The azimuth scanning mechanism can rotate, and the azimuth of the target sound wave emission is calculated by rotating the echo transducer. The azimuth scanning mechanism is driven by a stepper motor. The azimuth angle step value is calculated using the following formula:

[0091] θmin = arcsin(1 / 2XmaxD / F) s = D / F - θmin

[0092] θmax = D / F - θmin

[0093] Where θ is the echo angle, D is the maximum distance of the radar echo, Xmax is the maximum amplitude of the echo, F is the transmission frequency, θmin is the minimum value of the azimuth change, s is the step size of the stepper motor, and θmax is the maximum value of the azimuth change.

[0094] During specific implementation, the detection head is installed at the front of the ship. The echo transducer is made of piezoelectric ceramic material and has good acoustic-electric conversion characteristics. The azimuth scanning mechanism consists of a stepper motor, a reducer, and a transmission shaft. The stepper motor is a five-phase stepper motor, and each step rotates by 0.72 degrees. The reducer uses a planetary gear reduction mechanism with a reduction ratio of 50:1. The transmission shaft is made of stainless steel and is connected to the echo transducer.

[0095] In practical applications, assume that the maximum distance D of the radar echo is 5000 meters, the maximum amplitude Xmax of the echo is 0.8, and the transmission frequency F is 10 kHz. According to the above formula, the following results are calculated:

[0096] θmin = arcsin(1 / (2 * 0.8 * 5000 / 10000)) ≈ 11.5 degrees, s = 5000 / 10000 - 11.5 ≈ 38.5 degrees, θmax = 5000 / 10000 - 11.5 ≈ 38.5 degrees

[0097] The stepper motor rotates 0.72 degrees per step. After passing through the reducer, the actual rotation angle per step is 0.0144 degrees. To cover the entire scanning range, approximately 2670 steps of rotation are required.

[0098] Through the above technical solution, the present application realizes precise control of the rotation angle of the azimuth scanning mechanism, improving the accuracy of target azimuth measurement. Since the stepper motor is used for driving, the system has good positioning accuracy and repeat positioning ability. By reasonably setting the stepper parameters, the scanning speed can be optimized while ensuring the scanning accuracy, improving the efficiency of sonar detection. In addition, this solution has a simple structure, is easy to implement, and has high reliability and stability.

[0099] In some of the above solutions of the present application, when the radar data fusion module matches the target distance and azimuth information obtained from radar scanning with the electronic chart coordinates, there is a lack of an accurate coordinate conversion method, resulting in a deviation in the position display of the target on the electronic chart and affecting the navigation accuracy.

[0100] The present application further proposes that the radar data fusion module matches the distance and azimuth information of the target obtained by radar scanning with the coordinates on the electronic chart to obtain the coordinates of the target on the electronic chart: Ri = λ + (Xi - X1) / (X2 - X1) θi = Yi - Y1 + 90° where R refers to the origin of the navigation ship as the coordinate origin, X and Y are the Cartesian coordinates of the electronic chart. Taking the navigation ship as the coordinate origin, the azimuth and distance coordinates of the target on the electronic chart are calculated. When Xi = X1 and Yi = Y1, the target coordinates are the radar coordinate origin, and λ is the laser coordinate rotation parameter.

[0101] Among them, Ri represents the distance coordinate of the target on the electronic chart, which is calculated by normalizing the difference between the origin coordinates of the navigation ship and the X-axis direction in the Cartesian coordinate system of the electronic chart; θi represents the target azimuth coordinate, which is adjusted by the difference in the Y-axis direction and then increased by a 90° offset; the λ parameter is used to compensate for the coordinate rotation error caused by the radar installation angle or environmental factors. Specifically, the original distance data scanned by the radar is scaled by (Xi - X1) / (X2 - X1) to eliminate the influence of the different radar scanning ranges and the scale differences of the electronic chart; the azimuth angle in the radar polar coordinate system is converted to the angle of the Cartesian coordinate system of the electronic chart by Yi - Y1 + 90° to ensure that the azimuth display is consistent with the actual geographical azimuth.

[0102] Specifically, when the original coordinates of the target scanned by the radar are (X1, Y1), it corresponds to the origin of the electronic chart to avoid coordinate misalignment; the laser coordinate rotation parameter λ is dynamically adjusted according to the relative angle between the radar and the electronic chart coordinate system. For example, when the radar is installed with an inclination or the ship's attitude changes, λ is corrected through preset calibration data or real-time sensor feedback to ensure that the rotation error is eliminated during the coordinate conversion process. Through normalization processing and angle offset compensation, the coordinates of the target on the electronic chart can accurately reflect the actual position and improve the accuracy of the navigation display.

[0103] As a preferred embodiment, the solution of the present application is specifically implemented as follows:

[0104] The radar data fusion module matches the distance and azimuth information of the target obtained by radar scanning with the coordinates on the electronic chart. First, the radar scans to obtain the distance and azimuth information of the target. Then, this information is matched with the coordinate system on the electronic chart. During the matching process, a Cartesian coordinate system is established with the navigation ship as the coordinate origin.

[0105] For the calculation of the coordinates of the target on the electronic chart, the following formula is used:

[0106] Ri = λ + (Xi - X1) / (X2 - X1) θi = Yi - Y1 + 90°

[0107] Among them, R refers to the origin of coordinates of the navigating ship, and X and Y are the Cartesian coordinates of the electronic nautical chart. When Xi = X1 and Yi = Y1, the target coordinate is the origin of the radar coordinate. λ is the laser coordinate rotation parameter, which is used to adjust the rotation of the coordinate system.

[0108] During specific implementation, the radar data fusion module receives the target distance and azimuth information from the radar data processing unit. At the same time, it obtains the current position of the ship on the electronic nautical chart from the ship positioning system. These two sets of data are input into the coordinate conversion algorithm to calculate the precise coordinates of the target on the electronic nautical chart.

[0109] After the calculation is completed, the radar data fusion module transmits the result to the comprehensive display module to mark the position of the target on the electronic nautical chart. In this way, the crew can intuitively see the relative position relationship between the surrounding targets and the ship.

[0110] Through the above technical solution, the present application realizes the precise matching of the radar scan data and the electronic nautical chart coordinate system. This matching improves the perception accuracy of the ship navigation system for the surrounding environment. Thus, the crew can more accurately judge the positions of the surrounding targets and effectively avoid collision risks. Further, this solution simplifies the data processing flow and reduces the computing burden of the system by unifying different coordinate systems. This optimization not only improves the real-time performance of the navigation system but also enhances the reliability of the system in complex sea conditions.

[0111] In some of the above solutions of the present application, the existing ship navigation system has problems such as poor multi-source data synchronization, insufficient fusion processing ability, and single human-computer interaction method, resulting in low information integration efficiency and limited operation flexibility.

[0112] The present application further proposes a multi-modal perception fusion ship navigation system, including a radar data processing unit, a radar data fusion module, a radar sensor, a radar data fusion unit, a positioning system, a controller, a graphics display library, a comprehensive display module, a human-computer interaction system, a voice recognition engine, a sonar sensor, and a display module. The output of the radar sensor is connected to the radar data processing unit, the output of the radar data processing unit is connected to the radar data fusion module, the positioning system is connected to the controller, the output of the controller is connected to the radar data fusion module, the radar data fusion module is connected to the radar data fusion unit, the radar data fusion unit is connected to the comprehensive display module, the comprehensive display module is also connected to the graphics display library, the display module is connected to the human-computer interaction system, the human-computer interaction system includes a video recognition module and a voice recognition engine, both the video recognition module and the voice recognition engine are connected to the controller, the voice recognition engine is connected to the comprehensive display module, the output of the sonar sensor is connected to the controller, and the output of the human-computer interaction system is connected to the comprehensive display module.

[0113] Among them, the radar sensor transmits the collected raw data to the radar data processing unit. After calculating the target azimuth and distance information through the echo processing algorithm, it outputs to the radar data fusion module. The positioning system is connected to the controller to achieve the real-time transmission of quantum positioning information and the ship's motion state. The controller adjusts the matching parameters of the radar data fusion module according to the positioning data. The radar data fusion unit transmits the fused target data, position information, and graphic display data to the comprehensive display module. The comprehensive display module calls the route parameters in the graphic display library to generate a display screen and outputs it through the display module. The video recognition module and the voice recognition engine in the human-computer interaction system process gesture images and voice signals respectively. The recognized control instructions are parsed by the controller and trigger corresponding tasks. The underwater acoustic signals collected by the sonar sensor are preprocessed by the controller and then transmitted to the radar data fusion module for joint analysis.

[0114] Specifically, after receiving the azimuth and distance data from the radar data processing unit, the radar data fusion module performs a matching operation with the electronic chart coordinates and maps the radar scan results to the electronic chart coordinate system through the coordinate conversion formula. The controller dynamically corrects the fusion weights of the radar and sonar data according to the quantum position information provided by the positioning system to ensure the spatio-temporal consistency of multi-source data. The comprehensive display module automatically switches the display screen mode according to the current task type, calls the route drawing algorithm in the graphic display library to generate a real-time navigation view, and presents it in the display module. The human-computer interaction system processes gesture and voice commands in parallel, distributes the decomposed text commands to the radar data fusion unit or the display module through the controller, and realizes the rapid response of operation instructions. The underwater acoustic signals output by the sonar sensor are filtered and amplified, and then fused with the radar data in the controller. The target azimuth angle parameters are calculated through the azimuth scanning mechanism and finally integrated into the comprehensive display module for three-dimensional environment reconstruction.

[0115] As a preferred embodiment, the solution of the present application is specifically implemented as follows: Radar sensors are installed on both sides of the ship's deck and at the bow position, and their output interfaces are connected to the radar data processing unit through a data bus. A digital signal processor is configured inside the radar data processing unit to perform noise suppression and signal enhancement processing on the original radar echoes. The processed target azimuth and distance data are transmitted to the radar data fusion module through the Ethernet protocol. The positioning system includes a quantum positioning receiver and an inertial measurement unit, and their output ends are connected to the serial communication interface of the controller. The controller uses a multi-threaded architecture to parse the position data in real time and generate control instructions. The radar data fusion module matches the radar target information with the electronic chart coordinates through a data fusion algorithm, and the fused data packet is transmitted to the radar data fusion unit through an optical fiber for cache optimization. The integrated display module integrates a graphics rendering engine, calls a 3D chart model and a route layer from the graphics display library, and generates a dynamic navigation interface in combination with the real-time fusion data. The human-computer interaction system includes a binocular camera array and a microphone array. The video recognition module uses a convolutional neural network to parse gesture actions, and the speech recognition engine converts speech commands into control codes through an acoustic model. The sonar sensor is arranged in the keel area of the ship's bottom, and its output signal is input into the dedicated acquisition card of the controller after analog-to-digital conversion. The controller synchronously sends the processed sonar data and radar data to the integrated display module for superimposed display.

[0116] Through the above technical solution, the present application solves the problems of difficult multi-sensor data synchronization and low human-computer interaction efficiency in the existing ship navigation system. The parallel processing and fusion mechanism of radar and sonar data ensures the integrity and consistency of environmental perception information, and the collaborative work of quantum positioning and inertial measurement improves the real-time performance of ship position update. The deep integration of the graphics display library and the integrated display module realizes the dynamic visualization of multi-source data on the 3D chart, and the modular human-computer interaction system significantly improves the operation response speed and the parsing accuracy of control instructions through natural interaction methods. The standardized interface design between functional units ensures the system scalability and the plug-and-play ability of different sensor data.

[0117] In some of the above solutions of the present application, although the multi-modal perception technology realizes environmental perception and data fusion by integrating devices such as radar, sonar, and positioning systems, in the actual system construction process, the connection relationship between components and the data flow path lack clear definitions, resulting in difficulties in achieving efficient cooperation between the sensor data processing unit, the data fusion module, and the display module, limited response speed of data real-time update and task invocation, and insufficient linkage between the human-computer interaction system and the core processing unit, affecting the operation efficiency.

[0118] The present application further proposes a multimodal perception fusion ship navigation system, including a radar data processing unit, a radar data fusion module, a radar sensor, a radar data fusion unit, a positioning system, a controller, a graphics display library, a comprehensive display module, a human-computer interaction system, a voice recognition engine, a sonar sensor, and a display module.

[0119] Among them, the radar sensor is directly connected to the input port of the radar data processing unit through an output port. The output port of the radar data processing unit is connected to the input port of the radar data fusion module through a high-speed data bus. The positioning system is connected to the controller through a serial communication interface. The output port of the controller is connected to the control instruction input end of the radar data fusion module through a parallel data channel. The output port of the radar data fusion module is connected to the data receiving port of the radar data fusion unit through a data exchange unit. The data output port of the radar data fusion unit is connected to the data processing core of the comprehensive display module through a real-time transport protocol. The display data output port of the comprehensive display module is connected to the input port of the display module through a video interface. The human-computer interaction system includes an independently operating video recognition module and a voice recognition engine. The video recognition module is connected to the image processing unit of the controller through an image acquisition card. The voice recognition engine is connected to the voice processing unit of the controller through an audio input interface. The output port of the sonar sensor is connected to the data acquisition port of the controller through an analog signal converter.

[0120] Specifically, the original radar signal collected by the radar sensor is transmitted to the radar data processing unit through a dedicated interface. This unit executes an echo processing algorithm to generate target azimuth and distance data. The processed data is sent to the radar data fusion module through a high-speed bus and matched with the ship position information provided by the positioning system. The positioning system sends the ship coordinates to the radar data fusion module in real time through the controller. The coordinate mapping algorithm is used in the module to align the radar data with the electronic chart coordinate system. The fused data is subjected to standardization processing by the radar data fusion unit. The comprehensive display module calls the route drawing algorithm in the graphics display library to generate a visual interface and renders it in real time through the display module. In the human-computer interaction system, the video recognition module collects gesture images through a camera, sends them to the controller for action classification after feature extraction. The voice recognition engine converts the audio signal into a text instruction, which is parsed by the controller to trigger task invocation. The underwater acoustic signal collected by the sonar sensor is preprocessed and then transmitted to the controller. The controller completes signal amplification and target recognition and finally synchronizes the recognition result to the comprehensive display module. The connection method between components adopts a hierarchical architecture, and the data flow path realizes low-latency transmission through a preset protocol to ensure real-time synchronization and efficient processing of multi-source data.

[0121] As a preferred embodiment, the solution of the present application is specifically implemented as follows: The computer device includes a data processing unit composed of an Intel Xeon E5-2678 processor and 32GB of DDR4 memory. A solid-state drive is used as a storage medium to store an executable program containing a multi-modal perception fusion algorithm. When the processor runs, it loads program instructions from the memory, executes a radar signal synchronization calibration algorithm to generate a synchronization control signal, and sends a synchronization trigger pulse to a multi-channel radar device through a PCIe bus. Quantum positioning data is obtained by a serial port receiving module, and after decoding, it is jointly input into a Kalman filtering module with the attitude data output by an on-board gyroscope for fusion operation. Radar echo data is transmitted to a data fusion buffer through a DMA channel after FFT transformation processing, and sonar signals are processed by a wavelet denoising algorithm and then spatio-temporally aligned with radar data through a shared memory area. Human-computer interaction instructions are received through a USB interface, and a voice recognition thread and a gesture recognition thread run in parallel. The recognition results are stored in a circular buffer for the main control thread to call. A graphics rendering engine calls OpenGL library functions according to the fused target coordinate data to generate a three-dimensional navigation interface, which is output to an on-board display screen through an HDMI interface.

[0122] Through the above technical solution, the present application realizes the efficient synchronous acquisition and real-time fusion processing of multi-modal perception data, and solves the positioning deviation problem caused by asynchronous data sources in traditional navigation systems. The hardware-accelerated data processing architecture significantly reduces the latency of multi-sensor information fusion, effectively improving the prediction accuracy of obstacle trajectories in a dynamic environment. Through the modular program architecture design, the parallel execution of core functions such as quantum positioning, radar echo processing, and sonar signal analysis is ensured, enhancing the operational stability of the system in a complex electromagnetic interference environment. The integrated human-computer interaction instruction processing mechanism effectively reduces the operation response time, controlling the transmission delay of control instructions of the navigation system within milliseconds.

[0123] In some of the above solutions of the present application, although various multi-modal perception fusion ship navigation methods are proposed, in practical applications, there are still deficiencies in the program execution efficiency of the computer-readable storage medium and the real-time performance of data processing. Especially in a complex marine environment, the rapid invocation and stable operation of multi-sensor data fusion algorithms face challenges.

[0124] The present application further proposes a computer-readable storage medium storing a computer program for executing a multi-modal perception fusion ship navigation method.

[0125] Among them, when the computer program is executed by the processor, the following steps are implemented: calibrating the synchronization signals of the multi-channel radar equipment to keep the multi-channel radar synchronized; obtaining the ship position information in real time through the positioning system, and combining the quantum positioning system with the on-board sensor equipment to collect the motion state; calculating the target azimuth and distance information through the echo processing algorithm of the radar data processing unit; processing the signals received by the transducer of the sonar detection, filtering out the echo interference and amplifying the signals; identifying gestures and voice commands through the human-computer interaction unit and decomposing them into text instructions; fusing the radar target data, position information and graphic display data, updating and transmitting them to the display module in real time; calling the parameters of the graphic display library according to the task to draw the route and presenting it through the display module.

[0126] Specifically, after the computer program is loaded into the processor, it first calls the radar synchronization signal calibration module to ensure the time synchronization of the multi-channel radar and avoid the fusion error caused by the data acquisition time delay. Further, the program parses the ship position information through the light pulse sequence sending and receiving logic of the quantum positioning system, combines the light pulse signals at the receiving end, and associates the positioning data with the speed and heading parameters collected by the on-board sensors to form a dynamic position update link. In the radar data processing stage, when the program executes the echo processing algorithm, it generates the target coordinates according to the preset azimuth and distance calculation model, and matches and maps them with the electronic chart coordinates through the data fusion module. For sonar signal processing, the program calls the filtering preprocessing function to filter out the background noise according to the frequency-related parameters, and adjusts the echo signal strength through the amplifier gain parameter to ensure the accuracy of underwater target recognition. After the voice and gesture recognition instructions of the human-computer interaction module are decomposed into text, the program matches the task execution logic according to the text keywords. For example, it calls the route drawing function in the graphic display library and generates a navigation screen in combination with the current fusion data. The display module superimposes the fused azimuth, distance and electronic chart data through the real-time rendering engine and finally outputs it to the display. Through the automatic execution of the above steps, the program solves the real-time problem of multi-modal data fusion and the algorithm call efficiency problem, and improves the stability and response speed of the navigation system in complex environments.

[0127] As a preferred embodiment, the solution of the present application is specifically implemented as follows: The computer-readable storage medium uses a solid-state drive as a carrier, and the computer program stored therein includes an executable instruction set. The program is configured to perform the following operations when running on a processor: calibrate the synchronization signals of multiple radar devices to control the time reference error of each radar device within the microsecond level; generate an optical pulse sequence through a quantum positioning system, and parse the signals collected by the receiving end into the longitude and latitude coordinates of the ship; call an echo processing algorithm to calculate the target azimuth and distance information in the radar data, and generate a standardized data packet containing the target coordinates; input the underwater acoustic signals collected by the sonar transducer into a preprocessing module for frequency-domain filtering to eliminate the interference of the DC component with a frequency of 0; perform feature matching on the gesture contour through an image recognition model, and perform text conversion on the voice signal through an acoustic model to generate executable navigation control instructions; perform coordinate transformation and matching between the radar target data and the electronic chart coordinate system to generate a dynamic route trajectory superimposed on the display screen.

[0128] Through the above technical solution, the present application realizes the precise synchronous acquisition and standardized processing of multi-source perception data, and solves the problem that traditional storage media cannot support the real-time fusion of multi-modal data. Through the optimized configuration of program instructions, the time alignment accuracy between the quantum positioning signal and the radar echo data is guaranteed, and the target coordinate offset caused by timing deviation is avoided. Further, based on the preset filtering algorithm in the program, the low-frequency interference in the sonar signal is effectively suppressed, and the reliability of underwater obstacle detection is improved. By solidifying the interactive instruction parsing logic in the storage medium, the parallel processing ability of voice and gesture control instructions is realized, and the operation response efficiency in complex sea conditions is enhanced.

[0129] The above are only the embodiments of the present application and are not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multimodal perception fusion ship navigation method, characterized in that It includes the following steps: S1. Calibrate the synchronization signals of the ship's multi-channel radar equipment to keep the multi-channel radars synchronized at the same moment; S2. Obtain the ship's position information in real time through the positioning system, position the ship with the help of the quantum positioning system, and obtain the ship's motion state with the help of the on-board sensor equipment; S3. The radar data processing unit calculates the azimuth and distance information of the surrounding environmental targets and positioning targets through the echo processing algorithm, and transmits the obtained data to the radar data fusion module; S4. The transducer of sonar detection receives the signals formed by underwater acoustic propagation and background noise signals, and outputs them to the ship's sonar integrated processing unit for processing; S5. Identify gestures and voice commands through the human-computer interaction unit, decompose the commands into text after identification, and perform corresponding task calls according to the text; S6. The data fusion processing unit updates the radar target data, position information data, and graphic display data in real time after fusion, and transmits them to the display module; S7. Call the corresponding data fusion display screen according to the current task, obtain the corresponding data from the data fusion unit, call the graphic display library to draw the route according to the screen setting parameters, and present it on the display through the display module.

2. The multimodal perception fusion ship navigation method according to claim 1, wherein The steps for obtaining the ship's position information in real time through the positioning system are: S21. Measure the distance between the transmitter and receiver of the quantum positioning system according to actual requirements; S22. Send the quantum signal in the form of an optical pulse sequence; S23. Receive the signal according to the receiver carried by the crew; S24. Determine the quantum position information according to the received optical pulse and transmit it in real time.

3. The multimodal perception fusion ship navigation method according to claim 1, wherein, The steps for identifying gestures and voice commands through the human-computer interaction system are: S51. Obtain the image or audio signal from the human-computer interaction device; S52. The image processing module performs gesture recognition through feature extraction and recognition classification; S53. The voice recognition engine recognizes the received voice signal to obtain the corresponding text.

4. The multimodal perception fusion ship navigation method according to claim 1, wherein The processing steps for the signals received by the transducer of sonar detection are: S41. Perform filtering preprocessing to filter out echo interference and reduce signal distortion; the filtering function is a frequency-related function, f is the frequency of the signal to be filtered, f0 is the filtering center frequency, where A and μ are the filtering function parameters; for the signal with the frequency to be filtered f = 0, the filtering function is to calculate the signal angle and distance information as: θ = arctan(In / Im + π / 2), where In is the signal received inside the radar and Im is the signal received by the radar; S42. Amplify the signal, and through signal amplification, realize the amplification of the echo signal; the echo signal X is related to the amplification gain K and the noise N, and the amplified signal is Y = KX + N, where the amplification gain K can be set by controlling the amplifier parameters, and the noise N is the noise generated by the amplifier; S43. Identify the distance and azimuth information of the positioning target and the surrounding environmental targets according to the received echo signal.

5. The multimodal perception fusion ship navigation method according to claim 4, wherein After preprocessing, for the echo signal, it is received by an echo transducer. The output voltage of the transducer is \(U_i = 2\pi fC_0I_i\), where \(C_0\) is the capacitance characteristic and \(I_i\) is the output current of the echo transducer. The echo signal is processed and amplified by an amplifier, and the output signal is \(U_a\).

6. The multimodal perception fusion ship navigation method according to claim 5, characterized in that, The echo transducer and the azimuth scanning mechanism are installed on the detection head. The azimuth scanning mechanism can rotate, and the azimuth of the target acoustic wave emission is calculated by rotating the echo transducer. The azimuth scanning mechanism is driven by a stepper motor, and the azimuth angle step value is calculated using the following formula: \(\theta_{min}=\arcsin(1 / 2X_{max}D / F)\) \(\theta = D / F-\theta_{min}\) \(\theta_{max}=D / F-\theta_{min}\) where \(\theta\) is the echo angle, \(D\) is the maximum distance of the radar echo, \(X_{max}\) is the maximum amplitude of the echo, \(F\) is the transmission frequency, \(\theta_{min}\) is the minimum value of azimuth change, \(s\) is the step amount of the stepper motor, and \(\theta_{max}\) is the maximum value of azimuth change.

7. The multimodal perception fusion ship navigation method according to claim 1, characterized in that The radar data fusion module matches the distance and azimuth information of the target obtained by radar scanning with the coordinates on the electronic chart to obtain the coordinates of the target on the electronic chart: \(R_i=\lambda+(X_i - X_1) / (X_2 - X_1)\) \(\theta_i=Y_i - Y_1 + 90^{\circ}\) where \(R\) refers to the navigation ship as the coordinate origin, \(X\) and \(Y\) are the Cartesian coordinates of the electronic chart. Taking the navigation ship as the coordinate origin, the azimuth and distance coordinates of the target on the electronic chart are calculated. When \(X_i = X_1\) and \(Y_i = Y_1\), the target coordinates are the radar coordinate origin, and \(\lambda\) is the laser coordinate rotation parameter.

8. A multimodal perception fusion ship navigation system, characterized in that, It includes a radar data processing unit, a radar data fusion module, a radar sensor, a radar data fusion unit, a positioning system, a controller, a graphics display library, an integrated display module, a human-computer interaction system, a voice recognition engine, a sonar sensor, and a display module. The output of the radar sensor is connected to the radar data processing unit, the output of the radar data processing unit is connected to the radar data fusion module, the positioning system is connected to the controller, the output of the controller is connected to the radar data fusion module, the radar data fusion module is connected to the radar data fusion unit, the radar data fusion unit is connected to the integrated display module, the integrated display module is also connected to the graphics display library, the display module, and the human-computer interaction system. The human-computer interaction system includes a video recognition module and a voice recognition engine. Both the video recognition module and the voice recognition engine are connected to the controller. The voice recognition engine is connected to the integrated display module. The output of the sonar sensor is connected to the controller, and the output of the human-computer interaction system is connected to the integrated display module.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 7.

Images