Underwater robot fault diagnosis method based on online fine-tuning hybrid neural network

Abstract

The invention relates to an underwater robot technology, and aims to provide an underwater robot fault diagnosis method based on an online fine-tuning hybrid neural network. Comprising the following steps: performing normalization processing on collected state data in a navigation process of an underwater robot, and inputting the state data into a deep convolutional neural network to perform spatio-temporal feature extraction; the output data combination is converted into a pulse sequence in a Poisson coding mode, and the pulse sequence serves as the input of a pulse neural network; counting the excitation number of neurons in the spiking neural network, and determining and outputting a fault diagnosis result according to the fault mark type matched with the excitation number; parameters of the deep convolutional neural network are kept unchanged in the whole diagnosis process, and online fine tuning of the whole hybrid network model is realized based on unsupervised adaptive updating of the pulse neural network model. The method is suitable for low-frequency sampling of underwater robot data, and the accuracy of fault diagnosis is improved; the model can be dynamically adjusted according to actual working conditions, and the environmental adaptability of diagnosis is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater robots, and in particular to an underwater robot fault diagnosis method based on an online fine-tunable hybrid neural network. Background Art

[0002] As an essential marine operational tool, underwater robots (AUVs) are widely used in critical areas such as marine resource exploration, submarine pipeline inspection, and underwater rescue. In recent years, with the increasing complexity and depth of marine development tasks, the reliability of AUV systems has faced significant challenges. Due to the unique characteristics of the underwater environment, such as high pressure, severe corrosion, and complex hydrological conditions, AUVs are prone to various failures. These failures can impact operational efficiency and mission quality, or even lead to the robot's failure or a major safety incident. Therefore, establishing an efficient and reliable AUV fault diagnosis system is of great practical significance.

[0003] Currently, mainstream underwater robot fault diagnosis methods rely primarily on traditional data-driven approaches or fixed-structure neural network models. However, these methods have significant limitations in practical applications. First, due to the dynamic and variable nature of the underwater environment, pre-trained diagnostic models often struggle to adapt to the various operating conditions in actual operating environments. Second, existing diagnostic methods mostly rely on offline training or network parameter transfer, which involves re-collecting new datasets during actual applications. These methods are unable to timely learn and adapt to emerging or evolving fault modes. Furthermore, underwater robot data is sampled at a low frequency, while existing neural network diagnostic models are designed for high-frequency sampling and therefore perform poorly with low-frequency data. Finally, single-structure neural network models often struggle to fully exploit the correlations between data features when processing complex, multi-source, and heterogeneous underwater robot data, impacting the accuracy and reliability of diagnosis.

[0004] Therefore, it is urgent to propose a new fault diagnosis method that can adapt to complex underwater environments and has online learning capabilities to meet the actual needs of remote intelligent operation and maintenance of underwater robots. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the deficiencies in the prior art and propose an underwater robot fault diagnosis method based on an online fine-tunable hybrid neural network.

[0006] To solve the technical problem, the solution of the present invention is:

[0007] Provided is a fault diagnosis method for an underwater robot based on an online fine-tuning hybrid neural network. The hybrid neural network includes a multi-core deep convolutional neural network with an attention mechanism, and a spiking neural network that adopts a biologically inspired spike-timing-dependent plasticity learning rule, a neuron lateral inhibition mechanism, and a dynamic membrane potential balance mechanism.

[0008] When performing fault diagnosis, first normalize the state data collected during the navigation of the underwater robot, and then input it into the deep convolutional neural network for spatio-temporal feature extraction. Convert the output data combination into a spike train through Poisson encoding, and use it as the input of the spiking neural network. Count the number of neuron firings in the spiking neural network, and determine and output the fault diagnosis result according to the matching fault label type.

[0009] During the entire diagnosis process, keep the parameters of the deep convolutional neural network fixed, and realize the online fine-tuning of the overall hybrid network model based on the unsupervised adaptive update of the spiking neural network model.

[0010] As a preferred solution of the present invention, it includes: before using the hybrid neural network, train it in the following manner:

[0011] (2.1) Collect various state data of the underwater robot during normal operation and under different fault conditions, use them as training data, and establish a training set and a validation set.

[0012] (2.2) Use the training set to train a separate deep convolutional neural network, retain all parameters except the linear layer in the optimal model, and then transfer the parameters to the deep convolutional neural network with the same structure in the hybrid network model.

[0013] (2.3) Input the state data of the validation set into the hybrid network model, and then use the output of the deep convolutional neural network to tune the parameters of the spiking neural network. Count the number of neuron firings in the processing layer of the spiking neural network, and mark the neuron with the most firings according to the operating state or fault condition corresponding to the state data in the validation set. After sufficient data volume training and validation, obtain a hybrid neural network model for diagnosis.

[0014] As a preferred solution of the present invention, the calculation process in the deep convolutional neural network includes:

[0015] (3.1) In multiple convolutional modules, the output of the c-th output channel in the l-th convolutional layer at the spatial position (i, j) is calculated by the following formula:

[0016]

[0017] Where Denote the feature map value at position (i, j) of the c-th output channel in the l-th layer; Denote the weight value from the d-th input channel to the c-th output channel at position (m, n); Denote the input feature map of the previous layer, where s is the stride and p is the padding size; Is the bias term of the c-th output channel; the dimension of the output feature map is H l ×W l ×C l , the dimension of the convolutional kernel is K×K×C (l-1) ×C l , the dimension of the bias term is C l ;

[0018] (3.2) In the convolution and leaky rectified linear unit module, define the activation function as follows:

[0019]

[0020] where α represents the slope of the negative input, with a value of 0.01; y represents the input data;

[0021] (3.3) In the max pooling module, the max pooling operation of the l-th layer is defined as follows:

[0022]

[0023] where the maximum value is obtained within a P×P pooling window, Denote the feature map value at position (i, j) of the c-th output channel after max pooling; Denote the output value after pooling; the output dimension is H l ×W l ×C l ;

[0024] (3.4) In the attention mechanism module, it includes three operations: squeeze, excitation, and recalibration; among them,

[0025] The squeeze operation uses global average pooling to generate channel statistics z∈R^C, and the calculation formula is as follows:

[0026]

[0027] where x c(i,j) Denote the feature map value at the spatial position (i, j) of channel c; the input feature map X∈R H×W×C , H, W, and C represent height, width, and number of channels respectively, and F sq Is the squeeze function;

[0028] The excitation operation is defined as:

[0029] s = F ex(z,W) = σ(W2 · δ(W1 · z))

[0030]

[0031] where W1 and W2 are the weights of two fully connected layers; δ is the rectified linear unit activation function, σ is the Sigmoid activation function; z represents the final channel weight vector; r represents the dimensionality reduction ratio; F ex represents the activation function;

[0032] The calculation formula for the recalibration operation is as follows:

[0033] X c ′ = s c · X c , c = 1, 2, …, C

[0034] X ′ ∈ R H×W×C

[0035] where s c is the channel weight obtained from the activation operation, X c represents the feature map of channel c; X ′ represents the feature map after the recalibration operation;

[0036] (3.5) In the tiling module, the obtained multi-channel two-dimensional matrix data is reconstructed into a single vector data.

[0037] As a preferred embodiment of the present invention, the Poisson coding means encoding real values into a pulse sequence according to the Poisson distribution; the number of rows of the pulse sequence is equal to the number of column vectors in the corresponding data combination, and the length of the sequence is determined by the encoding fundamental frequency;

[0038] Specifically, the following formula is followed during encoding:

[0039]

[0040] In the formula, P is the probability of the occurrence of time, specifically referring to the probability of emitting a pulse here; X is the event; x is the actual number of occurrences of the event; λ is the average number of pulses sent per unit time.

[0041] As a preferred embodiment of the present invention, in the spiking neural network, the neuron model of the input layer fires spikes to the processing layer in the next layer according to different spike trains; the synaptic model between the input layer and the processing layer follows the bio-inspired spike timing and performs unsupervised online learning according to the spike-timing-dependent plasticity rule, and conducts online learning based on the arrival times of the pre- and post-synaptic spikes; after receiving the fired spikes, the processing layer accumulates the membrane potential, and when the threshold potential is exceeded, it fires a spike signal to the inhibitory layer in the third layer, and at the same time, a dynamic membrane potential balancing mechanism is used to adjust the threshold potential; the inhibitory layer adopts a neuron lateral inhibition mechanism, and after receiving the spike signal of an excitatory neuron, it sends an inhibitory spike signal back to the remaining excitatory neurons to inhibit their membrane potentials.

[0042] As a preferred embodiment of the present invention, in the spiking neural network, the neuron model adopted by the input layer is a leaky integrate-and-fire neuron model, and its mathematical model is as follows:

[0043]

[0044] S = (sgn(V - E th ) + 1) / 2

[0045] V = S·E rest + (1 - S)·V

[0046] Wherein, V is the neuron membrane potential; t is the time; E rest is the resting potential; E exc and E inh are the equilibrium membrane potentials of the excitatory neuron and the inhibitory neuron respectively; g e and g i are the conductance values connecting the excitatory / inhibitory neuron synapses, i.e., the network weights; E th is the threshold membrane potential of the neuron; S is the signal indicating whether a spike is emitted or not; sgn(*) is the sign function; τ is the membrane potential time constant;

[0047] After receiving the pre-synaptic spike signal encoding the input, the neuron model accumulates the membrane potential, and different encoded spike trains will generate different neuron dynamic behaviors; when the neuron membrane potential exceeds the threshold membrane potential, a spike signal is emitted, and the neuron membrane potential returns to the resting potential, otherwise the membrane potential continues to accumulate.

[0048] As a preferred embodiment of the present invention, for the unsupervised online learning of the spiking neural network following the bio-inspired spike-timing-dependent plasticity rule, the synaptic value of the synaptic model is updated online according to the arrival times of the pre- and post-synaptic spikes; specifically, it is calculated according to the following formula:

[0049]

[0050] a pre / post = a pre / post + A pre / post

[0051] g = g + a post / pre

[0052] where τ pre / post is the time constant of the pre / post synaptic change; a pre / post is the trajectory representing the pre / post synaptic change; A pre / post is the change rate of the pre / post synaptic; g is the synaptic conductance, i.e., the network weight.

[0053] As a preferred embodiment of the present invention, the dynamic membrane potential balance mechanism refers to calculating the neuron threshold membrane potential according to the following formula:

[0054] E th = E base + θ

[0055]

[0056] θ = θ + S· +

[0057] where E base is the reference potential of the threshold membrane potential; θ is the dynamic potential of the threshold membrane potential; τ th is the time constant of the dynamic potential of the threshold membrane potential; θ + is the increase amount of the dynamic potential; S is the signal of whether the above pulse is emitted or not;

[0058] When there is no pulse excitation, the dynamic potential of the threshold membrane potential will gradually decay, i.e., the threshold decreases; when there is pulse excitation, the dynamic potential will gradually increase, i.e., the threshold increases; the dynamic adjustment of the neuron threshold membrane potential is realized in this way.

[0059] The present invention further provides an underwater robot fault diagnosis system based on an online fine-tunable hybrid neural network, including:

[0060] Several sensors installed on the underwater robot, used to collect the state data generated during the operation of the underwater robot;

[0061] A signal processing and transmission device, used to collect the state data collected by the sensors, and after preprocessing, transmit it to the processor or controller through a serial port or a bus;

[0062] A memory, used to store the software function modules and instructions for implementing the foregoing method; the hybrid neural network is deployed in the software function modules;

[0063] A processor or controller, configured to call software function modules from the memory to enable the implementation of the method using state data when executing the instructions, and finally obtain a diagnostic result of the operation failure of the underwater robot.

[0064] As a preferred solution of the present invention, the software function modules include a data preprocessing module, a deep convolutional neural network module, and a spiking neural network module; among them, the deep convolutional neural network includes multiple convolutional modules, as well as a convolutional and leaky rectified linear unit module, a max pooling module, an attention mechanism module, and a flattening module; the spiking neural network module includes an input layer, a processing layer, and an inhibitory layer, and each layer includes a neuron model;

[0065] The sensors are attitude sensors, voltage sensors, and depth sensors installed on the underwater robot; the state data collected by the sensors at least includes: thruster control signals, underwater robot power supply voltage signals, water depth values, robot attitude angles, robot body coordinate accelerations, and robot three-axis rotational angular velocities.

[0066] Description of the invention principle:

[0067] The present invention proposes a fault diagnosis method for an underwater robot with an online fine-tuning hybrid neural network to solve the diagnosis problem under dynamically changing fault states and uncertain conditions. The model innovatively fuses a deep convolutional neural network and a spiking neural network to construct a novel hybrid architecture.

[0068] Specifically, the deep convolutional neural network part uses multi-scale convolutional kernels to extract the features of the underwater robot state data, and introduces an attention mechanism to reconstruct the feature channels to highlight key feature information. At the same time, the present invention uses a spiking neural network composed of leaky integrate-and-fire neurons to replace the linear layer of the traditional deep convolutional neural network as a classifier, and realizes effective online fine-tuning by introducing spike-time-dependent plasticity learning rules, neuron lateral inhibition mechanisms, and membrane potential balance mechanisms. To realize the data conversion between the deep convolutional neural network part and the spiking neural network part, the present invention designs a pulse coding scheme based on Poisson distribution.

[0069] Compared with the prior art, the present invention has the following beneficial effects and advantages:

[0070] 1. The hybrid neural network architecture gives full play to the feature extraction ability of the deep convolutional neural network and the biologically inspired learning characteristics of the spiking neural network, is particularly suitable for low-frequency sampling of underwater robot data, and improves the accuracy of fault diagnosis.

[0071] 2. Regarding the online fine-tuning mechanism introduced by the spiking neural network, the model can be dynamically adjusted according to the actual working conditions, improving the environmental adaptability of the diagnosis.

[0072] 3. The combination of multi-scale feature extraction and attention mechanism in the deep convolutional neural network enhances the model's ability to identify key fault features.

[0073] 4. Based on the learning rule of biological synaptic plasticity in the spiking neural network, the model has the ability of continuous learning and can cope with new and uncertain fault modes. Brief Description of the Drawings

[0074] Figure 1 It is a system block diagram implemented by the present invention.

[0075] Figure 2 It is an architecture diagram of the hybrid network model designed by the present invention. Detailed Implementation Manner

[0076] First of all, it should be noted that the present invention patent involves underwater robot technology, deep learning technology, neural network technology and fault diagnosis technology. During the implementation of the invention patent, the application of multiple software and hardware functional modules may be involved. The applicant believes that after carefully reading the application documents and accurately understanding the implementation principle and purpose of the present invention patent, and in combination with the existing well-known technologies, those skilled in the art can fully implement the present invention patent by using the technologies they have mastered. All that are mentioned in the application documents of the present invention patent belong to the scope, and the applicant will not list them one by one.

[0077] In addition, the implementation of the present invention patent depends on the application of a variety of computers and boards, and these instruments are all existing technologies and there are mature products available for purchase in the market. Secondly, using application-specific integrated circuits (including but not limited to ASIC) or field-programmable gate arrays (including but not limited to FPGA) to deploy the fault diagnosis method described in the present invention will help to achieve hardware acceleration. Implementing a high-speed and low-power fault diagnosis system through ASIC or FPGA also belongs to the technical features of the present invention. Furthermore, the fault diagnosis methods and solutions described in the present invention are not only applicable to objects such as ocean robots, ocean vehicles, and submersibles, but also applicable to other fields, including but not limited to machinery, medical, health, information technology, robots, information systems, electricity, transportation, energy, nuclear power, underwater facilities, hydropower, thermal power, pipelines, environmental monitoring and protection, etc.

[0078] It should be noted that the present invention relates to fault diagnosis technology and signal processing technology. During the implementation of the present invention, the principles and applications of multiple basic algorithms may be involved. The applicant believes that after carefully reading the application documents and accurately understanding the implementation principle and purpose of the present invention, and in combination with the existing well-known technologies, those skilled in the art can fully implement the present invention by using their algorithm writing abilities. All that are mentioned in the application documents of the present invention belong to this scope, and the applicant will not list them one by one.

[0079] The implementation process of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0080] For the underwater robot fault diagnosis method based on an online fine-tuning hybrid neural network of the present invention, the hybrid neural network includes a multi-core deep convolutional neural network with an attention mechanism, and a spiking neural network that adopts a spike-timing-dependent plasticity learning rule inspired by biology, a neuron lateral inhibition mechanism, and a dynamic membrane potential balance mechanism; when performing fault diagnosis, first normalize the state data collected during the underwater robot's navigation process, and then input it into the deep convolutional neural network for spatio-temporal feature extraction; convert the output data combination into a spike train through Poisson coding, as the input of the spiking neural network; count the number of neuron firings in the spiking neural network, and determine and output the fault diagnosis result according to the matching fault label type; keep the parameters of the deep convolutional neural network fixed during the entire diagnosis process, and realize the online fine-tuning of the overall hybrid network model based on the unsupervised adaptive update of the spiking neural network model.

[0081] This method runs in an underwater robot fault diagnosis system based on an online fine-tuning hybrid neural network. The system includes: several sensors installed on the underwater robot, used to collect the state data generated during the operation of the underwater robot; a signal processing and transmission device, used to collect the state data collected by the sensors, and after preprocessing, transmit it to the processor or controller through a serial port or bus; a memory, used to store the software function modules and instructions for implementing the method of the present invention; a processor or controller, configured to call the software function modules from the memory, and when executing the instructions, be able to use the state data to implement the method, and finally obtain the diagnosis result of the underwater robot's operation fault.

[0082] The underwater robot fault diagnosis method based on an online fine-tuning hybrid neural network specifically includes the following steps:

[0083] 1. Install sensors on the underwater robot, which may specifically include an attitude sensor, a voltage sensor, and a depth sensor; the state data collected by the sensors at least includes: thruster control signal, underwater robot power supply voltage signal, water depth value, robot attitude angle, robot body coordinate acceleration, and robot three-axis rotation angular velocity.

[0084] 2. Build a hybrid neural network that can run locally in the computing device of the underwater robot.

[0085] The hybrid neural network of the present invention is implemented in the form of software functional modules built in the memory of a computing device, and specifically includes a data preprocessing module, a deep convolutional neural network module, and a spiking neural network module; among them, the deep convolutional neural network includes multiple convolutional modules, as well as a convolutional and leaky rectified linear unit module, a max pooling module, an attention mechanism module, and a tiling module; the spiking neural network module includes an input layer, a processing layer, and an inhibitory layer, and each layer includes a neuron model.

[0086] 3. In the deep convolutional neural network, it includes multiple convolutional modules, a convolutional and leaky rectified linear unit module, a max pooling module, an attention mechanism module, and a tiling module.

[0087] (1) Convolutional module

[0088] The output at the spatial position (i, j) of the c-th output channel in the l-th convolutional layer is calculated by the following formula:

[0089]

[0090] Among them, represents the feature map value at the position (i, j) of the c-th output channel in the l-th layer; represents the weight value from the d-th input channel to the c-th output channel at the position (m, n); represents the input feature map of the previous layer, where s is the stride and p is the padding size; is the bias term of the c-th output channel. The dimension of the output feature map is H l ×W l ×C l , the dimension of the convolutional kernel is K×K×C (l-1) ×C l , the dimension of the bias term is C l .

[0091] Convolutional and leaky rectified linear unit module

[0092] In this module, the operation of convolution is the same as that of the convolutional module, and the leaky rectified linear unit is applied to the network as an activation function. The main difference from the standard rectified linear unit function is that the leaky rectified linear unit allows negative input values to have a small non-zero gradient, and its activation function is defined as follows:

[0093]

[0094] Among them, α represents the slope of the negative input, usually taking a value of 0.01, which makes negative values have a small non-zero gradient, so as to retain negative value information in the network; y represents the input data.

[0095] (2) Max pooling module

[0096] In a deep convolutional neural network, a max pooling layer is adopted to reduce the spatial dimension of the feature map, improve computational efficiency, and reduce the risk of overfitting. The max pooling operation of the l-th layer is defined as follows:

[0097]

[0098] where the maximum value is obtained within a pooling window of P×P, represents the value of the feature map at the position (i, j) in the c-th output channel after max pooling; represents the output value after pooling; the output dimension is H l ×W l ×C l .

[0099] (3) Attention mechanism module

[0100] The attention mechanism enhances the representation ability of the network by modeling the interdependencies between channels. It consists of three operations: Squeeze, Excitation, and Recalibration operations. In a deep convolutional neural network, the attention mechanism adaptively recalibrates the channel-level feature responses, enhancing meaningful features while suppressing less useful features.

[0101] The Squeeze operation uses global average pooling to aggregate the global spatial information of the feature map into a channel descriptor. Given an input feature map X ∈ R H×W×C , where H, W, and C represent height, width, and the number of channels respectively; the Squeeze operation uses global average pooling to generate channel statistics z ∈ R^C, calculated as follows:

[0102]

[0103] where, x c(i,j) represents the value of the feature map at the spatial position (i, j) in channel c, and F sq is the Squeeze function.

[0104] The Excitation operation aims to capture the interdependencies between channels and generate a set of weights for each channel. This is achieved through two fully connected layers, with a rectified linear unit activation in the middle and a Sigmoid function activation at the end to produce the final channel weights. The Excitation operation is defined as:

[0105] s = F ex(z,W) = σ(W2·δ(W1·z))

[0106] where, and are the weights of two fully connected layers, δ is the rectified linear unit activation function, and σ is the Sigmoid activation function. The dimensionality reduction ratio r limits the model complexity and controls the degree of dimensionality reduction; z represents the final channel weight vector; r represents the dimensionality reduction ratio; F ex represents the activation function;

[0107] Finally, the recalibration operation rescales the original feature map X using the learned channel weight s. The recalibrated feature map X ′ ∈R H×W×C is calculated as follows:

[0108] X c ′ = s c ·X c , c = 1, 2, …, C

[0109] X ′ ∈R H×W×C

[0110] where s c is the channel weight obtained from the excitation operation, X c represents the feature map of channel c; X ′ represents the feature map after the recalibration operation. This recalibration enables the network to emphasize informative features while suppressing less relevant features.

[0111] (4) Tiling module

[0112] The tiling module reconstructs the obtained multi-channel two-dimensional matrix data into a single vector data.

[0113] 4. Normalize the obtained vector data for subsequent encoding operations.

[0114] 5. Convert each data combination into a set of pulse sequences through Poisson encoding. The number of rows of each obtained set of pulse sequences is equal to the number of column vectors in the corresponding data combination, and the length is determined by the encoding base frequency;

[0115] The Poisson encoding refers to an encoding method that encodes real values into pulse sequences according to the Poisson distribution. The encoding follows the following formula:

[0116]

[0117] where λ is the average number of pulses sent per unit time; X is the event; x is the actual number of times the event occurs, and this pulse signal is converted from the real number signal output by the prefrontal deep convolutional neural network; P is the probability of the event occurring, specifically referring to the probability of emitting a pulse here.

[0118] 6. In a spiking neural network, the neuron model in the input layer fires spikes to the processing layer in the next layer according to different spike trains; the synaptic model between the input layer and the processing layer follows the bio-inspired spike timing and relies on the plasticity rule for unsupervised online learning, and conducts online learning based on the arrival times of the pre- and post-synaptic spikes; after receiving the fired spikes, the processing layer accumulates the membrane potential, and when it exceeds the threshold potential, it fires a spike signal to the inhibitory layer in the third layer, and at the same time, a dynamic membrane potential balance mechanism is used to adjust the threshold potential; the inhibitory layer adopts a neuron lateral inhibition mechanism, and after receiving the spike signal of an excitatory neuron, it sends an inhibitory spike signal to the remaining excitatory neurons in the reverse direction to inhibit their membrane potential.

[0119] First, the obtained spike train is used as an input signal and input into the spiking neural network. The neuron model in the input layer of the spiking neural network will fire spikes to the processing layer in the next layer according to different spike trains.

[0120] The neuron model is a Leaky Integrate and Fire (LIF) neuron model, and the mathematical model of this neuron is shown as follows:

[0121]

[0122] S = (sgn(V - E th ) + 1) / 2

[0123] V = S·E rest + (1 - S)·V

[0124] where V is the neuron membrane potential, t is time, E rest is the resting potential, E exc and E inh are the equilibrium membrane potentials of the excitatory neuron and the inhibitory neuron respectively, g e and g i are the conductance values connecting the synapses of the activating / inhibiting neurons, that is, the network weights, E th is the threshold membrane potential of the neuron, S is the signal for spike emission or not, sgn(*) is the sign function, and τ is the membrane potential time constant.

[0125] After the neuron model receives the spike signal encoding the input (i.e., the pre-synaptic spike signal), the membrane potential will be accumulated. Therefore, different encoded spike trains will produce different neuron dynamic behaviors. When the neuron membrane potential exceeds the threshold membrane potential, it emits a post-synaptic spike signal, and the neuron membrane potential then returns to the resting potential; otherwise, the membrane potential continues to accumulate.

[0126] 7. The synaptic model between the input layer and the processing layer performs unsupervised online learning of the spiking neural network according to the bio-inspired spike-time-dependent plasticity rule, and conducts online learning based on the arrival times of the pre- and post-synaptic spikes;

[0127] The synaptic change rule follows the bio-inspired spike-time-dependent plasticity (STDP) rule, which means that the synaptic value changes according to the STDP rule based on the arrival times of the pre- and post-synaptic spikes. The calculation formula is as follows:

[0128]

[0129] a pre / post = a pre / post + A pre / post

[0130] g = g + a post / pre

[0131] where τ pre / post is the time constant for pre- (post-) synaptic change, a pre / post represents the trajectory of pre- (post-) synaptic change, A pre / post is the change rate of pre- (post-) synaptic, and g is the synaptic conductance, i.e., the network weight.

[0132] As shown in the specification appendix Figure 2 When the pre-synaptic pulse signal precedes the post-pulse signal (i.e., when Δt > 0), the synaptic conductance (network weight) increases, and vice versa. Since it is extremely inconvenient and inefficient to record all the pulse emission times, the above-mentioned pulse trajectory method is used to characterize the synaptic dynamics behavior on the time axis. In this figure, "k" represents the kernel size, and "n" represents the number of channels. For example, "k1n32" means the kernel size is 1×1 and the number of channels is 32.

[0133] 8. The processing layer of the spiking neural network contains n excitatory neuron models, which will accumulate the membrane potential after receiving the excitation pulse, and will emit a pulse signal to the inhibitory layer of the third layer after exceeding the threshold potential. The threshold potential is adjusted by the dynamic membrane potential balance mechanism;

[0134] The dynamic membrane potential balance mechanism adjusts the neuron threshold membrane potential according to the following formula:

[0135] E th = E base + θ

[0136]

[0137] θ = θ + S·θ +

[0138] Among them, E base is the reference potential of the threshold membrane potential, θ is the dynamic potential of the threshold membrane potential, and τ th is the time constant of the dynamic potential of the threshold membrane potential, and θ + is the increment of the dynamic potential, and S is the signal indicating whether the above pulse is emitted or not.

[0139] When there is no pulse excitation, the dynamic potential of the threshold membrane potential will decay slowly, that is, the threshold will slowly decrease. When there is pulse excitation, the dynamic potential will increase, that is, the threshold will rise.

[0140] 9. The inhibitory layer of the pulse neural network only contains one inhibitory neuron and adopts the mechanism of lateral inhibition of neurons; that is, after receiving the pulse signal of a certain excitatory neuron, it will send an inhibitory pulse signal to the remaining excitatory neurons in the reverse direction to inhibit their membrane potential;

[0141] 10. Before using the hybrid neural network, train it in the following way:

[0142] Collect various state data of the underwater robot during normal operation and under different fault conditions, preprocess it, normalize it, and combine it into multiple groups of matrix data as training data, and establish a training set and a validation set;

[0143] Use the training set to train a separate deep convolutional neural network to ensure that it can effectively extract the spatio-temporal features of the state data; retain all parameters except the linear layer in the optimal model, and then transfer the parameters to the deep convolutional neural network with the same structure in the hybrid network model; the trained deep convolutional neural network and pulse neural network can be deployed in the FPGA or ASIC in the underwater robot system.

[0144] Input the state data of the validation set into the hybrid network model, and then use the output of the deep convolutional neural network to tune the parameters of the pulse neural network; count the number of firings of the neuron model in the processing layer of the pulse neural network, and mark the neuron with the most firings according to the operating state or fault condition corresponding to the state data in the validation set; after sufficient data volume training and validation, obtain the hybrid neural network model for diagnosis.

[0145] 11. Collect the sensor signals of the underwater robot in the target system and process the collected signals according to the steps described above; after the spatio-temporal features are extracted by the deep convolutional neural network and input into the pulse neural network, the marked type of the neuron with the most firings in the latter's processing layer is finally used to determine the diagnosis result of the operating fault of the underwater robot.

[0146] A specific application example:

[0147] This embodiment is implemented based on the technical solution of the present invention.

[0148] The underwater robot fault diagnosis method based on the online fine-tuning hybrid neural network proposed in the present invention is used to diagnose underwater robot faults, which include four state types (normal, depth sensor failure, body additional mass and propeller blade damage). Among them, the propeller failure adopts relatively minor damage during the training and verification process. In order to verify the adaptability of the online fine-tuning of the present invention to uncertain faults and variable working conditions, the fault diagnosis performance test is carried out in the test process using a propeller state that is more severely damaged than that during the training process.

[0149] This embodiment specifically includes the following steps:

[0150] Step 1: In this example, a real dataset from underwater robot A was used to validate the method. The signals used were divided into a training set and a test set. The training set only contained cases with slightly damaged propeller blades, with approximately 15% of the blades missing, while the test set contained cases with more severe propeller blade damage, with approximately 30% of the blades missing.

[0151] Step 2: Normalize the training data and reconstruct every 64 segments of sampled data into a 32×32 matrix data.

[0152] In the third step, the obtained matrix data frame is pre-trained through a deep convolutional neural network to obtain an optimal pre-training model that can effectively extract spatiotemporal features;

[0153] Step 4: All parameters of the obtained optimal pre-trained model except the linear layer are transferred to the deep convolutional neural network part of the hybrid network model;

[0154] Step 5: Input the remaining data into the hybrid network model to tune the parameters of the spiking neural network part;

[0155] Step 6: Use Poisson pulse coding to convert the real value output of the deep convolutional neural network part of the hybrid network model into a pulse train;

[0156] Step 7: Count the number of neurons firing in the processing layer of the spiking neural network, and mark the neuron with the largest number of firings as normal or faulty. This completes the training.

[0157] In step 8, the test data is processed according to the procedures in steps 2 to 7 in this example, and the diagnostic result is determined based on the stimulated neuron markers. After training, a hybrid neural network model that can be fine-tuned online for diagnosis is obtained.

[0158] In this example, a deep convolutional neural network is first used to extract spatio-temporal features from the collected state data, and then Poisson pulse coding is used to convert real values into pulse sequences for subsequent input into a spiking neural network. During the training process, only slightly damaged thrusters are used as training data to train the hybrid network model. In the deep convolutional neural network part, backpropagation is used for parameter training, and in the spiking neural network, the spike-timing-dependent plasticity learning rule is used to learn and update the network weights. After training, more severe thruster blade damage fault data is used for fault diagnosis verification. The firing of neurons in the processing layer of the spiking neural network in the hybrid model is counted, and the diagnosis result is obtained according to the label of the neuron with the most firings.

[0159] According to the actual measurement and verification of the applicant's research team in a large number of actual application scenarios, the underwater robot fault diagnosis method of the present invention is very effective and is of great help in solving the problem of underwater robot fault diagnosis with fault uncertainty or environmental condition deviation.

Claims

1. An underwater robot fault diagnosis method based on an online fine-tuning hybrid neural network, characterized in that the hybrid neural network includes a multi-core deep convolutional neural network with an attention mechanism, and a spiking neural network that adopts a biologically inspired spike-timing-dependent plasticity learning rule, a neuron lateral inhibition mechanism, and a dynamic membrane potential balance mechanism; when performing fault diagnosis, first normalize the state data collected during the navigation of the underwater robot, and then input it into the deep convolutional neural network for spatio-temporal feature extraction; convert the output data combination into a pulse sequence through Poisson coding, and use it as the input of the spiking neural network; count the number of neuron firings in the spiking neural network, and determine and output the fault diagnosis result according to the matching fault label type; during the whole diagnosis process, keep the parameters of the deep convolutional neural network fixed, and realize the online fine-tuning of the overall hybrid network model based on the unsupervised adaptive update of the spiking neural network model.

2. The method according to claim 1, wherein Including: Before using the hybrid neural network, train it in the following way: (2.1) Collect various state data of the underwater robot during normal operation and under different fault conditions, use them as training data, and establish a training set and a validation set; (2.2) Use the training set to train a separate deep convolutional neural network, retain all parameters except the linear layer in the optimal model, and then transfer the parameters to the deep convolutional neural network with the same structure in the hybrid network model; (2.3) Input the state data of the validation set into the hybrid network model, and then use the output of the deep convolutional neural network to tune the parameters of the spiking neural network; count the number of neuron firings in the processing layer of the spiking neural network, and mark the neuron with the most firings according to the operating state or fault condition corresponding to the state data in the validation set; after sufficient data volume training and validation, obtain a hybrid neural network model for diagnosis.

3. The method according to claim 1, wherein The calculation process in the deep convolutional neural network includes: (3.1) In multiple convolutional modules, the output of the c-th output channel in the l-th convolutional layer at the spatial position (i, j) is calculated by the following formula: Among them, represents the feature map value at the position (i, j) of the c-th output channel in the l-th layer; represents the weight value from the d-th input channel to the c-th output channel at the position (m, n); represents the input feature map of the previous layer, where s is the stride and p is the padding size; is the bias term of the c-th output channel; the dimension of the output feature map is H l ×W l ×C l , the dimension of the convolutional kernel is K×K×C (l-1) ×C l , the dimension of the bias term is C l ; (3.2) In the convolutional and leaky rectified linear unit module, define the activation function as follows: where α represents the slope of the negative input, and its value is 0.01; y represents the input data; (3.3) In the max pooling module, the max pooling operation of the l-th layer is defined as follows: Among them, the maximum value is obtained within the pooling window of P×P. represents the feature map value at the position (i, j) of the c-th output channel after max pooling; represents the output value after pooling; the output dimension is H l ×W l ×C l ; (3.4) In the attention mechanism module, it includes three operations: squeeze, excitation, and recalibration; among them, the squeeze operation uses global average pooling to generate channel statistics z∈R^C, and the calculation formula is as follows: where x c(i,j) represents the feature map value of channel c at the spatial position (i, j); the input feature map X ∈ R H×W×C , where H, W, and C represent height, width, and the number of channels respectively, and F sq is the squeeze function; the excitation operation is defined as: s = F ex(z,W) = σ(W2 · δ(W1 · z)) Among them, W1 and W2 are the weights of two fully connected layers; δ is the rectified linear unit activation function, σ is the Sigmoid activation function; z represents the final channel weight vector; r represents the dimensionality reduction ratio; F ex represents the activation function; the calculation formula of the recalibration operation is as follows: X c ′ = s c ·X c , c = 1, 2, …, C X ′ ∈R H×W×C Among them, s c is the channel weight obtained from the excitation operation, and X c represents the feature map of channel c; X ′ represents the feature map after the recalibration operation; (3.5) In the flattening module, reconstruct the obtained multi-channel two-dimensional matrix data into a single vector data.

4. The method according to claim 1, characterized in that The Poisson coding refers to encoding real values into a pulse sequence according to the Poisson distribution; the number of rows of the pulse sequence is equal to the number of column vectors in the corresponding data combination, and the length of the sequence is determined by the encoding base frequency; Specific encoding follows the following formula: Wherein, P is the possibility of an event occurring, specifically referring to the possibility of transmitting a pulse here; X is the event; x is the actual number of times the event occurs; and λ is the average number of pulses transmitted per unit time.

5. The method according to claim 1, wherein In a spiking neural network, the neuron model in the input layer fires pulses to the processing layer in the next layer according to different pulse sequences; the synaptic model between the input layer and the processing layer follows the biologically inspired spike timing and performs unsupervised online learning depending on the plasticity rule, and conducts online learning according to the arrival times of the pre- and post-synaptic pulses. After receiving the fired pulses, the processing layer accumulates the membrane potential. When it exceeds the threshold potential, it fires a pulse signal to the inhibitory layer in the third layer, and at the same time adopts a dynamic membrane potential balancing mechanism to adjust the threshold potential; the inhibitory layer adopts a neuron lateral inhibition mechanism. After receiving the pulse signal of an excitatory neuron, it sends an inhibitory pulse signal in the reverse direction to the remaining excitatory neurons to inhibit their membrane potentials.

6. The method according to claim 1, wherein In a spiking neural network, the neuron model adopted by the input layer is a leaky integrate-and-fire neuron model, and its mathematical model is as follows: S=(sgn(V - E th ) + 1) / 2 V = S·E rest +(1 - S)·V Among them, V is the neuron membrane potential; t is the time; E rest is the resting potential; E exc and E inh are the equilibrium membrane potentials of excitatory neurons and inhibitory neurons respectively; g e and g i are the conductance values of the synapses connecting the activating / inhibiting neurons, i.e., the network weights; E th is the threshold membrane potential of the neuron; S is the signal indicating whether a pulse is emitted or not; sgn(*) is the sign function; τ is the membrane potential time constant; After the neuron model receives the pre-synaptic pulse signal encoding the input, the membrane potential will be accumulated. Different encoded pulse sequences will produce different neuron dynamic behaviors; when the neuron membrane potential exceeds the threshold membrane potential, a pulse signal is emitted, and the neuron membrane potential returns to the resting potential, otherwise the membrane potential continues to be accumulated.

7. The method according to claim 1, characterized in that, Following the biologically inspired spike timing-dependent plasticity rule for unsupervised online learning of the spiking neural network, the synaptic value of the synaptic model is updated online according to the arrival times of the pre- and post-synaptic pulses; specifically, it is calculated according to the following formula: a pre / post = a pre / post + A pre / post g = g + a post / pre where τ pre / post is the time constant of the pre- / post-synaptic change; a pre / post is the trajectory representing the pre- / post-synaptic change; A pre / post is the change rate of the pre- / post-synaptic; g is the synaptic conductance, i.e., the network weight.

8. The method according to claim 1, wherein The said dynamic membrane potential balancing mechanism means calculating the neuron threshold membrane potential according to the following formula: E th = E base + θ θ = θ + S·θ + Among them, E base is the reference potential of the threshold membrane potential; θ is the dynamic potential of the threshold membrane potential; τ th is the time constant of the dynamic potential of the threshold membrane potential; θ + is the increment of the dynamic potential; S is the signal indicating whether the above pulse is emitted or not; When there is no pulse excitation, the dynamic potential of the threshold membrane potential will gradually decay, that is, the threshold decreases; when there is pulse excitation, the dynamic potential will gradually increase, that is, the threshold increases; in this way, the dynamic adjustment of the neuron threshold membrane potential is realized.

9. An underwater robot fault diagnosis system based on an online fine-tuning hybrid neural network, characterized in that, Including: Several sensors installed on the underwater robot, used to collect the state data generated during the operation of the underwater robot; A signal processing and transmission device, used to collect the state data collected by the sensors, and after preprocessing, transmit it to the processor or controller through a serial port or a bus; A memory, used to store the software function modules and instructions for implementing the method described in any one of claims 1 to 8; The said hybrid neural network is deployed in the software function module; A processor or controller, configured to call the software function module from the said memory, so as to be able to utilize the state data to implement the said method when executing the instructions, and finally obtain the diagnostic result of the operation failure of the underwater robot.

10. The system according to claim 9, wherein, The said software function module includes a data preprocessing module, a deep convolutional neural network module and a spiking neural network module; among them, the deep convolutional neural network includes multiple convolutional modules, as well as a convolutional and leaky rectified linear unit module, a max pooling module, an attention mechanism module and a flattening module; the spiking neural network module includes an input layer, a processing layer and an inhibitory layer, and each layer includes a neuron model. The sensors are an attitude sensor, a voltage sensor, and a depth sensor installed on the underwater robot; the status data collected by the sensors at least includes: thruster control signals, underwater robot power supply voltage signals, water depth values, robot attitude angles, robot body coordinate accelerations, and robot three-axis rotational angular velocities.

Images