Collaborative identification method based on optical saliency detection and acoustic online evolution

Abstract

The invention discloses a collaborative recognition method based on optical saliency detection and acoustic online evolution, and relates to the technical field of computer vision, and the method comprises the steps: firstly carrying out the light target recognition through employing an optical saliency detection algorithm based on a double-branch CNN, and then carrying out the acoustic online evolution recognition through employing an LSTM network, and finally, composite features are obtained through an acousto-optic information fusion recognition algorithm. According to the collaborative recognition method based on optical saliency detection and acoustic online evolution provided by the invention, the target recognition precision is further improved, and multimode online collaborative evolution detection is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a collaborative recognition method based on optical saliency detection and acoustic online evolution. Background Art

[0002] In the field of target recognition, there are currently three main methods for weak target recognition: model methods (such as wavelet transform, neural network), statistical methods (such as support vector machine SVM, random forest), and deep learning methods (such as convolutional neural network CNN, recurrent neural network RNN). However, each of these algorithms has certain problems. From the perspective of anti-interference, they are easily affected by various interference factors in complex environments, resulting in unstable recognition results. In terms of the rationality of feature selection, due to the characteristics of weak targets themselves, these algorithms may not be able to accurately select the most representative features, thereby affecting recognition accuracy. Moreover, in terms of environmental adaptability, the performance of these algorithms varies under different environmental conditions, and there are certain shortcomings or thresholds. At the same time, when only a single scale is used for target recognition, the information that can be obtained is limited, which not only limits the integrity of the information, but also makes the recognition accuracy poor.

[0003] Multi-sensor information fusion recognition methods have become a research direction for improving target recognition rates. Currently, there are mainly classic methods such as Bayesian reasoning, evidence theory, fuzzy theory, and deep learning. For example, in the 2002 Intelligent Vehicle Symposium journal paper "Multi-sensor data fusion using Bayesian programming: an automotive application", it is mentioned that the information fusion recognition of radar and laser is achieved through Bayesian reasoning. However, this method mainly relies on expert experience to determine the weight ratio, which makes it difficult to quantify the recognition results, because expert experience has a certain degree of subjectivity and cannot form a unified quantitative standard. For example, in the 2019 Neural Computing and Applications journal paper "Research on data fusion of multi-sensors based on fuzzy preference relations", fuzzy theory is used to calculate the weights of various objective functions to improve the fusion recognition accuracy. However, this method has large combination conflicts and poor robustness. In actual application scenarios, due to the complexity and variability of the environment, this poor robustness is not conducive to stable practical applications. There is also a journal paper in IEEE / ASME Transactions on Mechatronics in 2024 titled "Learning generalizable vision - tactile robotic grasping strategy for deformable objects via transformer", in which visual and tactile information fusion recognition is achieved by constructing a Transformer prediction model. However, this method is limited to fusion by connection and aggregation, and does not achieve feature-level fusion. This means that some key feature information may be lost in the fusion process, and the potential of multi-source information cannot be fully tapped. Therefore, a collaborative recognition method based on optical saliency detection and acoustic online evolution is urgently needed to address the shortcomings of the above-mentioned existing technologies. Summary of the invention

[0004] The purpose of the present invention is to provide a collaborative recognition method based on optical saliency detection and acoustic online evolution to solve the problems of low accuracy and poor stability of target recognition in the prior art.

[0005] To achieve the above object, the present invention provides a collaborative recognition method based on optical saliency detection and acoustic online evolution, comprising the following steps: Step 1: The optical saliency detection algorithm based on the dual-branch CNN is used to identify light targets; a main branch is set to provide the texture features of the optical target, and an auxiliary branch is set to provide the edge features of the optical target. Finally, the output features of the main branch and the auxiliary branch are fused through a fully connected layer; Step 2: Use LSTM network for acoustic online evolution recognition; adopt a three-level series structure, first use the convolution layer to extract feature signals from the acoustic image at discrete time points, then use the LSTM network to predict and analyze the mapping relationship between the time series and the detected target, and finally use the fully connected layer to perform nonlinear mapping on the feature data to achieve target type classification; Step 3: Obtain composite features through the acoustic-optical information fusion recognition algorithm; substitute the recognition results of the optical saliency detection algorithm based on the dual-branch CNN and the results of the acoustic online evolution recognition based on the LSTM network into the fusion recognition model, conduct comprehensive analysis and fusion processing on the extracted acoustic and optical feature information, classify the fused features and give the classification results.

[0006] Preferably, the main branch includes n convolutional layers for extracting and capturing texture features of the input image, as expressed as follows: (1) In the formula, is the texture feature map output by the i-th convolutional layer, Sigmoid is the excitation function of the main branch, is the weight of the i-th convolutional layer, is the input image of the i-1th convolutional layer, is the threshold of the i-th convolutional layer.

[0007] Preferably, the auxiliary branch also includes n convolutional layers for extracting and capturing edge features of the input image, as expressed as follows: (2) In the formula, is the edge feature map output by the i-th convolutional layer, is the activation function of the auxiliary branch.

[0008] Preferably, the process of step 1 is as follows: S11, after preprocessing the optical image, a 960×720 grayscale image is formed, which is then brought into the input layer for processing to form a 1×960×720 two-dimensional neuron, which is input into the main branch and the auxiliary branch at the same time to obtain the corresponding output; S12. The output features of the main branch and the auxiliary branch are fused through the fully connected layer. The fusion method is add. The expression is as follows: (3) In the formula, is the feature fusion result of the i-th convolutional layer, is the connection weight of the i-th convolutional layer.

[0009] Preferably, the process of step 2 is as follows: S21, extract feature signals from acoustic images at discrete time points through convolution layers; the convolution layer (Conv layer) includes 4 convolution layers, each of which extracts features from acoustic signals with time series through one-dimensional convolution; after multi-layer convolution depth extraction, the one-dimensional feature map output by the i-th convolution layer is obtained, and the calculation expression is as follows: (4) In the formula, is the one-dimensional feature map output by the i-th convolutional layer, is the activation function of the auxiliary branch, n represents the number of feature maps of the i-1th convolutional layer, is the weight of the qth channel of the i-th convolutional layer, is the input image of the qth channel of the i-1th convolutional layer, is the threshold of the i-th convolutional layer; S22, one-dimensional feature map of the output of the i-th convolutional layer Perform maximum pooling, set the pooling kernel length and step size to 2, and obtain the feature map of the qth channel of the i-th convolutional layer , the calculation expression is as follows: (5) In the formula, Represents the output of the qth channel of the i-th layer after the activation function; S23, predict and analyze the mapping relationship between the time series and the detection target through the LSTM network; the LSTM network includes a forget gate, an input gate, an output gate and a warehouse Cell; S23, the feature data is nonlinearly mapped through a fully connected layer; the ReLu activation function is used for nonlinear mapping to obtain a one-dimensional feature vector, and the expression is as follows: (6) In the formula, is the feature extraction result of the i-th layer fully connected network, is the weight of the i-th layer of the fully connected network, is the threshold of the i-th layer fully connected network, Indicates the connectivity status of the i-1th layer nodes after dropout processing.

[0010] Preferably, the specific contents of the forget gate, input gate, output gate and a warehouse Cell are as follows: The keyword information that is discarded is determined by the forget gate, specifically, the keyword information input at time t and the previous hidden state , after the Sigmoid function outputs a vector between 0 and 1 The closer the element in the vector is to 0, the greater the probability of discarding it, and the closer it is to 1, the more it is retained; the calculation expression is as follows: (7) In the formula, The weight matrix representing the candidate cell state, A bias term representing the candidate cell state; Update the cell state through the input gate, specifically: input at time t and the previous hidden state On the one hand, through The function outputs the information that needs to be updated in the Cell On the one hand, through Function creates a new candidate information vector , and finally multiply the two together, through Decide which The information in the cell is important and needs to be preserved to achieve self-update of the Cell state; the calculation expression is as follows: (8) (9) (10) In the formula, represents the cell state at time t, represents the cell state at time t-1; The output gate determines the value of the hidden state at the next moment: and the previous hidden state ,go through The function outputs the information that needs to be output in the Cell , passed to the tanh function, and the output value of the tanh function is compared with Multiply to determine the information that the hidden state should carry, and use this as the final output information ; The calculation expression is as follows: (11) (12) In the formula, represents the weight matrix of the output gate, represents the bias term of the output gate; Update hidden state through warehouse Cell and cell status The function of the warehouse Cell runs through the entire LSTM unit structure. The first step is to discard unimportant information by multiplying it with the output value of the forget gate; the second step is to retain important information by multiplying it with the output value of the input gate to achieve cell updating.

[0011] Preferably, the process of obtaining the composite feature by the acoustic-optical information fusion recognition algorithm in step 3 is as follows: S31, extracting the optical branch feature vector Input the branch points, then connect a batch normalization layer to adjust the hidden layer neurons to the standard normal distribution. The expression is as follows: (13) In the formula, is the original activation value corresponding to each neuron, is obtained for several instances in a small batch The mean of is the original activation value The variance of express Normalized data of Perform translation and scaling operations on each transformed activated neuron to obtain the optical branch feature extraction vector Normalized calculation results , the expression is as follows: (14) (15) (16) In the formula, is the shift parameter, is the scale parameter; the acoustic branch feature extraction vector is obtained by the same method Normalized calculation results ; S32, the one-dimensional feature vector of the optical branch after normalization , one-dimensional characteristic vector of acoustic branch Perform feature cascading and input to the fully connected layer for deep feature extraction. Perform nonlinear mapping through the ReLu activation function, and set some output nodes to 0 through dropout. The expression is as follows: (17) S33, through the classification layer, the depth extraction features are subjected to probability distribution statistics by the Softmax function to achieve target classification prediction. The calculation expression is as follows: (18) In the formula, the Softmax function is the target classification recognition result; is the output value of node i; m is the number of target categories / number of output nodes.

[0012] Therefore, the present invention adopts the above-mentioned collaborative recognition method based on optical saliency detection and acoustic online evolution, which has the following beneficial effects: (1) Through the dual-branch convolutional neural network structure, fusion processing is performed to improve the algorithm's anti-interference ability while optimizing the calculation rate and recognition accuracy, ultimately improving the adaptability and application effect of optical recognition; (2) By setting up a three-level series structure of convolutional layer, LSTM network layer, and fully connected layer, the characteristic evolution process of the acoustic target in the time series is obtained, the online prediction and recognition of the target type is realized, and the stability of acoustic target recognition is guaranteed during the close detection process.

[0013] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is an overall flow chart of a collaborative recognition method based on optical saliency detection and acoustic online evolution according to the present invention; Figure 2 : This is a diagram of an optical saliency detection model based on a dual-branch CNN according to an embodiment of the present invention; Figure 3 This is a diagram of an acoustic online evolution recognition model based on an LSTM network according to an embodiment of the present invention; Figure 4 This is an internal structure diagram based on an LSTM unit in an embodiment of the present invention; Figure 5 This is a model diagram of the sound and light information fusion recognition algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] See also Figure 1-Figure 5 , a collaborative recognition method based on optical saliency detection and acoustic online evolution, comprising the following steps: Step 1: The optical saliency detection algorithm based on the dual-branch CNN is used to identify light targets. A main branch is set to provide the texture features of the optical target, and an auxiliary branch is set to provide the edge features of the optical target. Finally, the output features of the main branch and the auxiliary branch are fused through the fully connected layer. This can effectively overcome the outliers or noise interference brought by a certain branch, effectively capture the characteristic information of the target while speeding up the calculation rate, and finally improve the algorithm recognition performance. The specific structure of the algorithm model is as follows: Figure 2 As shown in the figure, underwater weak feature targets are affected by background interference, obstacles, and their own small size, resulting in low optical imaging contrast, blurred contour details, and easy color distortion, which greatly increases the difficulty of optical target recognition. To address this target recognition problem, an optical saliency detection algorithm based on a dual-branch CNN is proposed.

[0017] Among them, the main branch includes n convolutional layers, which are used to extract and capture the texture features of the input image. The expression is as follows: (1) In the formula, is the texture feature map output by the i-th convolutional layer, Sigmoid is the excitation function of the main branch, is the weight of the i-th convolutional layer, is the input image of the i-1th convolutional layer, is the threshold of the i-th convolutional layer.

[0018] The auxiliary branch also includes n convolutional layers, which are used to extract and capture the edge features of the input image. The expression is as follows: (2) In the formula, is the edge feature map output by the i-th convolutional layer, is the activation function of the auxiliary branch.

[0019] The process for step 1 is as follows: S11, after preprocessing the optical image, a 960×720 grayscale image is formed, which is then brought into the input layer for processing to form a 1×960×720 two-dimensional neuron, which is input into the main branch and the auxiliary branch at the same time to obtain the corresponding output; S12. The output features of the main branch and the auxiliary branch are fused through the fully connected layer (FC layer). The fusion method is add. The expression is as follows: (3) In the formula, is the feature fusion result of the i-th convolutional layer, is the connection weight of the i-th convolutional layer.

[0020] Step 2: Use LSTM network for acoustic online evolution recognition; adopt a three-level series structure, first extract feature signals from the acoustic image at discrete time points through the convolution layer, then predict and analyze the mapping relationship between the time series and the detection target through the LSTM network, and finally perform nonlinear mapping of feature data through the fully connected layer to achieve target type classification; acoustic detection of underwater weak feature targets is generally not affected by weather and lighting, and the detection range is far, but the disadvantage is that it is easily affected by background noise, multipath interference and other factors, and the imaging effect quality is poor. For this target recognition problem, an acoustic online evolution recognition algorithm based on LSTM network is proposed. The specific structure of the algorithm model is as follows: Figure 3 As shown; and the internal structure of the LSTM unit is as follows Figure 4 As shown, the specific process is as follows: S21, extract feature signals from acoustic images at discrete time points through the convolution layer (Conv layer); the convolution layer (Conv layer) includes 4 convolution layers, each of which extracts features from acoustic signals with time series through one-dimensional convolution; after multi-layer convolution depth extraction, the one-dimensional feature map output by the i-th convolution layer is obtained, and the calculation expression is as follows: (4) In the formula, is the one-dimensional feature map output by the i-th convolutional layer, is the activation function of the auxiliary branch, n represents the number of feature maps of the i-1th convolutional layer, is the weight of the qth channel of the i-th convolutional layer, is the input image of the qth channel of the i-1th convolutional layer, is the threshold of the i-th convolutional layer; S22, one-dimensional feature map of the output of the i-th convolutional layer Perform maximum pooling, set the pooling kernel length and step size to 2, and obtain the feature map of the qth channel of the i-th convolutional layer , the calculation expression is as follows: (5) In the formula, Represents the output of the qth channel of the i-th layer after the activation function; S23. The mapping relationship between the time series and the detection target is predicted and analyzed through the LSTM network; wherein the LSTM network includes a forget gate, an input gate, an output gate and a warehouse Cell; wherein the specific contents of the forget gate, the input gate, the output gate and a warehouse Cell are as follows: The keyword information that is discarded is determined by the forget gate, specifically, the keyword information input at time t and the previous hidden state , after the Sigmoid function outputs a vector between 0 and 1 The closer the element in the vector is to 0, the greater the probability of discarding it, and the closer it is to 1, the more it is retained; the calculation expression is as follows: (7) In the formula, The weight matrix representing the candidate cell state, A bias term representing the candidate cell state; Update the cell state through the input gate, specifically: input at time t and the previous hidden state On the one hand, through The function outputs the information that needs to be updated in the Cell On the one hand, through Function creates a new candidate information vector , and finally multiply the two together, through Decide which The information in the cell is important and needs to be preserved to achieve self-update of the Cell state; the calculation expression is as follows: (8) (9) (10) In the formula, represents the cell state at time t, represents the cell state at time t-1; The output gate determines the value of the hidden state at the next moment: and the previous hidden state ,go through The function outputs the information that needs to be output in the Cell , passed to the tanh function, and the output value of the tanh function is compared with Multiply to determine the information that the hidden state should carry, and use this as the final output information ; The calculation expression is as follows: (11) (12) In the formula, represents the weight matrix of the output gate, represents the bias term of the output gate; Update hidden state through warehouse Cell and cell status The function of the warehouse Cell runs through the entire LSTM unit structure. The first step is to discard unimportant information by multiplying it with the output value of the forget gate; the second step is to retain important information by multiplying it with the output value of the input gate to achieve cell update; S23, the feature data is nonlinearly mapped through a fully connected layer; the ReLu activation function is used for nonlinear mapping to obtain a one-dimensional feature vector, and the expression is as follows: (6) In the formula, is the feature extraction result of the i-th layer fully connected network, is the weight of the i-th layer of the fully connected network, is the threshold of the i-th layer fully connected network, Indicates the connectivity status of the i-1th layer nodes after dropout processing.

[0021] Step 3: Obtain composite features through the acoustic-optical information fusion recognition algorithm; substitute the recognition results of the optical saliency detection algorithm based on the dual-branch CNN and the results of the acoustic online evolution recognition based on the LSTM network into the fusion recognition model, conduct a comprehensive analysis and fusion processing on the extracted acoustic and optical feature information, classify the fused features and give the classification results; the advantage of this algorithm is that under the premise of feature-level fusion, the existing extracted acoustic and optical features are comprehensively processed to obtain composite features, while retaining more of the respective target information, more potential features are excavated to further improve the target recognition accuracy. The specific structure of the algorithm is as follows: Figure 5 The specific process is shown as follows: S31, extracting the optical branch feature vector Input the branch points, and then connect a batch normalization layer (BNLayer) to adjust the hidden layer neurons to the standard normal distribution to avoid gradient disappearance. The expression is as follows: (13) In the formula, is the original activation value corresponding to each neuron, is obtained for several instances in a small batch The mean of is the original activation value The variance of express Normalized data of Perform translation and scaling operations on each transformed activated neuron to obtain the optical branch feature extraction vector Normalized calculation results , the expression is as follows: (14) (15) (16) In the formula, is the shift parameter, is the scale parameter; the acoustic branch feature extraction vector is obtained by the same method Normalized calculation results ; S32, the one-dimensional feature vector of the optical branch after normalization , one-dimensional characteristic vector of acoustic branch Perform feature cascading and input to the fully connected layer for deep feature extraction. Perform nonlinear mapping through the ReLu activation function, and set some output nodes to 0 through dropout. The expression is as follows: (17) S33, through the classification layer, the depth extraction features are subjected to probability distribution statistics by the Softmax function to achieve target classification prediction. The calculation expression is as follows: (18) In the formula, the Softmax function is the target classification recognition result; is the output value of node i; m is the number of target categories / number of output nodes.

[0022] Therefore, the present invention adopts the above-mentioned collaborative recognition method based on optical saliency detection and acoustic online evolution. First, the optical saliency detection algorithm based on the dual-branch CNN is used to recognize light targets; a main branch is set to provide the texture features of the optical target, and an auxiliary branch is set to provide the edge features of the optical target, and the output features of the main branch and the auxiliary branch are fused through a fully connected layer; secondly, an LSTM network is used to recognize acoustic online evolution; a three-level series structure is adopted, and feature signals are extracted from the acoustic image at discrete time points through a convolution layer, and then the mapping relationship between the time series and the detection target is predicted and analyzed through the LSTM network, and the feature data is nonlinearly mapped through a fully connected layer to realize target type classification; finally, a composite feature is obtained through an acoustic-optical information fusion recognition algorithm; the recognition results of the optical saliency detection algorithm based on the dual-branch CNN and the results of the acoustic online evolution recognition based on the LSTM network are respectively substituted into the fusion recognition model, the extracted acoustic and optical feature information is comprehensively analyzed and fused, the fused features are classified and the classification results are given, thereby further improving the target recognition accuracy and realizing multi-mode online collaborative evolution detection.

[0023] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A collaborative recognition method based on optical saliency detection and acoustic online evolution, characterized in that: The following steps are involved: Step 1: The optical saliency detection algorithm based on the dual-branch CNN is used to identify light targets; a main branch is set to provide the texture features of the optical target, and an auxiliary branch is set to provide the edge features of the optical target. Finally, the output features of the main branch and the auxiliary branch are fused through a fully connected layer; Step 2: Use LSTM network for acoustic online evolution recognition; adopt a three-level series structure, first use the convolution layer to extract feature signals from the acoustic image at discrete time points, then use the LSTM network to predict and analyze the mapping relationship between the time series and the detected target, and finally use the fully connected layer to perform nonlinear mapping on the feature data to achieve target type classification; Step 3, obtaining composite features through the acoustic and optical information fusion recognition algorithm; The recognition results of the optical saliency detection algorithm based on a dual-branch CNN and the results of the acoustic online evolution recognition based on an LSTM network are respectively substituted into the fusion recognition model, the extracted acoustic and optical feature information is comprehensively analyzed and fused, the fused features are classified and the classification results are given.

2. The collaborative recognition method based on optical saliency detection and acoustic online evolution according to claim 1, characterized in that: The main branch consists of n convolutional layers, which are used to extract and capture the texture features of the input image. The expression is as follows: (1) In the formula, is the texture feature map output by the i-th convolutional layer, Sigmoid is the excitation function of the main branch, is the weight of the i-th convolutional layer, is the input image of the i-1th convolutional layer, is the threshold of the i-th convolutional layer.

3. The collaborative recognition method based on optical saliency detection and acoustic online evolution according to claim 2, characterized in that: The auxiliary branch also includes n convolutional layers, which are used to extract and capture the edge features of the input image. The expression is as follows: (2) In the formula, is the edge feature map output by the i-th convolutional layer, is the activation function of the auxiliary branch.

4. The collaborative recognition method based on optical saliency detection and acoustic online evolution according to claim 3, characterized in that: The process for step 1 is as follows: S11, after preprocessing the optical image, a 960×720 grayscale image is formed, which is then brought into the input layer for processing to form a 1×960×720 two-dimensional neuron, which is input into the main branch and the auxiliary branch at the same time to obtain the corresponding output; S12. The output features of the main branch and the auxiliary branch are fused through the fully connected layer. The fusion method is add. The expression is as follows: (3) In the formula, is the feature fusion result of the i-th convolutional layer, is the connection weight of the i-th convolutional layer.

5. The collaborative recognition method based on optical saliency detection and acoustic online evolution according to claim 4, characterized in that: The process for step 2 is as follows: S21, extract feature signals from the acoustic image at discrete time points through the convolution layer; the convolution layer includes 4 convolution layers, each layer extracts features from the acoustic signal with time series through one-dimensional convolution; after multi-layer convolution depth extraction, the one-dimensional feature map output by the i-th convolution layer is obtained, and the calculation expression is as follows: (4) In the formula, is the one-dimensional feature map output by the i-th convolutional layer, is the activation function of the auxiliary branch, n represents the number of feature maps of the i-1th convolutional layer, is the weight of the qth channel of the i-th convolutional layer, is the input image of the qth channel of the i-1th convolutional layer, is the threshold of the i-th convolutional layer; S22, one-dimensional feature map of the output of the i-th convolutional layer Perform maximum pooling, set the pooling kernel length and step size to 2, and obtain the feature map of the qth channel of the i-th convolutional layer , the calculation expression is as follows: (5) In the formula, Represents the output of the qth channel of the i-th layer after the activation function; S23, predict and analyze the mapping relationship between the time series and the detection target through the LSTM network; the LSTM network includes a forget gate, an input gate, an output gate and a warehouse Cell; S23, the feature data is nonlinearly mapped through a fully connected layer; the ReLu activation function is used for nonlinear mapping to obtain a one-dimensional feature vector, and the expression is as follows: (6) In the formula, is the feature extraction result of the i-th layer fully connected network, is the weight of the i-th layer of the fully connected network, is the threshold of the i-th layer fully connected network, Indicates the connectivity status of the i-1th layer nodes after dropout processing.

6. The collaborative recognition method based on optical saliency detection and acoustic online evolution according to claim 5, characterized in that: The specific contents of the forget gate, input gate, output gate and a warehouse Cell are as follows: The keyword information that is discarded is determined by the forget gate, specifically, the keyword information input at time t and the previous hidden state , after the Sigmoid function outputs a vector between 0 and 1 The closer the element in the vector is to 0, the greater the probability of discarding it, and the closer it is to 1, the more it is retained; the calculation expression is as follows: (7) In the formula, The weight matrix representing the candidate cell state, A bias term representing the candidate cell state; Update the cell state through the input gate, specifically: input at time t and the previous hidden state On the one hand, through The function outputs the information that needs to be updated in the Cell On the one hand, through Function creates a new candidate information vector , and finally multiply the two together, through Decide which The information in the cell is important and needs to be preserved to achieve self-update of the Cell state; the calculation expression is as follows: (8) (9) (10) In the formula, represents the cell state at time t, represents the cell state at time t-1; The output gate determines the value of the hidden state at the next moment: and the previous hidden state ,go through The function outputs the information that needs to be output in the Cell , passed to the tanh function, and the output value of the tanh function is compared with Multiply to determine the information that the hidden state should carry, and use this as the final output information ; The calculation expression is as follows: (11) (12) In the formula, represents the weight matrix of the output gate, represents the bias term of the output gate; Update hidden state through warehouse Cell and cell status .

7. The collaborative recognition method based on optical saliency detection and acoustic online evolution according to claim 6, characterized in that: The process of obtaining the composite features through the acoustic-optical information fusion recognition algorithm in step 3 is as follows: S31, extracting the optical branch feature vector Input the branch points, then connect a batch normalization layer to adjust the hidden layer neurons to the standard normal distribution. The expression is as follows: (13) In the formula, is the original activation value corresponding to each neuron, is obtained for several instances in a small batch The mean of is the original activation value The variance of express Normalized data of Perform translation and scaling operations on each transformed activated neuron to obtain the optical branch feature extraction vector Normalized calculation results , the expression is as follows: (14) (15) (16) In the formula, is the translation parameter, is the scaling parameter; the acoustic branch feature extraction vector is obtained by the same method Normalized calculation results ; S32, the one-dimensional feature vector of the optical branch after normalization , one-dimensional characteristic vector of acoustic branch Perform feature cascading and input to the fully connected layer for deep feature extraction. Perform nonlinear mapping through the ReLu activation function and set the output node to 0 through dropout. The expression is as follows: (17) S33, through the classification layer, the depth extraction features are subjected to the Softmax function for probability distribution statistics, and the calculation expression is as follows: (18) In the formula, the Softmax function is the target classification recognition result; is the output value of node i; m is the number of target categories / number of output nodes.

Images