Sonar image target recognition method based on reinforcement learning

Abstract

The present invention discloses a sonar image target recognition method based on reinforcement learning, comprising: (1) modeling a layer selection process of a convolutional neural network as a Markov decision process; (2) defining a state space and an action space of a reinforcement learning agent; (3) using a reinforcement learning Q-Learning algorithm to allow the agent to automatically construct a series of convolutional neural networks; (4) selecting a network model with better performance from the automatically generated convolutional neural networks, adjusting parameters and migrating the network model to a target detection network, as shown in the abstract figure; (5) using a sonar image data set to perform supervised training on the entire target detection network to complete the target recognition work of the sonar image, and automating the construction process of the convolutional neural network based on the reinforcement learning Q-Learning algorithm, thereby eliminating a large amount of manual parameter adjustment input, obtaining a convolutional neural network with good performance, and improving the work efficiency of sonar image recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital image processing, and in particular relates to a sonar image target recognition method based on reinforcement learning. Background Art

[0002] Sonar images are obtained by converting acoustic signals from the sound waves actively emitted by the monitoring ship, obtaining echo signals after the sound waves are reflected. Sonar data is often obtained together with water depth detection and bottom detection data, so that we can observe the shallow structure of the seabed. Through sonar images, we can not only find obstacles that affect industrial transportation such as oil and natural gas, intuitively detect the status of submarine cables and pipelines, but also conduct targeted environmental research based on the unique topography of the seabed in the sonar image, and even detect the distribution of mines. Therefore, sonar images are widely used in marine science, marine engineering, underwater target detection and identification and other fields.

[0003] In traditional sonar image research, the main method is to first extract features from sonar images manually, and then classify them into different categories based on the extracted features. If the target is relatively simple and the features are relatively obvious, the traditional classification method will be very good, and the algorithm complexity is relatively low. However, due to its imaging mechanism and complex ocean environment during the acquisition process of sonar images, the image quality after imaging is usually poor, and traditional classification algorithms cannot complete the task well.

[0004] In recent years, deep learning technology, as a multi-layer neural network learning algorithm, has made breakthrough progress in many fields of machine learning. It uses a hierarchical structure similar to the human brain to achieve the extraction and mapping from low-level signals to high-level semantics, and then expresses the data in a hierarchical manner, with powerful visual information processing capabilities. Deep learning technology learns features through a multi-layer nonlinear network structure. Each layer of the network extracts and combines features. The high layer of the network combines the features extracted from the lower layer to obtain a more abstract representation of the data. It automates the manual design or feature extraction process of traditional machine learning, and uses complex functions to approximate and fit the original distribution of the data, so as to extract the essential characteristics of the data set.

[0005] With the continuous development of deep learning, deep learning-based methods are now widely used for the classification of sonar images. However, it is not easy to build a convolutional neural network with good performance. A typical structure consists of several convolutional layers, pooling layers, and fully connected layers. When building a convolutional neural network, the network designer must make many design decisions: the number of layers of each type, the order between layers, and the hyperparameters of each type of layer, such as the size of the receptive field, the step size, and the number of receptive fields in each convolutional layer. Although there are some automatic or computer-assisted neural network design works, the construction of convolutional neural networks mostly relies on the researchers' experience to manually adjust the parameters, which is not very efficient. Summary of the invention

[0006] The present invention aims at the problems existing in the practical application of the prior art and provides a sonar image target recognition method based on reinforcement learning, comprising: (1) modeling the layer selection process of the convolutional neural network as a Markov decision process; (2) defining the state space and action space of the reinforcement learning agent; (3) using the reinforcement learning Q-Learning algorithm to allow the agent to automatically construct a series of convolutional neural networks; (4) selecting a network model with better performance from the automatically generated convolutional neural networks, adjusting the parameters and migrating them to the target detection network; (5) using the sonar image data set to perform supervised training on the entire target detection network to complete the target recognition work of the sonar image, and automating the construction process of the convolutional neural network based on the reinforcement learning Q-Learning algorithm, thereby saving a lot of manual parameter adjustment input, obtaining a convolutional neural network with good performance, and improving the work efficiency of sonar image recognition.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is: a sonar image target recognition method based on reinforcement learning, comprising the following steps:

[0008] S1, the layer selection process of the convolutional neural network is modeled as a Markov decision process: the convolutional neural network includes two parts: feature extraction and feature classification. Feature extraction is at the front end of the network, consisting of a convolutional layer, a pooling layer, and a residual layer; feature classification is at the end of the network; the reinforcement learning agent continuously interacts with the environment, selects appropriate layer types and hyperparameters through feedback from the environment to form the feature extraction part of the convolutional neural network, and then connects the feature classification part to form a complete classification convolutional neural network;

[0009] S2, define the state space and action space of the reinforcement learning agent: the state space of the agent in the reinforcement learning process is composed of various layers of the convolutional neural network, and the action space of the agent is the layers that can be connected in the current state. Each state of the reinforcement learning agent is defined as a tuple of the relevant layer parameters. In the state tuple defined by the layer, the range of hyperparameters is restricted;

[0010] S3, using the reinforcement learning Q-Learning algorithm to let the agent automatically build a series of convolutional neural networks. This step further includes:

[0011] In S31, the agent selects the layer sequentially through reinforcement learning ε-greedy strategy, selects the action with the highest Q value in the current state with probability ε, and randomly selects other actions with probability 1-ε. The formula is as follows:

[0012]

[0013] Until the termination state is reached, it can be considered that a CNN network model is selected;

[0014] S32, training the CNN network model selected in step S31 on the sonar image dataset, and using the accuracy of the model on the validation set as a reward for the reinforcement learning Q-Learning algorithm;

[0015] S33, using the reward on the validation set to update the Q value formula, and then continuing step S31 according to the ε-greedy strategy of step S31 until the number of constructed models reaches a limit value;

[0016] S4, select the network with better performance from the generated convolutional neural network, adjust the parameters and migrate it to the target detection network;

[0017] S5, using the sonar image dataset to perform supervised training on the target monitoring network in step S4 to complete the target recognition task of the sonar image.

[0018] As an improvement of the present invention, in step S1, a spatial pyramid pooling layer is added at the end of the feature extraction part of the convolutional neural network.

[0019] As an improvement of the present invention, in step S2, in the state tuple defined by the layer, the action space of the agent is restricted, the agent is allowed to terminate the path at any node, and the state with a layer depth of i is allowed to be converted to a state with a layer depth of i+1.

[0020] As another improvement of the present invention, the adjustment parameters in step S4 at least include adjustment of the learning rate, optimizer, initialization method, batch size, weight decay and number of training times.

[0021] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0022] (1) The present invention uses reinforcement learning technology to automatically construct a convolutional neural network, which can complete the automatic construction of a convolutional neural network on a specified data set, eliminating a large amount of manual parameter adjustment in the network construction process, and obtaining several convolutional neural networks with good performance.

[0023] (2) The present invention combines reinforcement learning technology with deep learning technology, which improves the accuracy compared with traditional sonar image recognition methods. The automatically constructed network can obtain performance similar to that of a classic convolutional neural network with a small number of network layers.

[0024] (3) Through the study of sonar image target detection, the present invention can effectively identify underwater targets and correctly locate them, which is beneficial to the smooth progress of operations such as seabed salvage and mine distribution detection in a production environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of a Markov decision process for selecting a convolutional neural network structure in step S1 of the present invention;

[0026] Figure 2 Schematic diagram of the construction process of the convolutional neural network in step S3 of the present invention;

[0027] Figure 3 It is a schematic diagram of the M_net network architecture in the experimental test example of the present invention;

[0028] Figure 4 A schematic diagram of the process of using reinforcement learning to construct a network for target detection in the present invention;

[0029] Figure 5(a) is a diagram showing the target recognition result of a sonar image of a corpse;

[0030] Figure 5(b) is a diagram showing the target recognition results of a sonar image of a shipwreck;

[0031] Figure 5(c) shows the target recognition result of aircraft sonar image. DETAILED DESCRIPTION

[0032] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0033] Example 1

[0034] The sonar image target recognition method based on reinforcement learning includes the following steps:

[0035] S1, the layer selection process of the convolutional neural network is modeled as a Markov decision process, such as Figure 1 As shown:

[0036] A typical convolutional neural network structure consists of several convolutional layers, pooling layers, and fully connected layers. When building a CNN, the network designer must make many design decisions: the number of each type of layer, the order of the layers, and the hyperparameters of each type of layer, such as the size of the receptive field, the step size, and the number of receptive fields in each convolutional layer. By introducing reinforcement learning, the process of layer selection is automated. As long as the initial parameters and experimental data are given, a convolutional neural network that performs well under specified conditions can be automatically constructed.

[0037] The automatically constructed convolutional neural network is mainly composed of two parts. The first part is for feature extraction, which is at the front end of the network and consists of convolutional layers, pooling layers, and residual layers. The second part integrates all features for classification and is placed at the end of the network. It consists of fully connected layers and dropouts. The reinforcement learning algorithm acts on the feature extraction part of the network. After completing the construction of the feature extraction part of the network, the feature classification part is connected to obtain a complete classification convolutional neural network.

[0038] Since the feature extraction part is composed of convolutional layers and pooling layers of different sizes, the size of the output feature map is not fixed. Each node of the fully connected layer is connected to each node of the previous layer, and the output features of the previous layer are integrated. When calculating, the input feature map matrix and the fully connected layer weight matrix are inner-producted, so the feature map size is required to be fixed before inputting the full connection. For this reason, in a specific implementation, a spatial pyramid pooling layer is added at the end of the feature extraction part. Through spatial pyramid pooling, feature maps of different sizes can be pooled and outputs of fixed length can be generated, and finally input into the feature classification part to obtain the classification result. In this embodiment, the image is max-pooled with three different sizes of scales of 4, 3, 2, and 1 to obtain 10 features.

[0039] S2, define the state space and action space of the reinforcement learning agent:

[0040] Each state of the reinforcement learning agent is defined as a tuple of related layer parameters, and the action is the next set of layers that can be connected in the current state. In the reinforcement learning process, the state space of the agent is composed of various layers of CNN, and the action space of the agent is the layers that can be connected in the current state. The state transfer matrix P is unknown and needs to be learned. The reward R of the agent is the accuracy of CNN on the validation set determined by various layers.

[0041] The layer definition of the convolutional layer is simplified to a state tuple of related layer parameters, which consists of four different types of layers: convolution (C), pooling (P), fully connected (FC), residual network layer (RES) and softmax (SM). Although the composition of convolutional neural networks is not limited to these types, in order to converge faster, we have restricted the parameters, and different components can be added later according to actual needs. As shown in Table 1, this embodiment lists the relevant parameters and possible values ​​of each parameter for each layer type in the network.

[0042] Table 1

[0043]

[0044]

[0045] In addition, restricting the agent's action space makes convolutional neural networks easier to build. First, we allow the agent to terminate the path at any node, that is, it can choose a terminal state from any non-terminal state. In addition, we only allow states with layer depth i to transition to states with layer depth i+1. These restrictions ensure that the state-action graph is always a directed acyclic graph. As specified in Table 1, any state in the maximum layer depth can only transition to the terminal layer. An agent in a convolutional type state can transition to any layer type state. An agent in a pooling layer type can transition to any layer type state except the pooling type, because consecutive pooling layers can be equivalent to a single larger pooling layer.

[0046] S3 uses the reinforcement learning Q-Learning algorithm to allow the Agent to automatically build a series of convolutional neural networks:

[0047] The reinforcement learning algorithm Q-Learning restricts the environment to have a discrete and finite state space S and action space A. For any state s i ∈S, the agent can choose from a set of valid actions, In an environment with a random transition process, in state s i The agent takes some action a∈A(s i ) will be P s‘|s,a (s j |s i ,a) the probability of transitioning to state s j , this probability is unknown and needs to be learned by the agent.

[0048] At each time step t, the agent is given r according to the transition from state s to s′ and the action a taken. tThe agent's goal is to maximize the total expected reward over all possible trajectories, i.e. The trajectory T i The total expected reward is:

[0049]

[0050] The maximization problem is recursively defined in the form of subproblems as follows. For any state s i ∈S and the action a∈A(s i ), and define the maximum total expected reward as Q * (s i ,a).Q * (·) is called the action-value function, and a single Q * (s i ,u) is called the Q value. The recursive maximization equation is called the Bellman equation and can be written as

[0051]

[0052] In many cases, solving the Bellman equation analytically is impossible, but it can be expressed as an iterative update of the form

[0053]

[0054] For well-formed problems, if each transition is sampled infinitely many times, then lim t→∞ Q t (s,u)=Q * (s,u). The update equation has two parameters: (I) α is the rate of Q-learning, which determines the weight of new information and old information. The larger the learning rate α, the less the effect of previous training is retained. (II) γ is the decay coefficient, which determines the weight of short-term rewards and future rewards. The larger γ is, the more it values ​​past experience, and the smaller it is, the more it values ​​current interests.

[0055] Using the reinforcement learning Q-Learning algorithm to automatically build a convolutional neural network process is as follows:

[0056] S31, the agent selects layers sequentially through reinforcement learning ε-greedy strategy until reaching the termination state, at which point a CNN network model can be considered selected;

[0057] S32, train this CNN network model on the sonar image dataset, and use the accuracy of the model in the validation set as the reward of the reinforcement learning Q-Learning algorithm.

[0058] S33, use the reward on the validation set to update the Q value formula, and then continue step a according to the ε-greedy strategy until the number of constructed models reaches the limit.

[0059] In reinforcement learning, the agent needs to explore the surrounding environment continuously at the beginning, and then gain experience from previous attempts to find a good decision. If you blindly conduct random exploration, you will definitely not get a high reward, but if you use the best action known under the current experience every time, you may miss potential better actions. Therefore, in reinforcement learning, it is necessary to balance the two different stages of exploration (unknown areas) and utilization (existing knowledge). In this embodiment, the ε-greedy learning strategy is adopted, and the action with the highest Q value in the current state is selected with a probability of ε, and other actions are randomly selected with a probability of 1-ε. The formula is as follows:

[0060]

[0061] In the first step of the Q-Learning algorithm, it is necessary to find all the actions that can be taken in the current state according to the current environment. Figure 2 As shown, we first pass in the state. If the state is already a terminal state (the state terminate parameter is 1), then it cannot be converted to any other state in this state and returns directly. If the state is not a terminal state, then first determine whether it is a fully connected layer or a pooling layer. If any of the conditions are met, then the state can be marked as a terminal state, and a possible fully connected layer is added to the action space. If it is not a fully connected layer or a pooling layer, first determine whether the layer depth of the current state exceeds the maximum depth specified. If not, add all possible residual layers and convolutional layers to the action space, otherwise add a pooling layer to the action space. Following this process to find all actions in a certain state can ensure that the entire network consists of two halves. The first half consists of convolutional layers and residual layers for feature extraction, and the second half consists of pooling layers and fully connected layers to complete the classification of sonar images.

[0062] S4, select the network with better performance from the generated convolutional neural network, adjust the parameters and migrate it to the target detection network;

[0063] S5, supervised training of the target monitoring network in step S4 using the sonar image dataset to complete the target recognition task of sonar images. This process is deployed under ubuntu16.04, and the GPU version of pytorch.3 is configured as the framework. The graphics card is RTX 2070, and the graphics card driver is Cuda10.0 and Cudnn7.3.

[0064] Experimental test case

[0065] The Q-Learning algorithm models the convolutional network layer selection process as a Markov decision process, such as Figure 1 As shown in the figure. The convolutional neural network consists of several convolutional layers, pooling layers and full connections. As long as the appropriate layers are selected and the appropriate hyperparameters are determined, a complete convolutional neural network can be obtained. In the implementation test case, the original data consists of more than 200 sonar images of three categories (aircraft, shipwrecks and corpses). In order to expand the number of training samples and avoid overfitting, the data set was expanded, and finally there were more than 2,000 images.

[0066] In this embodiment, the learning rate (α) of the reinforcement learning Q-learning algorithm is set to 0.01, the decay coefficient (γ) is set to 1, and the ε-greedy strategy is used to sequentially select the layers in the neural network. During the experiment, ε is gradually reduced from 1.0 to 0, and the experimental results of the constructed convolutional network for the sonar image classification task are shown in Table 2. It can be seen that as ε decreases, the average accuracy of the network constructed by reinforcement learning continues to rise. When ε is 0, the average accuracy of the 20 classification convolutional networks constructed by reinforcement learning is 84.35. Among these 20 classification convolutional networks, the model with the highest performance has an accuracy of 86.7%.

[0067] Table 2: Average accuracy of ε plan and model in the embodiment

[0068]

[0069] In this embodiment, we constructed AlexNet and VGG16 networks as a comparison of classification task experiments, where AlexNet and VGG16 models were obtained by transfer learning using pre-trained models. The accuracy of AlexNet is 78.8%, the accuracy of VGG16 reaches 83.6%, and the accuracy of the best model obtained by our algorithm after multiple runs reaches 86.7%. In general experiments, if you want to get a model with good performance, you need to continuously optimize the parameters of the network through a large number of experiments. With the Q-Learning algorithm, we only need to set various hyperparameters, and the program can automatically build a convolutional neural network that performs well under given conditions, eliminating the trouble of manual parameter adjustment.

[0070] Then the convolutional neural network model with the highest classification accuracy was separated from the feature extraction part of the entire network and named M_net. Its network structure is as follows Figure 3 As shown in Figure 1, it is the best network automatically constructed by reinforcement learning. Then, M_net is combined with the region proposal network to form a complete target recognition network. The process is as follows: Figure 4As shown, firstly, a model with good classification performance is obtained through reinforcement learning, and then a sonar image target detection network is constructed through transfer learning and combined with a region proposal network. In this embodiment, the target recognition network consists of three parts: a shared convolutional network, a region proposal network, and a region-based fast convolutional network. The shared convolutional layer is used to extract the features of the image, which is the convolutional network model M_net obtained in the previous step. Its input is the entire image, and its output is the corresponding feature map.

[0071] Finally, the entire target detection network is trained and parameterized, mainly including the learning rate and the number of epochs. For example, a higher learning rate is conducive to speeding up the iterative update, but is not conducive to finding the local / global optimum; a lower learning rate is conducive to finding the local optimum, but may make the model unable to jump out of the local optimum and obtain the global optimum. In comparison, the target detection algorithm using vgg16 has an accuracy of about 58%, but it takes 50-60 milliseconds to process a picture. The results are shown in Table 3. After tuning, our M_net has an accuracy of 64% and a running time of just over 30 milliseconds, which greatly improves the running speed of the algorithm.

[0072] Table 3: Algorithm comparison

[0073]

[0074] In summary, the method in this case automates the construction process of convolutional neural networks based on the Q-Learning algorithm of reinforcement learning. By setting the initial parameters of the experiment, the reinforcement learning agent can build a convolutional neural network with good performance under specified conditions, and as the experiment proceeds, the accuracy of the convolutional neural network will continue to improve until it stabilizes, saving a lot of manual parameter adjustment input, obtaining a convolutional neural network with good performance, and improving the work efficiency of sonar image recognition. As shown in Figure 5(a), Figure 5(b) and Figure 5(c), the three images represent the partial detection results of the three types of sonar images of corpses, sunken ships and aircraft wreckage by the method in this case. In these images, the results of the network model's target recognition are represented by a rectangular box, and the recognized objects are placed in the rectangular box. The text in the picture is the model's classification prediction of the target, and the number represents the confidence of the prediction result. It can be seen from the figure that the objects in the three images are all enclosed in the rectangular box, the classification prediction results are correct, and the confidence of the prediction results is very high (more than 90%), indicating that the network model has well recognized the targets in the image, proving the effectiveness of this method.

[0075] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A sonar image target recognition method based on reinforcement learning, characterized by , including the following steps: S1, the layer selection process of the convolutional neural network is modeled as a Markov decision process: the convolutional neural network includes two parts: feature extraction and feature classification. Feature extraction is at the front end of the network, consisting of a convolutional layer, a pooling layer, and a residual layer; feature classification is at the end of the network; In the continuous interaction with the environment, the reinforcement learning agent selects the appropriate layer type and hyperparameters through the feedback of the environment to form the feature extraction part of the convolutional neural network, and then connects the feature classification part to form a complete classification convolutional neural network; S2, define the state space and action space of the reinforcement learning agent: the state space of the agent in the reinforcement learning process is composed of various layers of the convolutional neural network, and the action space of the agent is the layers that can be connected in the current state. Each state of the reinforcement learning agent is defined as a tuple of the relevant layer parameters. In the state tuple defined by the layer, the range of hyperparameters is restricted; S3, using the reinforcement learning Q-Learning algorithm to let the agent automatically build a series of convolutional neural networks. This step further includes: In S31, the agent selects the layer sequentially through reinforcement learning ε-greedy strategy, selects the action with the highest Q value in the current state with probability ε, and randomly selects other actions with probability 1-ε. The formula is as follows: Until the termination state is reached, it can be considered that a CNN network model is selected; S32, training the CNN network model selected in step S31 on the sonar image dataset, and using the accuracy of the model on the validation set as a reward for the reinforcement learning Q-Learning algorithm; S33, using the reward on the validation set to update the Q value formula, and then continuing step S31 according to the ε-greedy strategy of step S31 until the number of constructed models reaches a limit value; S4, select the network with better performance from the generated convolutional neural network, adjust the parameters and migrate it to the target detection network; S5, using the sonar image dataset to perform supervised training on the target monitoring network in step S4 to complete the target recognition task of the sonar image.

2. The sonar image target recognition method based on reinforcement learning as claimed in claim 1, characterized in that: In step S1, a spatial pyramid pooling layer is added at the end of the feature extraction part of the convolutional neural network.

3. The sonar image target recognition method based on reinforcement learning as claimed in claim 2, characterized in that: In step S2, in the state tuple defined by the layer, the action space of the agent is restricted, the agent is allowed to terminate the path at any node, and the state with a layer depth of i is allowed to be converted to a state with a layer depth of i+1.

4. The sonar image target recognition method based on reinforcement learning according to claim 2 or 3, characterized in that: The adjustment parameters in step S4 at least include adjustment of learning rate, optimizer, initialization method, batch size, weight decay and number of training times.

Images