An image recognition method and system for a remote sensing satellite, and a terminal device

Abstract

The application discloses an image recognition method and system for a remote sensing satellite, and a terminal device. The recognition method comprises the following steps: acquiring an image to be recognized; inputting the image to be recognized into a pre-constructed full convolutional neural network for pre-processing to obtain a noise-removed image; inputting the noise-removed image into a pre-constructed deep convolutional neural network, and recognizing the image by using the deep convolutional neural network; the pre-constructed deep convolutional neural network comprises two branch convolutional neural networks; and acquiring a recognition result of the image. The method improves the recognition efficiency by pre-processing the image to be recognized. The image can be directly input into the full convolutional neural network, thereby avoiding complex feature extraction, reducing the difficulty of data pre-processing, reducing the complexity of algorithm design, and improving the recognition accuracy. The two branch convolutional neural networks extract color features and shape features of the image respectively, thereby improving the image analysis capability and improving the detection accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of intelligent processing of visible light spectrum, and relates to an image recognition method and system for a remote sensing satellite and a terminal device. BACKGROUND

[0002] The remote sensing satellite is equipped with various visualizations, such as cameras, which can obtain a large number of photos for earth observation, have high resolution, small distortion, moderate scale and other advantages, and can be widely applied to scientific research and industrial and agricultural production fields. When the existing equipment takes photos, the system algorithm mainly considers the features of the photographed objects and the design of the classifier in identifying the photographed objects. Common classifiers include BP neural networks and SVMs. For object feature design, the following two aspects need to be considered: ①selecting features with large differences between different categories to improve the recognition effect and reduce the probability of recognition errors; and ②too many features will increase the complexity of calculation, the recognition result is not easy to converge, and the recognition efficiency is reduced. The existing method is relatively complex in feature selection, and the recognition result is not accurate enough, resulting in a large amount of manual interaction data between the on-board processing machine and the ground control station, and low efficiency of the satellite in on-orbit execution of tasks. SUMMARY

[0003] In view of the problems in the prior art, the application provides an image recognition method and system for a remote sensing satellite and a terminal device, thereby effectively solving the technical problems of inaccurate recognition result and low recognition efficiency of the remote sensing satellite.

[0004] The application is implemented by the following technical solutions:

[0005] An image recognition method for a remote sensing satellite, comprising the following steps:

[0006] S1: obtaining an image to be recognized;

[0007] S2: inputting the image to be recognized into a pre-constructed full convolutional neural network for preprocessing to obtain a noise-removed image;

[0008] S3: inputting the noise-removed image into a pre-constructed deep convolutional neural network, and using the deep convolutional neural network to recognize the image; the pre-constructed deep convolutional neural network comprises two branch convolutional neural networks, and the two branch convolutional neural networks extract color features and shape features of the image respectively;

[0009] S4: obtaining a recognition result of the image.

[0010] Preferably, the construction process of the full convolutional neural network is as follows:

[0011] S201: set the initial weight of the full convolutional neural network, input the existing image data after adding noise to the full convolutional neural network, train the full convolutional neural network, and obtain the denoised image;

[0012] S202: compare the denoised image with the image before adding noise to obtain the pixel average absolute error of the comparison image, and propagate the pixel average absolute error of the comparison image backward to obtain the error of each convolutional layer in the full convolutional neural network;

[0013] S203: correct the initial weight with the error of each convolutional layer in the full convolutional neural network obtained, and train the full convolutional neural network again with the corrected initial weight until the error of each convolutional layer in the full convolutional neural network meets the design requirements, and the construction of the full convolutional neural network is completed.

[0014] Preferably, the step S201 specifically inputs the existing image data after adding noise to the full convolutional neural network, and uses an n*n convolution kernel for convolution in the process of forward propagation of the image data with noise, and the matrix after convolution of the convolution kernel is nonlinearized by an activation function to obtain the denoised image output by the convolutional layer.

[0015] Preferably, the two branch convolutional neural networks each include an input layer, a plurality of convolutional layers and a plurality of maximum pooling layers, and the output layers of the two branch convolutional neural networks are both convolutional layers.

[0016] Preferably, the pre-constructed deep convolutional neural network further includes an identification network connected to the ends of the two branch convolutional neural networks; the identification network includes two maximum pooling layers, a convolutional layer and a fully connected layer.

[0017] Preferably, the construction process of the deep convolutional neural network is as follows:

[0018] S301: manually label the existing image data, and set the initial running parameters of the deep convolutional neural network;

[0019] S302: under the initial running parameters, train the deep convolutional neural network with the RGB image data and the grayscale image data of the existing image data, and output the network identification label;

[0020] S303: compare the manually set label with the network identification label to obtain the average absolute error of the comparison label, propagate the average absolute error of the comparison label backward to obtain the error of each convolutional layer in the deep convolutional neural network;

[0021] S304: correcting the initial operation parameters by using the errors of each convolutional layer in the deep convolutional neural network, and continuing to train the deep convolutional neural network by using the corrected initial operation parameters until the errors of each convolutional layer in the deep convolutional neural network meet the requirements, and completing the construction of the deep convolutional neural network.

[0022] Preferably, the step S302 specifically comprises: inputting the RGB image data and the gray image data of the existing image into two branch convolutional neural networks respectively for feature recognition, and then merging and inputting into a recognition network, and outputting a network recognition label.

[0023] An image recognition system for a remote sensing satellite, comprising:

[0024] An image acquisition module, configured to acquire an image to be recognized;

[0025] An image preprocessing module, configured to input the image to be recognized into a pre-constructed full convolutional neural network for preprocessing to obtain a noise-removed image;

[0026] An image recognition module, configured to input the noise-removed image into a pre-constructed deep convolutional neural network, and to recognize the image by using the deep convolutional neural network;

[0027] An image output module, configured to acquire a recognition result of the image.

[0028] A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to run the steps of the above method.

[0029] A computer readable storage medium, storing a computer program, wherein the computer program is executed by a processor to run the steps of the above method.

[0030] Compared with the prior art, the present application has the following beneficial technical effects:

[0031] The application discloses an image recognition method for remote sensing satellites, which comprises the following steps: preprocessing an image to be recognized by a pre-constructed full convolutional neural network, removing noise from the image, improving the recognition efficiency, directly inputting the collected image into the full convolutional neural network, simplifying the feature extraction, reducing the data processing difficulty and the algorithm complexity, and improving the accuracy; and recognizing the image by using a deep convolutional neural network after the preprocessing, wherein the deep convolutional neural network comprises two branch convolutional neural networks, the two branch convolutional neural networks extract color features and shape features of the image respectively, the image analysis capability is improved, and the detection accuracy is improved. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the application, and therefore should not be regarded as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0033] Figure 1 The application discloses a flowchart of an image recognition method for remote sensing satellites.

[0034] Figure 2 The application discloses a network structure framework involved in the image recognition method for remote sensing satellites.

[0035] Figure 3 The application discloses a structure diagram of an image recognition system for remote sensing satellites. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the application more clear, the following will combine the drawings in the embodiments of the application to clearly and completely describe the technical solutions in the embodiments of the application, and obviously, the described embodiments are some embodiments of the application, but not all the embodiments. The components of the embodiments of the application described and shown in the drawings can be arranged and designed in various different configurations.

[0037] Therefore, the following detailed description of the embodiments of the application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the protection of the application.

[0038] It should be noted that similar reference numerals and letters refer to like items throughout the accompanying drawings, and once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings.

[0039] In the description of the embodiments of the present application, it should be noted that if the terms "upper", "lower", "horizontal", "inner" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the present application is usually placed, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0040] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly inclined. For example, "horizontal" only means that its direction is relatively more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0041] In the description of the embodiments of the present application, it should also be noted that unless otherwise explicitly specified and limited, if the terms "arrangement", "installation", "connection", "connection" appear, they should be understood in a broad sense. For example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication between two elements inside. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0042] The present application will be described in further detail below in conjunction with the accompanying drawings:

[0043] As shown in the drawings, a kind of image recognition method for remote sensing satellite, comprising the following steps: Figure 1

[0044] S1: obtaining the image to be identified;The image is the visible spectrum image information of three wave bands near 700mm (R), 550mm (G) and 440mm (B), which can be captured by a CCD camera.

[0045] S2: input the image to be identified into the pre-constructed full convolutional neural network for pre-processing, to obtain the image removed noise;In the pre-processing stage, the data set is expanded by image enhancement to prevent overfitting phenomenon in the model training process;

[0046] The construction process of the full convolutional neural network is as follows: ​

[0047] S201: set the initial weight of the full convolutional neural network, input the existing image data after adding noise to the full convolutional neural network, train the full convolutional neural network, and obtain the denoised image;

[0048] Specifically: input the existing image data after adding noise to the full convolutional neural network, use an n*n convolution kernel for convolution in the process of forward propagation of the image data with added noise, and perform non-linearization on the matrix after convolution by the convolution kernel through an activation function to obtain the denoised image output by the convolution layer;

[0049] The convolution formula is:

[0050]

[0051] In the formula, x is the matrix to be convolved;

[0052] x i+u,j+v is the point at the (i+u, j+v) position;

[0053] Z (u,v) is the value of the point after convolution;

[0054] k is an n*n convolution kernel;

[0055] k rot is a matrix obtained by rotating k by 180°;

[0056] The activation function in step S202 is a Relu function, and the Relu function formula is:

[0057] f (x) =max(0, x), max is used to obtain the maximum value of 0 and x.

[0058] S202: compare the denoised image with the image before adding noise to obtain the pixel average absolute error of the comparison image, propagate the pixel average absolute error of the comparison image backward, and sequentially obtain the error of each convolution layer in the full convolutional neural network;

[0059] Wherein, the loss function of comparing the denoised image with the image before adding noise is the pixel average absolute error of the comparison image, and the loss function is:

[0060]

[0061] In the formula, y i is the actual value, i.e. the pixel of the image before adding noise;

[0062] is the predicted value, i.e. the pixel of the denoised image;

[0063] n is the total number of training samples.

[0064] S203: Correct the initial weights using the errors of each convolutional layer in the obtained fully convolutional neural network, and retrain the fully convolutional neural network using the corrected initial weights until the errors of each convolutional layer in the obtained fully convolutional neural network meet the design requirements, thus completing the construction of the fully convolutional neural network.

[0065] The network structure of the above fully convolutional neural network is shown in Table 1.

[0066] Table 1. Network structure of the fully convolutional neural network in this invention.

[0067]

[0068] Note: ①input: Input a 224×224×3 image; ②convn-m: Convolve m n×n kernels and activate them through a ReLU layer.

[0069] S3: Input the noise-removed image into a pre-built deep convolutional neural network, and use the deep convolutional neural network to recognize the image;

[0070] like Figure 2 As shown, the pre-constructed deep convolutional neural network includes two branch convolutional neural networks and a recognition network connected to the ends of the two branch convolutional neural networks. The recognition network uses convolutional layers with one-dimensional convolutional kernels instead of fully connected layers to form a fully connected module, which does not destroy the spatial structure of the image, and the input can be of any size. Both branch convolutional neural networks include an input layer and several convolutional layers and max-pooling layers spaced apart. Simultaneously, the output layers of both branch convolutional neural networks are convolutional layers. The recognition network includes two max-pooling layers, convolutional layers, and fully connected layers. The two branch convolutional neural networks extract color and shape features from the image respectively. Different features are extracted using two convolutional layers, and then the recognition network performs recognition, improving the feature analysis capability. The input image of the branch convolutional neural network extracting color features is an RGB image, which can be 224*224 in size. The input image of the branch convolutional neural network extracting shape features is a grayscale image, which can be 224*224 in size. The grayscale image conversion formula is:

[0071] Gray=R×0.299+G×0.587+B×0.114

[0072] In the formula, Grad is the gray value of the image at that point;

[0073] R, G, and B are the values ​​of the R, G, and B channels of the image at that point.

[0074] The network structure table of the deep convolutional neural network is shown in Table 2.

[0075] Table 2 Network structure table of deep convolutional neural network in the application

[0076]

[0077]

[0078] Note: ①input: input 224x224x3 picture; ②convn-m: convolution with m n*n convolution kernel, and activated by ReLU layer; ③maxpool: maximum pooling, with 2x2 pooling unit size for pooling; ④fc n: convolution with 1x1x n convolution kernel, full connection, and activated by ReLU layer.

[0079] In addition, in this step, the construction process of the deep convolutional neural network is:

[0080] S301: manually label the images in the existing image data set, and set the initial running parameters of the deep convolutional neural network;

[0081] S302: under the initial running parameter condition, train the deep convolutional neural network by using the RGB image data and the gray image data of the existing image data, and output the network recognition label;

[0082] Specifically, the RGB image data and the gray image data of the existing image are respectively subjected to feature recognition by two branch convolutional neural networks, and then input to the recognition network for merging, and the network recognition label is output;

[0083] In this step, the convolution formula involved is:

[0084]

[0085] In the formula, x is the matrix to be convolved;

[0086] x i+u,j+v is the point at the position (i+u, j+v);

[0087] Z (u,v) is the value of the point after convolution;

[0088] k is an n*n convolution kernel;

[0089] k rot is a matrix obtained by rotating k by 180°.

[0090] The maximum pooling formula involved is:

[0091] y ij= max(x i+r,j+s ), i≤m-n, j≤m-n;

[0092] x is an m x m matrix, the convolution kernel size is n x n, and max is used to obtain the maximum value;

[0093] The matrix after convolution by the above convolution kernel is subjected to an activation function to obtain the output of the convolution layer. In the feature merging process, different features are directly superimposed and transmitted to the pooling layer.

[0094] The activation function is a Relu function, specifically:

[0095] f (x) = max(0, x);

[0096] max is used to obtain the maximum value of 0 and x,

[0097] S303: Compare the manually set label with the network recognition label to obtain the average absolute error of the comparison label, and propagate the average absolute error of the comparison label backward to obtain the error of each convolution layer in the deep convolutional neural network.

[0098] Wherein, wherein, the loss function of comparing the manually set label with the network recognition label is the average absolute error of the comparison label, and the loss function is:

[0099]

[0100] In the formula, y i is the actual value, i.e. the manually set label;

[0101] is the predicted value, i.e. the network recognition label;

[0102] n is the total number of training samples.

[0103] S304: Correct the initial operating parameters using the error of each convolution layer in the deep convolutional neural network, and continue to train the deep convolutional neural network using the corrected initial operating parameters until the error of each convolution layer in the deep convolutional neural network meets the requirements, and the construction of the deep convolutional neural network is completed.

[0104] The existing image data used in the above step S201, the noise data and the existing image data used in step S301 are randomly collected and randomly generated.

[0105] S4: Obtain the recognition result of the image.

[0106] The application provides an image recognition method for a remote sensing satellite, wherein a full convolutional neural network is constructed in advance, and an image to be recognized is preprocessed by using the full convolutional neural network, so that noise of the image is removed, and the recognition efficiency is improved; meanwhile, the collected image can be directly input into the full convolutional neural network, so that the complex feature extraction is avoided, the difficulty of data preprocessing is reduced, the complexity of algorithm design is reduced, and the recognition accuracy is improved. After the preprocessing, the image recognition is realized by using a deep convolutional neural network, the deep convolutional neural network comprises two branch convolutional neural networks, the two branch convolutional neural networks extract color features and shape features of the image respectively, the image analysis capability is improved, and the detection accuracy is improved. Due to the environmental factors such as light and background during the sample collection, compared with the manual selection of a sample feature with high robustness, the deep convolutional neural network can avoid the complex feature algorithm, and the recognition accuracy is improved.

[0107] The image recognition method based on the deep convolutional neural network is applied to a next-generation intelligent and autonomous control remote sensing satellite, and the collection and processing of task demand data are completed, so that the automatic and intelligent control of the satellite can be effectively realized.

[0108] As shown in Figure 3 a system for image recognition on a remote sensing satellite, comprising:

[0109] an image acquisition module, configured to acquire an image to be recognized;

[0110] an image preprocessing module, configured to input the image to be recognized into a pre-constructed full convolutional neural network for preprocessing, so as to obtain an image from which noise is removed;

[0111] an image recognition module, configured to input the image from which noise is removed into a pre-constructed deep convolutional neural network, and to recognize the image by using the deep convolutional neural network;

[0112] an image output module, configured to acquire a recognition result of the image.

[0113] The embodiment of the application further provides a terminal device, which comprises a processor, a memory, and a computer program stored in the memory and executable on the processor. The processor implements each step in the above infrared video action recognition process when executing the computer program. Alternatively, the processor implements the functions of each module / unit in the above each device embodiment when executing the computer program.

[0114] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the application.

[0115] The terminal device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The terminal device can include, but is not limited to, a processor and a memory.

[0116] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like.

[0117] The memory can be used to store the computer programs and / or modules, and the processor can realize various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory, and by calling data stored in the memory.

[0118] The modules / units integrated in the terminal device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the present application can realize each step or part of the process in the infrared video action recognition process in the above-mentioned embodiment method, and can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and the computer program can realize each step in the above-mentioned infrared video action recognition process when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the computer-readable medium can include or exclude contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0119] The above merely describes the preferred embodiments of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image recognition method for a remote sensing satellite, characterized in that, The method comprises the following steps: S1: obtaining an image to be identified; S2: inputting the image to be identified into a pre-constructed full convolutional neural network for preprocessing to obtain a noise-removed image; S3: inputting the noise-removed image into a pre-constructed deep convolutional neural network, and identifying the image by using the deep convolutional neural network; the pre-constructed deep convolutional neural network comprises two branch convolutional neural networks which extract color features and shape features of the image respectively; S4: obtaining an identification result of the image; The construction process of the full convolutional neural network is as follows: S201: setting an initial weight of the full convolutional neural network, inputting existing image data with added noise into the full convolutional neural network, training the full convolutional neural network to obtain a noise-removed image; S202: comparing the noise-removed image with the image before adding noise to obtain a pixel average absolute error of the comparison image, backward propagating the pixel average absolute error of the comparison image to obtain errors of all convolutional layers in the full convolutional neural network; S203: correcting the initial weight by using the errors of all convolutional layers in the full convolutional neural network, training the full convolutional neural network again by using the corrected initial weight until the errors of all convolutional layers in the full convolutional neural network meet design requirements, and completing the construction of the full convolutional neural network; The construction process of the deep convolutional neural network is as follows: S301: performing artificial labeling processing on existing image data, and setting initial running parameters of the deep convolutional neural network; S302: training the deep convolutional neural network by using RGB image data and grayscale image data of the existing image data under the initial running parameters, and outputting network identification labels; S303: comparing the artificial labels with the network identification labels to obtain an average absolute error of comparison labels, backward propagating the average absolute error of the comparison labels to obtain errors of all convolutional layers in the deep convolutional neural network; S304: correcting the initial running parameters by using the errors of all convolutional layers in the deep convolutional neural network, and continuing to train the deep convolutional neural network by using the corrected initial running parameters until the errors of all convolutional layers in the deep convolutional neural network meet requirements, and completing the construction of the deep convolutional neural network.

2. The image recognition method for a remote sensing satellite according to claim 1, wherein, The step S201 specifically comprises adding noise to the existing image data and then inputting the image data to a full convolutional neural network. In the process of propagating the image data with noise forward, a convolution kernel is used for convolution, and a matrix after convolution by the convolution kernel is nonlinearized by an activation function to obtain a denoised image output by a convolution layer. The step S201 specifically comprises adding noise to the existing image data and then inputting the image data to a full convolutional neural network. In the process of propagating the image data with noise forward, a convolution kernel is used for convolution, and a matrix after convolution by the convolution kernel is nonlinearized by an activation function to obtain a denoised image output by a convolution layer.

3. The method for image recognition on a remote sensing satellite according to claim 1, wherein, Both of the two branch convolutional neural networks comprise an input layer, a plurality of convolutional layers and a plurality of maximum pooling layers, and the output layers of the two branch convolutional neural networks are both convolutional layers.

4. The method for image recognition on a remote sensing satellite according to claim 1, wherein, The pre-constructed deep convolutional neural network further comprises an identification network connected to the ends of the two branch convolutional neural networks; the identification network comprises two maximum pooling layers, a convolutional layer and a fully connected layer.

5. The method for image recognition on a remote sensing satellite according to claim 1, wherein, The step S302 specifically comprises inputting the RGB image data and the grayscale image data of the existing image into the two branch convolutional neural networks for feature identification, and then merging the image data into the identification network to output network identification labels.

6. An image recognition system for use on a remote sensing satellite, characterized by A method for image recognition on a remote sensing satellite for implementing any one of claims 1-5, comprising: an image acquisition module for acquiring an image to be recognized; an image preprocessing module for inputting the image to be recognized into a pre-constructed full convolutional neural network for preprocessing to obtain a noise-removed image; an image recognition module for inputting the noise-removed image into a pre-constructed deep convolutional neural network and using the deep convolutional neural network to recognize the image; an image output module for acquiring a recognition result of the image.

7. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to run the steps of the method of any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to run the steps of the method of any one of claims 1-5.

Images