An image processing and neural network fusion-based water organism recognition and matching method

Abstract

This invention discloses a method for aquatic organism identification and matching based on image processing and neural network fusion, belonging to the fields of underwater robotics and marine resource development. In the underwater environment, due to the attenuation of the red light channel, images captured by camera equipment generally have a greenish tint, and identification of various aquatic organisms is difficult and slow. Existing algorithms directly identify organisms based on the acquired underwater images, resulting in poor accuracy and slow processing speed due to depth estimation applied to the entire image. This invention performs red light channel compensation on the original underwater image, and image preprocessing achieves underwater image restoration. Then, a dedicated DarkNet neural network is used to identify and classify various aquatic organisms. Finally, based on a binocular camera, targets that meet threshold requirements are identified using DBNet for ORB matching to obtain matching features, facilitating subsequent depth calculation by the binocular camera. Compared with existing algorithms, this method is easier to use, and improves the quality of the restored image, the underwater organism identification rate, the depth estimation accuracy, and the processing speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of underwater robots for catching wild or cultured aquatic organisms, and marine resource development, and particularly relates to an aquatic organism recognition and matching method based on image processing and neural network fusion. BACKGROUND

[0002] Machine vision has achieved great success in the field of classification and recognition, and object recognition in underwater environment is the key to the transformation of artificial fishing to automatic fishing. The accuracy of aquatic organism target detection and the accuracy of aquatic organism target depth estimation directly affect the accuracy of subsequent underwater robot aquatic organism target grabbing.

[0003] Underwater robots are widely used in marine resource exploitation, marine organism exploration and submarine engineering applications, and play an important role in the seabed that humans cannot access or long-term stay. However, in the underwater environment, mainly due to the lack of red channel values in the image, the overall image shows a greenish state. And the number of aquatic organisms is too large, and the environment is relatively complex and variable, and target classification and recognition is still a key and difficult point. Moreover, the traditional binocular estimation algorithm has poor real-time performance in underwater environment, and the processing speed is slow.

[0004] Therefore, the present application is an aquatic organism recognition and matching method based on image processing and neural network fusion. For the original underwater image, the red light channel compensation is performed, and the image preprocessing is performed to restore the underwater image. Then, the DBNet neural network is used to recognize and classify the underwater aquatic organisms. Finally, based on the binocular camera device, the neural network output target frame is used for threshold calculation, the target area meeting the requirements is extracted by ORB feature extraction and depth estimation. Thus, the problems of difficult seabed image processing, difficult aquatic organism recognition and classification, and difficult underwater depth estimation are solved, and rich information is provided for subsequent underwater robot grabbing. SUMMARY

[0005] The present application proposes an aquatic organism recognition and matching method based on image processing and neural network fusion, which includes the following contents:

[0006] Step one, image preprocessing module. In order to solve the problem of underwater image distortion, increase the contrast of the image, and improve the subsequent classification and recognition accuracy. The original seabed image I collected by the camera is output through the red light compensation module, the white balance module, and the contrast enhancement module. balance .

[0007] Step two, water organism identification module. In order to reduce the processing speed and improve the accuracy of multiple water organism identification, the application proposes DBNet neural network. For the water organism classification task self-made data set, for the DBNet neural network based on yolov3 network, the DBL minimum unit and the DB minimum module are proposed, and the loss function is improved.

[0008] Step three, underwater binocular image matching algorithm. In order to reduce the processing amount, speed up the overall matching speed and improve the overall matching accuracy, the application proposes an ORB matching algorithm based on the recognition box constraint. The overall matching area is limited to the recognition box area of the left and right image threshold output; and the ORB feature matching is carried out for the threshold recognition box.

[0009] The main implementation process of the step one image preprocessing module is as follows:

[0010] 1. Red light compensation module. According to the underwater environment, only the red channel needs to be processed. Let the original input image be I, I r , I g , I b is the red, green and blue color channel set of the original image, I r (x, y), I g (x, y), I b (x, y) represents a pixel point of a red, green and blue channel size WxH image, wherein x and y are respectively the horizontal index and vertical index of the image pixel, and the value is limited in the range of [0, 1] by normalization, and is solved according to the following formula:

[0011]

[0012]

[0013]

[0014] The green channel mean value is calculated, and the calculation formula is as follows:

[0015]

[0016] Then the red light compensation formula is as follows:

[0017]

[0018] Wherein α g is a constant value, is the red light channel value obtained after the input picture is compensated by red light, instead of the original red light channel.

[0019] Then the improved red light channel image is

[0020] 2. white balance module.

[0021] The average of the blue and red color channels is calculated for the red channel value after compensation:

[0022]

[0023]

[0024] The average of the overall three channel sum is calculated Then:

[0025]

[0026] The final output color channel value I is calculated r ", I g ", I b Instead of the original image color channel value, the formula is:

[0027]

[0028]

[0029]

[0030] Then the white balance improved image I is white = {I r ", I g ", I b "}.

[0031] 3. Contrast enhancement module. For underwater images, contrast enhancement is the process of separating objects.

[0032] Gamma correction is a commonly used method in histogram correction, which is to correct the overall contrast of underwater pictures. By changing the gamma value, the overall performance of the image is changed, and in this invention, the default gamma is 1.3. The formula is as follows:

[0033]

[0034] Sharpening is the most commonly used technique to highlight the details of the edges of the image, which can handle areas that cannot be gamma corrected, to eliminate the problems of insufficient exposure and overexposure caused by gamma correction. This invention uses Priwitt sharpening, which differentiates in one direction and averages in the other direction, and is relatively insensitive to noise, which has the effect of suppressing noise.

[0035] I ga After x-direction convolution, G is obtained x(x, y), then the formula is:

[0036] G x (x, y) = [I ga (x-1, y-1) + I ga (x-1, y+1) + I ga (x-1, y+1)] - [I ga (x+1, y-1) + I ga (x+1, y) + I ga (x+1, y+1)]

[0037] I ga (x, y) is obtained by y direction convolution y (x, y), then the formula is:

[0038] G y (x, y) = [I ga (x-1, y+1) + I ga (x, y+1) + I ga (x+1, y+1)] - [I ga (x-1, y-1) + I ga (x, y-1) + I ga (x+1, y-1)]

[0039] I ga (x, y) is obtained by two direction convolution and summation balance (x, y) then the formula is:

[0040] I balance (x, y) = G x (x, y) + G y (x, y)

[0041] The second step of the water biological recognition module mainly realizes the following process:

[0042] 1. The basic module of DBNet is DBL (Dense Block Layer), which includes a convolution layer, a BN (Batch Normalization) layer and a Mish activation layer. The specific implementation steps of using DBNet algorithm are as follows:

[0043] (1) The Mish activation function ensures the smoothness at each point, ensures the existence of gradient at each point, and can also be not completely truncated when the input is negative, allowing small negative gradient to flow, and avoiding the problem of gradient saturation. The formula of Mish activation function is as follows:

[0044] Mish(x) = x*tanh(ln(1+e x ))

[0045] (2), in order to accelerate the convergence of neural network model, increase BN layer, specific derivation formula as follows:

[0046] Let the input of the i-th layer is x (l) , the output of a certain neuron is y (l) , that is:

[0047] y (l) =f(x (l) )=f(wy (l-1) +b))

[0048] Where f(·) is the activation function, w, b are all training parameters. In order to improve the optimization efficiency, the net input x (l) data distribution into normal distribution. The present application will be normalized to the standard normal distribution, normalization layer after affine transformation, before the activation function. That is:

[0049]

[0050] Where, E(x (l) ) and var(x (l) ) refers to the current parameters, x (l) Each dimension in the entire training set on the expectation and variance. The main optimization algorithm used in the present application is based on the small batch random gradient descent algorithm, usually using the mean and variance of the current small batch sample to replace the expectation and variance. Given a small batch sample set, which contains K samples, let (x (1，l) ,..., x (K，l) ) is the mean and variance of the net input of the l-th layer neuron is:

[0051]

[0052]

[0053] To ensure the nonlinearity of the network after normalization, the value range is changed by adding scaling and translation changes, that is:

[0054]

[0055] The above formula can be expressed as BN γ，β (x (l) ), wherein γ, β represent the scaling and translation parameter vector respectively, BN layer can be seen as a special neural layer, added before the nonlinear mapping function, after the convolution layer operation layer, that is:

[0056] y (l) =f(BN γ，β (x (l) ))=f(BNγ，β (wy (l-1) ))

[0057] 2. The sub-modules that make up the DBNet of this invention are DB (DenseBlock). In each sub-module, the input of each layer is the output of all layers above it. That is, all shallow features in this module can be directly input into subsequent sub-modules, so that all effective feature data within the module can be reused. In order to control the amount of computation, the number of feature maps output by the convolutional layers in the original network structure DBL module is greatly reduced.

[0058] 3. The formula for the DBNet localization loss function (CIoU) is as follows:

[0059]

[0060] Where p1 and p2 represent the center points of the predicted and ground truth bounding boxes, respectively, and ρ represents the Euclidean distance between the two center points. l is the diagonal length of the minimum closed region between the two bounding boxes. αv is the penalty term coefficient, which accelerates network convergence. α is the weight function, and v measures the similarity in aspect ratio. The formulas for α and v are as follows:

[0061]

[0062] Among them W gl H gl W and H are the width and height of the ground truth bounding box at a fixed position relative to the input image, respectively, and the width and height of the target bounding box predicted by the algorithm relative to the input image.

[0063] The DBNet confidence score loss and category loss are as follows:

[0064]

[0065] The loss function in the above formula introduces a balance factor λ to balance the imbalance between positive and negative samples. In the experiment of this invention, λ is selected as 0.25, meaning that the proportion of positive samples is smaller than that of negative samples. However, adding only the balance factor can only solve the problem of imbalance between positive and negative samples, but cannot solve the problem of easy and difficult samples. Therefore, a parameter γ is added to adjust the rate at which the weight of easy samples is reduced. When γ is 0, the loss function is the cross-entropy loss function. As γ increases, the influence of the adjustment factor also increases. In the experiment of this invention, γ is selected as 2.

[0066] The category loss function of this invention is an improvement on YOLOv3, resulting in the final loss function, as shown in the following formula:

[0067] L myloss =L CIOU +L conf +L yolov3_cls

[0068] 3. The main implementation process of the underwater binocular image matching algorithm in step three is as follows:

[0069] (1) Let the region map identified in the left figure be P. l The region detected in the right image is designated as P. r Calculate P l With P r The distance between them is set to 2.6 in this invention. If the distance is less than the threshold ∈, proceed to the second step. The distance formula is as follows:

[0070]

[0071] (2) Feature point extraction. The grayscale values ​​of a given pixel are compared with those of pixels on a circle of fixed radius surrounding it. Let the grayscale value of the center pixel be I. p The grayscale value of the pixels on a circle with a radius of 3 and 16 pixels is I. p→x ΔI is the set grayscale threshold. Then the classification S of the surrounding pixels... p→x The formula is as follows:

[0072]

[0073] If, among the surrounding 16 points, the number of consecutive pixels classified as d (darker) or b (brighter) is greater than N (N is typically 12), then that point is considered a potential feature point. To speed up the process, during actual feature extraction, we can first choose to compare only pixels 1, 5, 9, and 13 among the surrounding 16 points. If the number of pixels belonging to d or b is greater than or equal to 3, then we further verify these pixels using the 16-point detection method.

[0074] Using the above method, a large number of possible feature points can be extracted from the region to be detected. However, the detection order and the distribution of feature points around the image edges may reduce the efficiency of image feature extraction. Therefore, in the ORB algorithm, a decision tree trained by the ID3 algorithm is usually used to select the optimal point. Furthermore, feature points may be clustered together; non-maximum suppression is used to further filter the pixels. Let the gray value of the center coordinate be p, and the gray values ​​of the surrounding N consecutive pixels greater than or less than a threshold t be determined by v. i The evaluation index function is as follows:

[0075]

[0076] When two possible feature points P and Q are connected, the evaluation index function values ​​V of these pixels are compared, and the points with smaller V values ​​are discarded, thus completing the selection of feature points.

[0077] Technical effects: The present application aims at the original underwater image, carries out red light channel compensation, and realizes the restoration of underwater image through image preprocessing. Then a special DarkNet neural network is used to identify and classify various aquatic organisms underwater. Finally, based on the binocular camera equipment, the target meeting the threshold requirement is matched by using the DBNet to obtain the matching features, which is convenient for subsequent binocular camera depth calculation. Compared with the existing algorithm, the use is convenient, the image quality, underwater organism recognition rate and estimated depth precision processing speed are improved.

[0078] Drawings

[0079] Figure 1 . System flowchart.

[0080] Figure 2 .DB-Net structure diagram.

[0081] Figure 3 .DB-Net network training flowchart.

[0082] Figure 4 . Network output variable diagram.

[0083] Figure 5 . Binocular camera mathematical approximation model diagram.

[0084] Figure 6 . The left graph is the original underwater image, and the right graph is the preprocessed effect diagram.

[0085] Figure 7 . The left graph is the original image through DBNet effect diagram, and the right graph is the preprocessed image through DBNet effect diagram.

[0086] Figure 8 .DBNet identification of different species effect diagram.

[0087] Figure 9 .DBNet identification of a large number of similar sea urchins effect diagram.

[0088] Figure 10 .DBNet identification of different sea urchins effect diagram.

[0089] Figure 11 . ORB overall image processing matching effect diagram.

[0090] Figure 12 . After using the DBNet to segment in advance, the effect of using the constraint ORB matching Figure 1 .

[0091] Figure 13 . After using the DBNet to segment in advance, the effect of using the constraint ORB matching Figure 2 . DETAILED DESCRIPTION

[0092] The technical solutions of the present application will be described clearly and completely below in conjunction with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The embodiments listed below are only for further understanding and implementing the technical solutions of the present application, and do not constitute further limitation on the claims of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0093] The present application proposes a water organism recognition and matching method based on image processing and neural network fusion, and the system flow chart is as Figure 1 The complete method can be described by the following steps:

[0094] Step one, image preprocessing module. Solve the distortion problem of underwater images, increase the contrast of images, and can improve the subsequent classification and recognition accuracy, and get high-quality images P balance The implementation process is as follows:

[0095] 1. Red light compensation module.

[0096] Because the underwater image is relatively blurred and the visualization is low, the main reason is that the red channel is attenuated to a small value, and the overall photo color is blue-green.

[0097] For the problem of complete attenuation of the red channel, the present application refers to the opponent color theory to explain that the disappearance of the red channel is to make the green channel more deviated from the origin. Therefore, the color channel compensation compensates for the lost channel by bringing the red channel back to the mean origin. And the unbalanced attenuation in the traditional algorithm causes color deviation, the dark channel prior cannot estimate the propagation, and the histogram stretching causes noise amplification.

[0098] According to the underwater environment, only the red channel needs to be processed for the opponent channel, assuming that the original input image is I, I r , I g , I b is the red, green and blue color channel set of the original image, I r (x, y), I g (x, y), I b (x, y) represents a pixel point of a red, green and blue channel size WxH image, wherein x and y are respectively the horizontal index and vertical index of the image pixel, and the value is limited in the range of [0, 1] by normalization, and is solved according to the following formula:

[0099]

[0100]

[0101]

[0102] The green channel mean value is calculated, and the calculation formula is as follows:

[0103]

[0104] The red light compensation formula is as follows:

[0105]

[0106] Wherein, α g is a constant value, is the red light channel value obtained after the input picture is compensated by red light, instead of the original red light channel.

[0107] The red light channel improved image is as follows:

[0108] 2. The white balance module is a common technique for eliminating color distortion of underwater images.

[0109] Let I b be the blue channel set of the original image, be the red channel value compensated by the red channel, and the mean value of the blue and red color channels is calculated:

[0110]

[0111]

[0112] The mean value of the sum of the three channels is calculated Then, we have:

[0113]

[0114] The final output color channel value I r is calculated, g and I b is used to replace the original image color channel value, and the formula is as follows:

[0115]

[0116]

[0117]

[0118] The white balance improved image I white is as follows: r {I g ″, I b ″}.

[0119] 3. Contrast enhancement module. For underwater images, contrast enhancement is the process of separating objects.

[0120] Gamma correction is a commonly used method in histogram correction, which is to correct the overall contrast of underwater pictures. By changing the value of gamma, the overall performance of the image is changed. In this invention, the default value of gamma is 1.3. Then the formula is as follows:

[0121]

[0122] Sharpening is the most commonly used technique to highlight the details of the edges of the image. It can handle areas that cannot be gamma corrected, and it can eliminate the problem of insufficient exposure and overexposure caused by gamma correction. This invention uses Priwitt sharpening. Priwitt operator differentiates in one direction and averages in the other direction, and it is relatively insensitive to noise, which has the effect of suppressing noise.

[0123] I ga After x-direction convolution, G x (x, y) is obtained, and the formula is as follows:

[0124] G x (x, y) = [I ga (x-1, y-1) + I ga (x-1, y+1) + I ga (x-1, y+1)] - [I ga (x+1, y-1) + I ga (x+1, y) + I ga (x+1, y+1)]

[0125] I ga After y-direction convolution, G y (x, y) is obtained, and the formula is as follows:

[0126] G y (x, y) = [I ga (x-1, y+1) + I ga (x, y+1) + I ga (x+1, y+1)] - [I ga (x-1, y-1) + I ga (x, y-1) + I ga (x+1, y-1)]

[0127] I ga After two-direction convolution and summation, I balance (x, y) obtained by Priwitt sharpening processing is obtained, and the formula is as follows:

[0128] I balance (x, y) = Gx (x, y) + G y (x, y)

[0129] Step two, water organism identification module.

[0130] The data set used by the present application is a laboratory collected and manually labeled data set, including 30 kinds of water organisms, including Tripneustes gratilla, Temnopleurus toreumatcus, Echinometra mathaei, Strongylocentrotus nudus, Hemicentrotus Pulcherrimus, Diadema setosum, Diadema palmeri, Heterocentrotus mammillatus, Featherscale urchin, Pseudocentrotus deprssus, Anthocidaris crassispina, Actinopyga mauritiana, Holothuria Mexicana, Holothuria hilla, Thelenota ananas, Red polar ginseng, Holothuria pardalis, Stuchopus japonicus, Halodeima atra, Starfish, Fish, Crab, Shell, Conch, Actinia, Shrim, Turtle, Hippocampus, Squid and Scaleph. A total of 2302 water organism images are included, of which 1841 are included in the training set and 461 are included in the test set.

[0131] Using the preprocessed image I balance Target recognition is performed using the DBNet neural network for underwater organisms. The present application improves the backbone part of yolov3 to propose DBNet and adds a spatial pyramid pooling structure in the feature fusion part, further improving the detection accuracy of the algorithm, and the DBNet network structure diagram is as follows Figure 2 .

[0132] Assuming the input image size is W x H, after the operation of the network model, three sizes of feature map sizes will be output, respectively: For example, when the input image size is 416 x 416 x 3, the output feature map sizes are 13 x 13, 26 x 26 and 52 x 52, respectively.

[0133] Assuming that the number of predicted categories is N class = 30, the number of channels output by each scale is equal, which is 3 x (N class + 5), wherein 3 represents that 3 aspect ratio bounding box predictions are output for each scale, and each aspect ratio prediction can be described as (t x , t y , t W , t H , t conf , class), that is, target center coordinates, bounding box width and height, confidence score and belonging category.

[0134] Since the prediction data output by the yolov3 network model are all offset values, the loss function is not mapped to the original image size for calculation during network training. Assuming that the size of the output feature map of a certain size is 13 x 13, the left upper corner coordinate of the cell responsible for predicting the target in the network, in order to perform loss calculation, the prediction output of the network needs to be uniformly constrained and mapped to the same space as the labeled data. The constraint method is normalization. First, the form of the left upper corner and right lower corner coordinates of the labeled data is converted into the form of the target center point coordinates and the bounding box width and height, and it is normalized relative to the size of the input image. The real bounding box width and height after normalization need to be calculated IoU with the anchor width and height after normalization, so as to obtain the anchor with the best matching degree for loss calculation with the bounding box offset value predicted by the network. The values of the 9 groups of anchors participating in the training are obtained by the kmeans clustering algorithm. The kmeans algorithm classifies the input data and finds out 9 groups of clustering centers as prior bounding boxes for subsequent bounding box regression. The training network diagram is as shown in Figure 3 .

[0135] The coordinate offset and the bounding box offset are normalized relative to the feature map. After normalization, the offset information is changed into the bounding box positioning information in the feature map, and the operation formula is as follows:

[0136] b x = σ (t x ) + g x

[0137] b y = σ (t y ) + g y

[0138]

[0139]

[0140] Among them, (g x g y (p) represents the offset of the top-left corner coordinate of the cell responsible for predicting the target. w p h ) represents the width and height of the corresponding anchor, σ(·) is the sigmoid function, and the calculation yields the prediction result (b) on the feature map size. x b y b w b H , t conf (class), such as Figure 4 As shown.

[0141] After obtaining the prediction results at the feature map size, it is necessary to convert them to prediction results at the original image size. The original image needs to be transformed to a specific size before it can be used as input to the network model for computation. Therefore, an image size transformation bias needs to be added to reduce the error. Let this bias be η, and the original image size (image_shape) be:

[0142] image_shape = W image ×H image

[0143] Minimum feature map size s hape is:

[0144] feature_shape = W feature ×H feature

[0145] Assuming the input image is downsampled by a factor of 32 to obtain the minimum feature map, then the input image size input_shape is:

[0146] input_shape = 32 × W feature ×32×H feature

[0147] Then η is calculated as follows:

[0148]

[0149] in Indicates that in this tensor The minimum value in the scale factor is used because images in a batch may not be of the same size before being transformed to a fixed size; that is, the original images in the same batch may have different image shapes. A size factor β is set to map the location information to the original image. The scale factor is calculated as follows:

[0150]

[0151] The prediction frame positioning information in the original drawing is obtained as follows:

[0152] x = (b x - η) * beta

[0153] y = (b y - eta) * beta

[0154] W = b W * beta

[0155] H = b H * beta

[0156] In the input real sample, the category information is represented by an N classes dimensional vector, the category is numbered, and the category to which the real boundary frame belongs is set to 1 in the corresponding vector, and the other positions in the vector are 0. The confidence score and category information predicted by the network only need to be normalized by the sigmoid function, and the confidence score vector is multiplied by the predicted category vector before output, and the predicted confidence score that does not belong to the category to which the target belongs is set to 0.

[0157] In target detection, the features extracted by the low-dimensional neural network model contain rich positioning information, and the features extracted by the high-dimensional neural network model contain rich semantic information. In order to better complete the target detection function, a deep neural network model needs to be built. The residual structure can well alleviate the problems of gradient disappearance and network degradation, but there is still a certain degree of data flow loss. A new backbone DBNet is designed to replace Darknet53. The basic module of DBNet is DBL (Dense Block Layer), which includes a convolution layer, a BN (Batch Normalization) layer and a Mish activation layer. The specific implementation steps of using the DBNet algorithm are as follows:

[0158] 1. The Mish activation function ensures the smoothness at each point, ensures the existence of the gradient at each point, and can also be not completely truncated when the input is negative, allowing small negative gradient flow, and avoiding the problem of gradient saturation. The Mish activation function formula is as follows:

[0159] Mish(x) = x * tanh(ln(1 + ex))

[0160] 2. In order to accelerate the convergence of the neural network model, the BN layer is added, and the specific derivation formula is as follows:

[0161] Let the input of the i-th layer be x (l) , and the output of a certain neuron be y (l) , that is:

[0162] y (l) =f(x (l) (l-1) +b))

[0163] where f(·) is the activation function, w, b are learnable parameters during training. To improve the efficiency of optimization, the net input x (l) is assumed to be normally distributed. The present invention normalizes the input to the standard normal distribution, and the normalization layer is placed after the affine transformation and before the activation function. That is:

[0164]

[0165] where E(x (l) ) and var(x (l) ) are the expectation and variance of each dimension of x (l) over the entire training set under the current parameters. The main optimization algorithm used in the present invention is the stochastic gradient descent algorithm based on small batches, and the mean and variance of the current small batch of samples are usually used instead of the expectation and variance. Given a small batch of samples, which contains K samples, let (x (1，l) ,..., x (K，l) ) be the mean and variance of the net input of the l-th layer neurons:

[0166]

[0167]

[0168] To ensure the nonlinearity of the network after normalization, the value range is changed by adding scaling and translation changes, that is:

[0169]

[0170] The above formula can be expressed as BN γ，β (x (l) ), where γ, β represent the parameter vectors of scaling and translation respectively, and the BN layer can be regarded as a special neural layer, which is added before the nonlinear mapping function and after the convolution layer operation layer, that is:

[0171] y (l) =f(BN γ，β (x (l) ))=f(BN γ，β (wy (l-1) ))

[0172] Because the BN layer itself has a translation change, the affine transformation wy (l-1) ​The bias parameter is not needed. In addition, the normalization operation is performed layer by layer, which not only improves the optimization efficiency, but also serves as a regularization method. During training, the prediction of the neural network for a sample is related not only to the sample itself, but also to other samples in the same batch. Since the selection of the batch is random, the network model will not be over-fitted to a specific sample, thereby improving the generalization ability of the network.

[0173] 3. The sub-modules constituting the DB-Net of the application are DB (Dense Block), and in the sub-modules, the input of each layer is the output of all layers above the layer, that is, all shallow features in the module can be directly input into the subsequent sub-module, so that the effective feature data in the module can be reused. In order to control the amount of calculation, the number of feature maps output by the convolution layer in the original network structure DBL module is greatly reduced. The specific manifestation of gradient disappearance is that the input information and gradient information disappear after multi-layer transmission, but in the DB module, the output of the previous mode is directly used as the output information of the current module after sampling, and the gradient of the current module can directly reach the previous module during direction propagation, while the problems of gradient disappearance and network degradation can be effectively avoided.

[0174] Because the DBL layer needs to be replaced in the original Darknet-53 network, the output dimension of the module needs to be kept unchanged. The number of feature maps output by the Lth layer of the original neural network Darknet-53 network is M L The number of feature maps output by a DBL module in a DB module with the same number of feature map outputs is L l Suppose there are c DBL modules in the DB module, that is:

[0175]

[0176] Therefore, compared with the connection structure of the residual structure and the DB module for building a network model with the same number of layers, the network structure built by the DB module will far exceed the residual structure in operation speed. After the DB module outputs the feature map quantity through the Concat splicing, the number of feature maps output by the corresponding module of Darknet-53 is the same, so the model expression ability will not be worse due to the reduction of the calculation amount. In addition, in order to further improve the operation speed of the model and reduce the parameter amount, a DBL module with a convolution kernel size of 1x1 is added before each convolution layer with a convolution kernel size of 3x3, and the output feature map channel number of the convolution layer in the added DBL module is which not only plays a role in data dimension reduction, but also can fuse the features of each channel.

[0177] 4、Network model optimization is a common method of using neural network to optimize various indicators. In order to maximize the characteristic expression ability of the model, reduce the amount of model parameters and improve the operation efficiency, the application replaces the ordinary convolution operation with depth separable convolution. The depth separable convolution divides the ordinary convolution process into two parts: depthwise convolution and pointwise convolution.

[0178] The depthwise convolution process is that one convolution kernel is responsible for the convolution operation of one channel, that is, the number of convolution kernels is the same as the number of channels of the input feature map, assuming that the input picture dimension is WxHxC, and the convolution kernel size is w x h, then the parameter amount of the depthwise convolution process is:

[0179] N Depthwise =w x h x C

[0180] The number of feature maps after the depthwise convolution is consistent with the number of input feature maps, and cannot be expanded, and the feature maps in different channels are operated by different convolution kernels, and the spatial position information between the feature maps is not effectively utilized, so the feature maps in different channels are fused by pointwise convolution. The convolution kernel size of the pointwise convolution is 1x1xC, and the convolution operation of this process will weight and combine the feature maps output by the depthwise convolution process in the depth direction, and the number of convolution kernels set in the operation layer determines the number of feature map channels output by the layer, and it is assumed that there are k convolution kernels in the layer. The parameter amount calculation process of the pointwise convolution process can be represented as:

[0181] N Pointwise =1 x 1 x C x K

[0182] The total parameter amount of the depth separable convolution process is N all =N Depthwise +N Pointwise =C(q+K), if the ordinary convolution kernel size is p x q, the parameter amount of the convolution process is:

[0183] N conv =K x p x q x C

[0184] When the dimension of the input picture and the convolution kernel size are both greater than 1, it is easy to prove that the parameter amount of the depth separable convolution is less than that of the ordinary convolution.

[0185] 5、The predictor of the present application acquires positioning information by splicing the feature map of a certain shallow layer in the feature extraction network, and obtains three different size feature maps through upsampling operation, thereby obtaining three different size prediction results, which can effectively improve the detection capability of the size target. The predictor of the present application is built by using the building method of SPP-Net and PANet. The spatial pyramid pooling in the SPP-Net used by the present application realizes the receptive field mechanism. Because the kernel size needs to be increased, the use of convolution layer will greatly slow down the detection efficiency. The maximum pooling operation is the most suitable choice to realize this function. The maximum pooling layer with three kernel sizes of 5*5, 9*9 and 13*13 is used to traverse the feature map to extract new features. However, the maximum pooling layer will lose important information of the feature map. In order to ensure the integrity of the feature map, the feature map inputted into the receptive field mechanism is spliced.

[0186] The addition of the receptive field module makes the network more sensitive to targets of different sizes. The upsampling part of the predictor is similar to FPN, and the corresponding size feature map of the feature extraction network is spliced to obtain positioning information. Compared with the traditional yolo-v3 predictor, the modified predictor deepens the layer, and in order to prevent overfitting, 1*1 convolution layer is used to realize channel compression and expansion flexibly, and the output features of the receptive field part are used to complete the prediction in the output of the three scales.

[0187] The DB-Net network structure built by the present application, wherein the Sampling is a sampling layer composed of DBL modules with a convolution kernel size of 3*3 and a convolution step size of 2*2, which is used to reduce the size of the feature map and further provide feature information. It is found that when the ordinary convolution is changed to the separated convolution, the loss value will be NAN during the network training process. In order to ensure the effect of the gradient back propagation, the module residual structure is introduced in the present application, which further integrates deep and shallow features (high and low dimensional features), and to some extent, the idea of grouped convolution is used. A 1*1 convolution is added in the branch of the module residual structure to expand the feature map dimension of the previous module, which can ensure the smooth operation of the module residual operation. The Tiny version of the network model is built, that is, the model structure is simplified, in order to abandon a certain accuracy and obtain higher operation efficiency. Through multiple Add operations, the feature map of the shallow module can be inputted into the predictor, which is beneficial to the positioning of the target by the predictor. After the ordinary convolution is changed to the depth separable convolution, the training is prone to gradient anomaly, but the module residual structure added in the present application can effectively improve the problem.

[0188] 6、The present application selects yolov3 as the target detection basic algorithm of the present application, designs the loss function according to the yolov3 basic algorithm, and designs the loss function of DB-Net.

[0189] The loss function in yolov3 has four parts, respectively, the positioning coordinate loss, the bounding box loss, the confidence score loss and the category loss.

[0190] The yolov3 positioning coordinate loss is calculated as follows:

[0191]

[0192] Where λ is the positioning loss weight, S 2 The size of the feature map corresponding to the size is SxS, B is the maximum number of predicted boxes output by the network model, i represents the prediction information output by the i-th network, and j represents the j-th prediction data in all output B prediction boxes. The loss function prototypes of different parts are different, so the loss values of different parts are different. In order to avoid the influence of the overall back propagation on the weight update process due to the too large or too small loss value of a part, the calculation results of different loss functions need to be balanced. The determination coefficient is different for different prediction boxes, which is generated by different feature points. The determination coefficient is used to determine whether the feature point responsible for predicting the target generates the prediction box. If yes, the determination coefficient is 1, and the loss value is calculated; if not, the coefficient is 0, and the loss value is not calculated.(2-p W ×p H The scale coefficient is 2, p W ×p H The width and height data of the anchor normalized bounding box information are the most optimal matching with the real label information. The prototype of the coordinate loss function is square loss, (x i , y i ) is the target center point coordinate of the real label mapped to the feature map size, That is, the target center point coordinate.

[0193] The yolov3 bounding box height loss is calculated as follows:

[0194]

[0195] The function prototype of the width and height loss is square root loss. The reason is that in yolov3, nine groups of prior anchors are obtained by using k-means clustering algorithm, and the anchor value is close to the width and height of the real label, which is not suitable for large adjustment. Using square loss will cause the network to be difficult to converge due to the positioning error.

[0196] The yolov3 confidence score loss has two parts, background loss and foreground loss:

[0197]

[0198] The above formula is the background loss, i.e. the negative sample region loss, λ noobj is the weight when there is no positive sample in the prediction box, because the area of the negative sample region (no target region) is much larger than the area of the positive sample region (region where the target exists) in an image, and there is an imbalance between positive and negative samples. If only the positive sample region is calculated for loss, it is difficult to recall the missed targets in the subsequent training process once the first training prediction misses, so it is necessary to calculate the loss of the negative sample region, and the weight exists to balance the imbalance between the negative sample region and the positive sample. Wherein, is the determination coefficient, which is 1 if the region is a negative sample, otherwise 0. The following formula is the foreground loss, and the calculation parameters are similar to the background loss calculation.

[0199]

[0200] The yolov3 class loss is calculated as follows:

[0201]

[0202] Wherein, λ class is the weight of the class loss part, is the predicted target class, p i (c) is the class to which the real target belongs.

[0203] The yolov3 loss function is integrated as follows:

[0204] L yolov3 = L x，y + L w，H + L conf1 + L conf2 + L cls

[0205] In yolov3, the positioning loss is divided into coordinate loss and width-height loss, which are calculated separately. However, the coordinates and height of the actual target are both descriptions of the target positioning, and the present application believes that they should be optimized in a unified manner. IoU is used as the basis of the loss function, and the loss function is optimized on this basis. The present application proposes CIoU loss and improves the confidence score loss and class loss.

[0206] The DB-Net positioning loss function (CIoU) formula is as follows:

[0207]

[0208] Where, p1, p2 respectively represent the center points of the predicted frame and the real frame, and represents the Euclidean distance between the two center points. L is the diagonal length of the minimum closed region of the two bounding boxes. Alpha v is the penalty term coefficient, which has the effect of accelerating the network convergence. Alpha is the weight function, and v is the similarity of the aspect ratio. The formulas of alpha and v are as follows:

[0209]

[0210] Where W gl , H gl are the width and height of the real bounding box in the fixed position input picture, and W, H are the width and height of the target bounding box predicted by the algorithm relative to the input picture.

[0211] The DB-Net confidence score loss and the category loss are as follows:

[0212]

[0213] Where the loss function in the above formula introduces a balance factor lambda to balance the proportion of positive and negative samples themselves. In the experiments of the application, lambda is selected as 0.25, that is, the proportion of positive samples is less than that of negative samples. However, only adding the balance factor can only solve the problem of imbalance between positive and negative samples, and cannot solve the problem of simple and difficult samples, so the parameter gamma is added to adjust the rate of weight reduction of simple samples. When gamma is 0, the loss function is the cross entropy loss function, and when gamma increases, the influence of the adjustment factor also increases. In the experiments of the application, gamma is selected as 2.

[0214] The category loss of the application is improved on the basis of yolo-v3 to obtain the final loss function, and the formula is as follows:

[0215] L myloss =L CIOU +L conf +L yolov3_cls

[0216] Step three, binocular camera matching algorithm.

[0217] 1. The binocular camera matching algorithm can be used for depth calculation according to the same label object identified in step three. The following steps are divided:

[0218] First step, coarse matching: set the area graph identified in the left graph as P l , and set the area detected in the right graph as P r . Calculate the distance between P l and P r . The application sets epsilon as 2.6, and the distance less than the threshold epsilon enters the second step. The distance formula is as follows:

[0219]

[0220] The same area meeting the first step requirement collected by left and right cameras is subjected to ORB feature extraction, and the matched points are subjected to depth distance calculation using a binocular mathematical model for subsequent robot grabbing. The subsequent part is ORB feature extraction.

[0221] In the second step, a scale space is constructed. The ORB algorithm is subjected to scale change problem, and an image pyramid is constructed by 0.5 times down-sampling, and then feature points are extracted from each layer.

[0222] In the third step, feature points are extracted. The gray scale between a certain pixel and the pixels on the circle with a fixed radius around the certain pixel are compared. The gray scale of the center pixel is denoted as I p , the gray scale of the pixel on the circle with a radius of 3 and a pixel number of 16 is denoted as I p→x , and ΔI is a set gray scale threshold. The classification S p→x of the surrounding pixel points is as follows:

[0223]

[0224] If the number of the pixel points with continuous classification of d (darker) or b (brighter) among the 16 surrounding points is greater than N (N is generally 12), the point is considered as a possible feature point. In order to speed up, in the actual feature extraction process of detection, the pixels at 1, 5, 9 and 13 of the 16 surrounding points can be selected for comparison first, and if the number of the pixels belonging to d or b is greater than or equal to 3, the 16-point detection method is used for further verification of these pixels.

[0225] Using the above method, a large number of possible feature points can be extracted in the detection area, but the detection order and the distribution of the feature points around the image edge may reduce the efficiency of image feature extraction, therefore, in the ORB algorithm, a decision tree trained by ID3 algorithm is usually used to screen the optimal points. In addition, the feature points may be mutually adhered, and the non-maximum suppression is used for further screening of the pixel points. The center coordinate gray scale value is denoted as p, the pixel gray scale values of the surrounding continuous N pixels greater than or less than the threshold t are denoted as v i , and the evaluation index function is as follows:

[0226]

[0227] When two possible feature points P and Q are connected, the evaluation index function values V of these pixel points are compared, and the point with a smaller V value is discarded, and thus the screening of the feature points is completed.

[0228] Fourthly, constructing the feature descriptor. After getting the position of the feature points, we need to describe the feature points. ORB uses BRIEF feature descriptor. This feature description method only needs to compare the gray value of 256 groups of points around the feature points.

[0229] Because BRIEF feature descriptor is sensitive to noise, first use the dead filter to smooth the image. Then take points in the SxS size field around a feature point p according to the Gaussian distribution of (X, Y) .

[0230] Binary test is performed on the n points (x, y) taken from the SxS size field of the feature point p:

[0231]

[0232] Where I(x), I(y) are the gray values of a group of points in the feature point field. Then the BRIEF feature descriptor of a feature point can be expressed by the following formula:

[0233]

[0234] On the basis of BRIEF algorithm, ORB algorithm uses integral image to further enhance the anti-noise ability of the feature descriptor; and in the 31x31 field, take 5x5 sub-window with random points as the key, and compare and encode the sum of 25 pixel points.

[0235] Let the point pair S selected by the BRIEF feature descriptor be:

[0236]

[0237] At this time, the BRIEF feature descriptor formula is as follows:

[0238] g n (p, θ) := f n (p) | (x i , y i ) ∈ S

[0239] Fifthly, the generated guide BRIEF feature descriptor is a one-dimensional vector containing 01, so the Hamming distance of the feature descriptor can be used to reflect the similarity between two feature descriptors. Hamming distance is commonly used in data transmission error control coding, which represents the number of corresponding bit differences between two same length words. XOR operation is performed on two strings, and the number of results with value 1 is counted, which is the Hamming distance. Compare all the Hamming distances of the selected regions, and the smallest distance is the matching point.

[0240] 2、Binocular stereo vision technology as a branch of computer vision, binocular camera mathematical model as shown in Figure Figure 5 , with low cost, fast, high precision advantages. The present application uses binocular stereo vision technology in order to quickly locate the object distance in the scene. According to the binocular camera mathematical model, by similar triangle theory, the depth information is calculated:

[0241]

[0242]

[0243]

[0244] Where X R is the horizontal offset obtained by the left camera to collect the image, XT is the horizontal offset obtained by the right camera to collect the image, d is the camera offset, B is the camera baseline distance, f is the camera focal length, Z is the depth information.

[0245] Although embodiments of the present application have been disclosed as above, it is not limited to the use listed in the specification and embodiments. It can be fully applied to various fields suitable for the present application. For those skilled in the art, additional modifications can be easily realized. Therefore, the present application is not limited to specific details and figures shown and described herein without departing from the general concept defined by the claims and the equivalent scope.

Claims

1. An image processing and neural network fusion-based water organism recognition and matching method, characterized in that, Comprising the following steps: Step one, image preprocessing module, its function is, in order to solve the distortion problem of underwater image, increase the contrast of image, and improve the subsequent classification and recognition accuracy, will adopt the camera to collect the original image of seabed , through the red light compensation module, white balance module, and contrast enhancement module, output ; Step two, water organism identification module, the role is to reduce the processing speed and improve the accuracy of various water organism identification, using DBNet neural network, self-made data set for water organism classification task, DBL minimum unit and DB minimum module based on yolov3 network are proposed for DBNet neural network, and the loss function is improved; The implementation process of the water organism identification module is as follows: (1) The basic module of DBNet is , including a convolution layer, a layer and an activation layer; (2) The sub-modules constituting the DBNet are In the sub-modules, the input of each layer is the output of all layers above the layer, that is, all shallow features in the module can be directly input into the subsequent sub-modules, so that the effective feature data in the module can be reused, and in order to control the amount of calculation, the number of feature maps output by the convolutional layer in the module is greatly reduced ​ (3) DBNet positioning loss function The formula is as follows: ； wherein, respectively represent the center points of the predicted and real bounding boxes, denotes the Euclidean distance between the two center points, is the diagonal length of the minimum closed region of the two bounding boxes, is a penalty term coefficient, which has the effect of accelerating the network convergence, is a weight function, is a measure of the similarity of the aspect ratio; Step three, underwater binocular image matching algorithm, the role is to reduce the processing amount, speed up the overall matching speed and improve the overall matching accuracy, using the recognition frame constraint ORB matching algorithm, the overall matching area is limited to the recognition frame area of the left and right image threshold output; And for the threshold recognition frame, ORB feature matching is carried out; The main implementation process of the underwater binocular image matching algorithm is as follows: (1) Set the left figure identified region map as , the right figure detected region as , calculate the distance between and , set as 2.6, the distance less than the threshold , if it is consistent with the second step, the distance formula is as follows: ； (2) Feature point extraction, comparing the gray scale between a certain pixel and the pixels on the circle with a fixed radius around it, setting the gray scale of the center pixel as , the gray scale of the pixel on the circumference of the circle with a radius of 3 and a pixel number of 16 as , , and the gray scale threshold set as , the classification of the surrounding pixel points is as follows: If the number of pixels classified as or in the surrounding 16 points is greater than , Generally, 12 is taken, and the point is considered as a possible feature point. In order to speed up, in the actual feature extraction process of detection, first, only the pixels at 1, 5, 9, and 13 of the surrounding 16 points are compared, and if the number of pixels belonging to or is greater than or equal to 3, the 16-point detection method is further verified for these pixels.

2. The water organism recognition and matching method based on image processing and neural network fusion according to claim 1, characterized in that, The implementation process of the image preprocessing module in step one is as follows: (1) Red light compensation module, according to the underwater environment, only the red channel needs to be processed, and the original input image is , is the red, green and blue color channel set of the original image, represent the red, green and blue channel size a pixel point of the image, wherein and are the horizontal index and vertical index of the image pixel respectively, and the numerical value is limited in the range of by normalization{ }to calculate the green channel mean value and red light compensation; (2) white balance module, white balance improved image ; (3) Contrast enhancement module, for underwater images, contrast enhancement is the process of separating objects, using gamma correction and sharpening to improve image contrast.

3. The water organism recognition and matching method based on image processing and neural network fusion according to claim 2, characterized in that, The green channel mean value is calculated, and the calculation formula is as follows: ； Then the red light compensation formula is as follows: ； wherein is a constant value, is the red light channel value of the input picture after red light compensation, instead of the original red light channel, then the improved red light channel image is .

4. The water organism recognition and matching method based on image processing and neural network fusion according to claim 1, characterized in that, In step (3): The formula is as follows: ， ； wherein respectively the width and height of the real bounding box at the fixed position input picture, are the width and height of the target bounding box predicted by the algorithm relative to the input picture.

5. The water organism recognition and matching method based on image processing and neural network fusion according to claim 1 or 4, characterized in that, DBNet confidence score loss and class loss are as follows: ； wherein the loss function of the above formula introduces a balance factor , which is used to balance the imbalance of the proportion of positive and negative samples. In the experiment, the value of is 0.25, that is, the proportion of positive samples is less than that of negative samples, but only adding the balance factor can only solve the imbalance problem of positive and negative samples, and cannot solve the problem of simple and difficult samples. Therefore, the parameter is added to adjust the rate of weight reduction of simple samples. When is 0, the loss function is the cross-entropy loss function, and when increases, the influence of the adjustment factor also increases; in the experiment, the value of is 2; The class loss is improved based on yolov3 to obtain the final loss function, and the formula is as follows: 。 6. The water organism recognition and matching method based on image processing and neural network fusion according to claim 1, characterized in that, Using the above method, When extracting a large number of potential feature points in the region to be detected, to overcome the fact that the detection order and the distribution of feature points around the image edges may reduce the efficiency of image feature extraction, In the algorithm, using The algorithm trains a decision tree to select the optimal point. To avoid the possibility of feature points sticking together, non-maximum suppression is used to further filter the pixels. Let the center coordinate gray value be... , surrounding continuous Pixels greater than or less than the threshold The pixel grayscale value is determined by The evaluation index function is as follows: ； When two possible feature points When connected, compare the evaluation index function values ​​of these pixels. Discarding one of them Points with relatively small values ​​are selected, thus completing the filtering of feature points.

Images