A ship IMO detection method based on wavelet decomposition and residual neural network

Abstract

The application discloses a kind of based on wavelet decomposition and residual neural network's detection method of ship IMO, mainly include as follows: step 1) utilize wavelet technology to the input ship IMO image is decomposed by wavelet and forms wavelet decomposition module, obtains output feature map;Step 2) by the feature map obtained in step 1) is convolved, while with the same size feature map that residual neural network module executes in output is weighted average etc. Process obtains the final feature map output;By the feature map obtained in step 2) is deconvolved, convolution, batch standardization and activation operation separates three features for detecting IMO feature map, and the IMO of ship is detected by the feature information of three feature maps, completes the detection identification of input image ship IMO.Combining the data collected under actual environment carries out experiment, and the experimental results show that the application has improvement on average accuracy, precision, recall, F1-Score, and is very suitable for unmanned aerial vehicle or unmanned ship automatic patrol ship monitoring.

Description

Technical Field

[0001] This invention relates to the field of target detection, and in particular to a method for detecting and identifying ship IMOs based on wavelet decomposition and residual neural networks. Background Technology

[0002] Ship name codes are uniformly registered using the International Maritime Organization (IMO), and the Lloyd's Register of Registers is the corresponding IMO number. The registration code structure serves as the main code structure for ship names, and is categorized based on the registration status of Chinese container ships with the IMO.

[0003] Object detection is a crucial research area in computer vision, and it forms the foundation for solving higher-level visual tasks such as image segmentation, scene understanding, object tracking, image captioning, and event detection. Object detection has wide applications in many fields of artificial intelligence and information technology, including robot vision, consumer electronics, security, autonomous driving, human-computer interaction, content-based image retrieval, intelligent video surveillance, and augmented reality. Current object detection primarily focuses on objects such as people, bicycles, cats, and dogs, with very little research on object detection for ship IMOs (Intrusions at Ports). However, ship IMO detection is a vital foundation for automated patrol and monitoring of designated vessels in harbors by drones or unmanned surface vessels (USVs). Unlike the detection of objects with clearly defined boundaries, such as people, vehicles, cats, dogs, and license plates, ship IMO detection lacks clear boundaries, and the background color and viewing angle vary significantly, greatly increasing the difficulty of detection. The original images contain feature information of different frequencies, and there is information interference between these different frequencies. This makes it easy for existing neural network models to overlook some important feature information during the extraction process, resulting in low recognition rates. Current state-of-the-art (SOTA) technologies have low accuracy in detecting and recognizing ship IMOs, with an average accuracy of only 49.84% (which may vary slightly depending on the experimental platform), a precision of 80.10%, a recall of 39.65%, and an F1-Score of 59%. There is still significant room for improvement in current technologies for ship IMO detection. This also prevents port entry and exit management departments from automatically conducting patrols and monitoring of vessels in port areas using drones.

[0004] In the paper "Mallat SGA theory for multiresolution signal decomposition: the wavelet representation[J].IEEE Transactions on Pattern Analysis & Machine Intelligence, 1989, 11(4)", Mallat proposed an engineering practice of using wavelet analysis technology (wavelet transform) for image decomposition and synthesis. Wavelet transform is an optimization and improvement of Fourier transform. Unlike Fourier transform, which considers the spectral characteristics of the signal from an overall perspective, wavelet transform analyzes the spectral characteristics of the signal from a shorter time or space. Therefore, wavelet transform is essentially a time-frequency or space-frequency analysis method. In the paper "Lin TY, Goyal P, Girshick R, et al. Focal Loss for Dense Object Detection[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2017,PP(99):2999-3007.", Lin et al. proposed an improved focal loss based on the cross-entropy of the loss function to improve the recognition rate of samples that are difficult to classify. Two hyperparameters α and β are set in the loss function. By adjusting the two hyperparameters, the weight of easy-to-classify samples is reduced, so that the deep learning model can focus on some samples that are difficult to classify, thereby improving the model's classification ability. In the paper "Zhou X, Wang D, P Krhenbühl. Objects as Points[J].2019," Zhou et al. proposed an anchor-free object detection method, which has higher detection accuracy compared to similar methods. However, this method is mainly used for multi-object detection, and the neural network is very large, which is not conducive to engineering applications for single-object detection. At the same time, like previous methods, this method does not fully analyze the features of the input image information. Based on this, this invention introduces a ship IMO detection method based on wavelet decomposition and residual neural network, and the resulting ship recognition method. Summary of the Invention

[0005] To achieve the above objectives, the technical solution provided by this invention is as follows:

[0006] A method for detecting ship IMOs based on wavelet decomposition and residual neural networks, characterized by comprising the following steps:

[0007] 1) Wavelet decomposition is performed on the input ship IMO image using wavelet technology to form a wavelet decomposition module, obtaining the output feature maps C0, ..., C10. j-1 ;

[0008] 2) By analyzing the feature maps C0, ..., C obtained in step 1), j-1 Each layer in the process undergoes convolution, weighted averaging, and other processing to obtain the final feature map output O0; at the same time, a residual neural network module is formed.

[0009] 3) The input is completed by performing deconvolution, convolution, batch normalization, and activation operations on the O0 obtained in step 2).

[0010] Detection and identification of IMOs on ships in images.

[0011] Step 1) uses the following formula to obtain the output feature map C0, ..., C10. j-1 :

[0012]

[0013] In the formula, L 2 (R) denotes the set of all square-integrable functions over the real number field, {{V j ;j∈Z};φ(x)} is L 2 An orthogonal multiresolution analysis on (R), {V j ;j∈Z} is L 2 A set of closed subspaces on (R), where φ(x) is a sequence of closed subspaces on L. 2 A function in R that satisfies orthogonal multiresolution analysis. It is space V j Orthonormal basis, W j For V j+1 Chinese V j orthogonal complement, It is space W j An orthonormal basis, where <·, ·> denotes the inner product of two functions over their domain, H j (ω) and Γ j (ω) represents the low-pass filter LPF used in wavelet decomposition of the signal. j and high-pass filter HPF j The frequency response, where j is a positive integer representing the current level of the wavelet decomposition. and For H j (ω) and Γ j The coefficient sequence of (ω), and They represent and The conjugate complex number of H, where δ represents the unit impulse function. j and Γj There are two coefficient matrices, each with a size of N / 2 × N, where N represents the width and height of the input image signal. In this invention, the width and height of the input image are equal. and H represents j and Γ j The conjugate transpose of the matrix, cA j Indicates the input image signal, cA j-1 , cH j-1 , cV j-1 , cD j-1 They represent cA respectively j The wavelet decomposition results in scale components, horizontal wavelet components, vertical wavelet components, and diagonal wavelet components. The `cat` function represents the concatenation (or chaining) of the matrix. (C) j-1 This represents the output after matrix concatenation.

[0014] Step 2) involves the wavelet decomposition of the image signal into C0, ..., C10. j-1 Processing, for C j-1 Perform convolution, max pooling, batch normalization, and activation operations to obtain a feature map with the same size as C. j-2 The same output O but different depths j-1 Then for C j-2 Perform a convolution operation to make its depth the same as O. j-1 Same, then according to O j-1 and C j-2 The two outputs are weighted and averaged based on their proportion in the original image signal. The above steps are repeated until the output is O0.

[0015] The mathematical expression used in step 2) is as follows:

[0016] (N in ×N in ×M in )*(k×k×M out ) s，p =(N out ×N out ×M out (7)

[0017] N out =floor[(N in -k+2×p) / s+1] (8)

[0018]

[0019]

[0020] In the formula, N inM represents the width and height of the current input feature map (hidden layer) in the neural network. in The input feature map depth is represented by *, the convolution operation by *, the kernel size by k, the stride of the kernel through the input feature map by s, and the padding size by p. out This represents the width and height of the output feature map in the neural network, with the function floor representing rounding down. M out x represents the depth of the output feature map. i This represents the value of a neuron in a feature map of the i-th training sample, where m represents the number of training samples, and the statistic is... and S 2 These are point estimates of the mean and variance of the neuron's values ​​across m training samples, respectively, where ∈ is a positive constant to avoid a zero denominator. For x i Standardized normalized output for The corrected output is given by γ and β, where γ and β are the hyperparameters used in the correction, and ReLU is the activation function.

[0021] Step 3) includes the following:

[0022] The output of step 2) is used for upsampling to obtain a feature map with width and height of N. in / 2, its N in This represents the width, height, and depth of the original image, M. out and not less than 16M in M in The output represents the depth of the original image. From this output, three parallel feature map input lines are drawn. The first line obtains an output of size N through operations such as convolution, batch normalization, and activation. in / 2×N in A 2×1 feature map is used, where each neuron determines whether the image location corresponding to that neuron contains a ship IMO. The second line obtains an output of size N through convolution and other operations. in / 2×N in The two feature maps, each 2×2, are used to locate the center point of the original image containing the ship's IMO. The third line uses convolution and other operations to obtain an output of size N. in / 2×N in The two-layer feature map (2×2) is used to determine the width and height of the ship's IMO in the original image. Finally, the information from the three routes is combined to locate the ship's IMO in the original image.

[0023] The mathematical expression used in step 3) is as follows:

[0024]

[0025]

[0026] In the formula, The width and height of the input feature map are used for deconvolution. The width and height of the output feature map during deconvolution are given, ° represents the deconvolution operation, sigmoid is the activation function, and the definitions of other parameters are the same as in step 2).

[0027] Step 1) obtains the output feature map C0, ..., C10. j-1 Specifically, it includes the following steps:

[0028] 1.1) According to formula (1), we can obtain and V j The orthonormal basis of each orthogonal subspace after decomposition;

[0029] 1.2) According to formula (2), V can be obtained j Decompose into V j-1 and W j-1 The frequency responses of the filters used are H j (ω) and Γ j (ω), and based on the frequency components of the two filter outputs, H can be... j (ω) and Γ j (ω) is denoted as the low-pass filter LPF. j and high-pass filter HPF j The frequency response is obtained, and the coefficient sequences of the two frequency responses are obtained simultaneously. and

[0030] 1.3) According to formula (3), the coefficient sequence can be obtained. and Constraint relationships between them;

[0031] 1.4) According to formula (4), two coefficient matrices H can be constructed. j and Γ j The size of the matrix is ​​N / 2×N, where N represents the width and height of the input image signal. In this invention, the width and height of the input image are equal.

[0032] 1.5) According to formula (5), for the input image signal cA j By decomposing the matrix, the coefficient matrix H can be easily obtained. j and Γ j conjugate transpose and Finally, cA can be obtained through matrix multiplication. j The scale component, horizontal wavelet component, vertical wavelet component, and diagonal wavelet component are denoted as cA, respectively.j-1 , cH j-1 , cV j-1 , cD j-1 ;

[0033] 1.6) According to formula (6), when j>1, for cH j-1 , cV j-1 , cD j-1 Perform matrix concatenation and output C j-1 When j=1, perform matrix concatenation on cA0, cH0, cV0, and cD0 and output C0, finally outputting feature maps C0, ..., C j-1 .

[0034] In step 2), by processing the feature maps C0, ..., C j-1 Each layer in the algorithm undergoes convolution, weighted averaging, and other processing to obtain the final feature map output O0; this includes the following steps:

[0035] 2.1) According to formulas (7) and (8), the input feature map C is... j-1 Perform convolution or max pooling operations. The number of convolutions can be customized according to the actual situation, but the width and height of the final output feature map are the same as C. j-2 Similarly, the depth can be customized according to the actual situation, but it cannot be less than C. j-2 The depth is calculated, and the final output feature map is denoted as O. j-1 ;

[0036] 2.2) According to formulas (9), (10), and (11), perform batch normalization and activation operations (the default is ReLU activation operation) on each convolution result in 2.1);

[0037] 2.3) According to formulas (7)(8)(9)(10)(11), for C j-2 Perform convolution, batch normalization, and activation operations; the output feature map size is the same as O. j-1 Same, then according to C j-1 and C j-2 The proportion of O in the original input image j-1 and C after the operation j-2 Perform a weighted average;

[0038] 2.4) Repeat steps 2.1), 2.2), and 2.3) until the output is O0.

[0039] In step 3), the detection and recognition of ship IMOs in the input image are completed by performing deconvolution, convolution, batch normalization, and activation operations on O0; including the following steps:

[0040] 3.1) According to formulas (9)(10)(11)(12)(13), perform multiple deconvolution operations on the input O0. The number of deconvolutions (denoted as k) can be customized according to the actual situation. After each deconvolution, batch normalization and activation operations are performed to ensure that the final output feature map (denoted as O0) is consistent with the actual situation. j+k-1 The width and height of the image are one-quarter of the original image's width and height, rounded down, and the depth is no less than 16 times the original image's depth.

[0041] 3.2) According to formulas (7)(8)(9)(10)(11)(14), for O j+k-1 Three parallel lines are introduced. The first line undergoes convolution, batch normalization, and activation operations. The number of convolutions is customized according to the actual situation to obtain a feature map with unchanged width and height and a depth of 1. Each neuron in this feature map is activated by sigmoid. Each activated neuron is then used to determine whether the image location corresponding to that neuron contains a ship IMO. Then, the other two lines undergo convolution, batch normalization, and activation operations. The number of convolutions is customized according to the actual situation to obtain feature maps with unchanged width and height and a depth of 2. These are used to predict the center point location and width and height of the ship IMO, respectively. Focal loss is used for neural network parameter learning on the first feature map, while L1 absolute loss or L2 squared loss function can be used for parameter learning on the other two feature maps.

[0042] A method for identifying ship registration numbers formed by the ship IMO detection method based on wavelet decomposition and residual neural network as described above.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] 1. The ship IMO detection method based on wavelet decomposition and residual neural network has an overlap rate of not less than 0.5. The average accuracy of this detection method is 56.22%, the precision is 83.33%, and the recall is 45.45%. Therefore, the measurement or identification results of this invention improve the average accuracy by 6.38%, the precision by 3.23%, the recall by 5.8%, and the F1-Score by 6%.

[0045] 2. By using the method of this invention to detect the IMO of ships, the ship entry and exit management department can automatically complete the patrol and monitoring of ships in the port area through drones and video surveillance, thereby improving the efficiency of police work. Attached Figure Description

[0046] Figure 1 This is a diagram illustrating the overall operational framework of the IMO target detection for ships according to the present invention.

[0047] Figure 2This is a flowchart for performing an image decomposition using wavelet analysis.

[0048] Figure 3 This is a diagram illustrating the implementation of the wavelet decomposition module for a specific input image in this invention.

[0049] Figure 4 This is a diagram illustrating the implementation of the residual neural network module for a specific input image in this invention.

[0050] Figure 5 This is a diagram illustrating the implementation of the target detection module for a specific input image in this invention. Detailed Implementation

[0051] The present invention will be further described below with reference to specific embodiments.

[0052] like Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 As shown in this embodiment, the method for detecting ship IMO based on wavelet decomposition and residual neural network is as follows:

[0053] 1) Wavelet technology is used to perform wavelet decomposition on the input ship IMO image or to construct a wavelet decomposition module.

[0054] The main technology used is wavelet decomposition of the input ship IMO image. The mathematical expression for the wavelet decomposition module is as follows:

[0055]

[0056] In the formula, L 2 (R) denotes the set of all square-integrable functions over the real number field, {{V j ;j∈Z};φ(x)} is L 2 An orthogonal multiresolution analysis on (R), {V j ;j∈Z} is L 2 A set of closed subspaces on (R), where φ(x) is a sequence of closed subspaces on L. 2 A function in R that satisfies orthogonal multiresolution analysis. It is space V j Orthonormal basis, W j For V j+1 Chinese V j orthogonal complement, It is space W j An orthonormal basis, where <·, ·> denotes the inner product of two functions over their domain, H j (ω) and Γ j (ω) represents the low-pass filter LPF used in wavelet decomposition of the signal. jand high-pass filter HPF j The frequency response, where j is a positive integer representing the current level of the wavelet decomposition. and For H j (ω) and Γ j The coefficient sequence of (ω), and They represent and The conjugate complex number of H, where δ represents the unit impulse function. j and Γ j There are two coefficient matrices, each with a size of N / 2 × N, where N represents the width and height of the input image signal. In this invention, the width and height of the input image are equal. and H represents j and Γ j The conjugate transpose of the matrix, cA j Indicates the input image signal, cA j-1 , cH j-1 , cV j-1 , cD j-1 They represent cA respectively j The wavelet decomposition results in scale components, horizontal wavelet components, vertical wavelet components, and diagonal wavelet components. The `cat` function represents the concatenation (or chaining) of the matrix. (C) j-1 This represents the output after matrix concatenation.

[0057] Assume cA j Given an image with width and height of N, the process of performing an image decomposition using wavelet analysis is as follows (see attached). Figure 2 ) is: cA j Each line with LPF j Perform convolution, then save the even-numbered columns of the convolution result, and output an image matrix of size N×N / 2, where each column of this image matrix is ​​concatenated with the LPF (Local Perspective). j Perform convolution, save the even-numbered rows of the convolution result, and output the scale component cA of size N / 2 × N / 2. j-1 Each column of this image matrix is ​​associated with HPF. j Perform convolution, save the even-numbered rows of the convolution result, and output the horizontal wavelet component cH of size N / 2×N / 2. j-1 ;cA j Each line with HPF j Perform convolution, then save the even-numbered columns of the convolution result and output an image matrix of size N×N / 2, where each column of this image matrix is ​​concatenated with the LPF. j Perform convolution, save the even-numbered rows of the convolution result, and output the vertical wavelet component cV of size N / 2×N / 2. j-1 Each column of this image matrix is ​​associated with HPF. jPerform convolution, save the even-numbered rows of the convolution result, and output a diagonal wavelet component cD of size N / 2×N / 2. j-1 If the input image size is N×N×3, then wavelet decomposition needs to be performed on each layer, and the results corresponding to each component need to be concatenated into a matrix to obtain the output of each component.

[0058] Assuming the input image has dimensions of 512×512×3 and wavelet decomposition level j=4, the specific implementation flow of the wavelet decomposition module of this invention for this image is as follows (see appendix). Figure 3 The process involves: obtaining four components cA3, cH3, cV3, and cD3 of size 256×256×3 through wavelet decomposition; performing matrix concatenation on components cH3, cV3, and cD3 to obtain a feature output of size 256×256×9; and performing wavelet decomposition on cA3 to obtain four components cA2, cH2, cV2, and cD2 of size 128×128×3; and performing matrix concatenation on components cH2, cV2, and cD2 to obtain a feature output of size 128×128×9. The feature output is ×9; wavelet decomposition of cA2 yields four components cA1, cH1, cV1, and cD1 of size 64×64×3, and matrix concatenation of components cH1, cV1, and cD1 yields a feature output of size 64×64×9; wavelet decomposition of cA1 yields four components cA0, cH0, cV0, and cD0 of size 32×32×3, and matrix concatenation of the four components yields a feature output of size 32×32×12;

[0059] 2) The wavelet decomposition module generates wavelet decomposition C0, ..., C0 from the image signal. j-1 Process or form a residual neural network module

[0060] The main method utilizes the wavelet decomposition module in step 1) to generate wavelet decomposition C0, ..., C0 from the image signal. j-1 Processing, for C j-1 Perform convolution, max pooling, batch normalization, and activation operations to obtain a feature map with the same size as C. j-2 The same output O but different depths j-1 Then for C j-2 Perform convolution and other operations to make its depth the same as O. j-1 Same, then according to O j-1 and C j-2 The two outputs are weighted and averaged based on their proportion in the original image signal. This process is repeated until the output is O0. The relevant mathematical expression for the residual neural network module is as follows:

[0061] (N in ×N in ×M in )*(k×k×M out) s，p =(N out ×N out ×M out (7)

[0062] N out =floor[(N in -k+2×p) / s+1] (8)

[0063]

[0064] In the formula, N in M represents the width and height of the current input feature map (hidden layer) in the neural network. in The input feature map depth is represented by *, the convolution operation by *, the kernel size by k, the stride of the kernel through the input feature map by s, and the padding size by p. out This represents the width and height of the output feature map in the neural network, with the function floor representing rounding down. M out x represents the depth of the output feature map. i This represents the value of a neuron in a feature map of the i-th training sample, where m represents the number of training samples, and the statistic is... and S 2 These are point estimates of the mean and variance of the neuron's values ​​across m training samples, respectively, where ∈ is a positive constant to avoid a zero denominator. For x i Standardized normalized output for The corrected output is given by γ and β, where γ and β are the hyperparameters used in the correction, and ReLU is the activation function.

[0065] To better describe the implementation process of the residual neural network module, three neural network sub-modules need to be defined (see appendix). Figure 4 The specific functions of the three sub-modules, Identity, Conv1, and Conv2, are as follows:

[0066] Identity Block: The input and output sizes of the feature map are the same. Assuming the input is X (size N×N×M), then the depth of X is M. First, a convolution operation is performed on X with a 1×1 kernel. The depth of the kernel is equal to the depth of the input feature map (the same applies below), and the output depth is M / 4. Therefore, the output feature size is N×N×M / 4. Batch normalization (BN) and activation function operations are performed on the output. Then, a convolution operation with a 3×3 kernel and an output depth of M / 4 is performed (denoted as 3×3, M / 4), along with batch normalization and activation operations (denoted as BN & ReLU). Finally, a (3×3, M / 4) convolution operation is performed, along with BN processing, to obtain the output F(X) (size N×N×M). The sum of F(X) + X is then output.

[0067] Conv1: The depth of the output feature map of the convolutional block is 4 times the depth of the input, while the width and height remain unchanged;

[0068] Conv2: The convolutional block performs convolution on a feature map with an input size of N×N×M, and the output result is N / 2×N / 2×2M.

[0069] Let C3, C2, C1, and C0 represent the... Figure 3 Given feature maps with decomposed dimensions of 256×256×9, 128×128×9, 64×64×9, and 32×32×12, the specific implementation flow of the residual neural network module of this invention for these feature maps is as follows (see appendix). Figure 4 )for:

[0070] (1) Perform a (3×3, 64) convolution operation and BN & ReLU operation on C3 to output a feature map with a size of 256×256×64. Then perform max pooling operation with a pooling kernel of 3×3, a stride of 2, and an output depth of 64 to obtain a feature map of 128×128×64. Perform BN operation, and then pass through a Conv1 convolution block and two Identity blocks to obtain a feature map of 128×128×256 (denoted as O3).

[0071] (2) Perform a (3×3, 256) convolution operation and BN&ReLU operation on C2 to output a feature map of size 128×128×256 and assign it to C2. It is easy to find that the proportions of O3 and C2 in the original signal are 3 / 4 and 3 / 16, respectively. Then, the feature map of size 128×128×256 is obtained by weighted averaging. After passing through the Conv2 convolution block and 3 Identity blocks, a feature map of 64×64×512 (denoted as O2) is obtained.

[0072] (3) Perform a (3×3, 512) convolution operation and BN&ReLU operation on C1 to output a feature map of size 64×64×512 and assign it to C1. It is easy to find that the proportions of O2 and C1 in the original signal are 15 / 16 and 3 / 64, respectively. Then, the feature map of size 64×64×512 is obtained by weighted averaging. After passing through the Conv2 convolution block and 5 Identity blocks, a feature map of 32×32×1024 (denoted as O1) is obtained.

[0073] (4) Perform a (3×3, 1024) convolution operation and BN&ReLU operation on C0 to output a feature map of size 32×32×1024 and assign it to C0. It is easy to find that the proportions of O1 and C0 in the original signal are 63 / 64 and 1 / 64, respectively. Then, the feature map of size 32×32×1024 is obtained by weighted averaging. After passing through the Conv2 convolution block and 2 Identity blocks, a feature map of size 16×16×2048 (denoted as O0) is obtained and output.

[0074] 3) Forming a target detection module

[0075] The main process involves upsampling (using deconvolution in this invention) of the output from step 2) to obtain a feature map with width and height of N. in / 2(N in (representing the width and height of the original image), with a depth of M. out (not less than 16M) in M in The output represents the depth of the original image. Three parallel feature map input lines are drawn from this output. The first line performs convolution, batch normalization, activation, and other operations to obtain an output of size N. in / 2×N in A 2×1 feature map is used, where each neuron determines whether the image location corresponding to that neuron contains a ship IMO. The second line obtains an output of size N through convolution and other operations. in / 2×N in The two feature maps, each 2×2, are used to locate the center point of the original image containing the ship's IMO. The third line uses convolution and other operations to obtain an output of size N. in / 2×N in The two-layer feature map (2×2) is used to determine the width and height of the ship IMO in the original image. Finally, the information from the three routes is combined to locate the ship IMO in the original image. The relevant mathematical expressions for the target detection module are as follows:

[0076]

[0077] In the formula, The width and height of the input feature map are used for deconvolution. The width and height of the output feature map during deconvolution. This represents the deconvolution operation, where sigmoid is the activation function, and the definitions of other parameters are the same as in step 2);

[0078] The target detection module of the present invention is for targets made by Figure 4 The specific implementation process of the output feature map O0 (attached) Figure 5 )for:

[0079] (1) Perform deconvolution operation on the feature map O0 output by the residual neural network module. The deconvolution kernel is 4×4, the stride is 2, the padding is unchanged, and the number of output layers is 512 (abbreviated as 4×4, stride=2, 516). Then perform BN&ReLU operation to output a feature map with a size of 32×32×512 (denoted as O4). Perform deconvolution and BN&ReLU operation on O4 (4×4, stride=2, 256) to output a feature map with a size of 64×64×256 (denoted as O5). Perform deconvolution and BN&ReLU operation on O5 (4×4, stride=2, 64) to output a feature map with a size of 128×128×64 (denoted as O6).

[0080] (2) Three parallel lines are drawn from O6. The first line is subjected to (3×3, 64) convolution and BN&ReLU operation, and (1×1, 64) convolution and BN&ReLU operation, outputting a feature map of size 128×128×1. Each neuron in the feature map is activated by sigmoid. Each activated neuron is used to determine whether the image position corresponding to the neuron contains a ship IMO. Then, the other two lines are subjected to (3×3, 64) convolution and BN&ReLU operation, and (1×1, 64) convolution and BN&ReLU operation, outputting two feature maps of size 128×128×1, which are used to predict the center point position and width and height of the ship IMO, respectively.

[0081] First, the input image is decomposed using a wavelet decomposition module to extract target feature information at several levels (the number of levels can be customized according to the actual situation). Each level is weighted and averaged with the corresponding (same feature size) feature layer in the residual neural network module according to its proportion in the original signal. The result is then encoded by the residual neural network module. Finally, the encoded result is decoded and decoupled by the target detection module to separate three feature maps for IMO detection. The ship's IMO is detected by the feature information of these three feature maps.

[0082] By decomposing the input image using wavelet decomposition technology, more and more detailed image feature information can be obtained, enabling the deep learning model to learn from more image input information. Specific formula steps or modules of wavelet decomposition can extract feature information of different frequencies. At the same time, it is processed with specific formula steps of the residual neural network module, thereby avoiding possible interference problems between feature information of different frequencies in the original image. Finally, it is used for ship IMO detection, which is designed for marine environment and IMO detection, and can be used for detection through UAVs or automatic video monitoring.

[0083] Over 100 IMO images of ships were collected from a border checkpoint in a certain city and from a shipping information website. Simultaneously, 2000 IMO images were generated using image enhancement technology, of which 1768 were usable. 1372 were selected for training and validation, and 396 were used for testing. The overlap rate of the prediction results was set to be no less than 0.5. Experimental results showed that the existing state-of-the-art (SOTA) algorithm had an average accuracy of 49.84%, a precision of 80.10%, a recall of 39.65%, and an F1-score of 53%. The present invention achieved an average accuracy of 56.22%, a precision of 83.33%, a recall of 45.45%, and an F1-score of 59%. Therefore, the measurement or identification results of the present invention improved the average accuracy by 6.38%, the precision by 3.23%, the recall by 5.8%, and the F1-score by 6%.

[0084] By conducting IMO (Immigration Management Office) inspections of ships, immigration authorities can automatically patrol and monitor vessels in port areas using drones and video surveillance, thereby improving police efficiency.

[0085] The above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Therefore, any changes made in accordance with the shape and principle of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for detecting ship IMO based on wavelet decomposition and residual neural network, characterized in that, Includes the following steps: 1) Wavelet decomposition is performed on the input ship IMO image using wavelet technology to form a wavelet decomposition module, and the output feature map is obtained. ; 2) By analyzing the feature map obtained in step 1), Each layer in the algorithm undergoes convolution and weighted averaging to obtain the final feature map output. Simultaneously, a residual neural network module is formed; 3) Based on the results obtained in step 2), Perform deconvolution, convolution, batch normalization, and activation operations to complete the detection and recognition of ship IMOs in the input image; Step 1) uses the following formula to obtain the output feature map. : （1） （2） （3） （4） （5） （6） In the formula, Let represent the set of all square-integrable functions over the real number field. yes An orthogonal multiresolution analysis on the , yes A set of closed subspaces on a column, yes A function that satisfies orthogonal multiresolution analysis. It is space orthonormal basis for middle orthogonal complement, It is space orthonormal basis This represents the inner product of two functions over their domain. and Low-pass filter for wavelet decomposition of signal and high-pass filter frequency response, A positive integer represents the current level of the wavelet decomposition. and for and The coefficient sequence, and They represent and The conjugate of complex numbers, Represents the unit impulse function. and There are two coefficient matrices, with dimensions of . , This represents the width and height of the input image signal; the width and height of the input image are equal. and express and The conjugate transpose of the matrix. Indicates the input image signal. , , , They represent The wavelet decomposition results in the scale component, horizontal wavelet component, vertical wavelet component, and diagonal wavelet component. The function represents concatenating or chaining matrices. This represents the output after matrix concatenation.

2. The method for detecting ship IMO based on wavelet decomposition and residual neural network according to claim 1, characterized in that: Step 2) involves wavelet decomposition of the image signal. To process, Convolution, max pooling, batch normalization, and activation operations are performed to obtain a feature map of the same size. Same output but different depths Then to Perform a convolution operation to make its depth the same Same, then according to and The two outputs are weighted and averaged based on their respective proportions in the original image signal. This process is repeated until the output is calculated. .

3. The method for detecting ship IMO based on wavelet decomposition and residual neural network according to claim 2, characterized in that: The mathematical expression used in step 2) is as follows: （7） （8） （9） （10） （11） In the formula, This represents the width and height of the hidden layer of the current input feature map in the neural network. This indicates the depth of the input feature map. This represents the convolution operation. Indicates the size of the convolution kernel. This indicates the stride of the convolution kernel when performing convolution on the input feature map. This indicates the padding size for the input feature map. The function represents the width and height of the output feature map in a neural network. Indicates rounding down. This indicates the depth of the output feature map. Indicates the first The value of a specific neuron in a specific feature map of a training sample. Represents the number of training samples, statistic and These are the neurons in Point estimates of the mean and variance of the values ​​taken from each training sample. To avoid the denominator being zero, use a constant greater than zero. for Standardized normalized output for The corrected output, where and The hyperparameters used in the correction. This is the activation function.

4. The method for detecting ship IMO based on wavelet decomposition and residual neural network according to claim 3, characterized in that: Step 3) includes the following: The output of step 2) is used for upsampling to obtain the feature map's width and height. ,That This represents the width, height, and depth of the original image. and not less than , The output represents the depth of the original image. Three parallel feature map input lines are derived from this output. The first line undergoes convolution, batch normalization, and activation operations to obtain an output of size [size missing]. The feature map is used to determine whether the image location corresponding to that neuron contains a ship IMO. The second line obtains the output size through a convolution operation. The two feature maps are used to locate the center point of the original image where the ship IMO is located. The third line obtains the output size through a convolution operation. The two feature maps are used to determine the width and height of the ship IMO in the original image. Finally, the information from the three routes is combined to locate the ship IMO in the original image.

5. The method for detecting ship IMO based on wavelet decomposition and residual neural network according to claim 4, characterized in that: The mathematical expression used in step 3) is as follows: （12） （13） （14） In the formula, The width and height of the input feature map are used for deconvolution. The width and height of the output feature map during deconvolution. This represents the deconvolution operation. The activation function is defined as follows (other parameters are defined in step 2).

6. The method for detecting ship IMO based on wavelet decomposition and residual neural network according to claim 1, characterized in that: Obtain the output feature map Specifically, it includes the following steps: 1.1) According to formula (1), we can obtain ,as well as The orthonormal basis of each orthogonal subspace after decomposition; 1.2) According to formula (2), we can obtain Decomposed into and The frequency responses of the filters used are respectively and Furthermore, based on the frequency components of the two filter outputs, it can be determined that... and These are respectively denoted as low-pass filters and high-pass filter The frequency response is obtained, and the coefficient sequences of the two frequency responses are obtained simultaneously. and ; 1.3) According to formula (3), the coefficient sequence can be obtained. and Constraint relationships between them; 1.4) According to formula (4), two coefficient matrices can be constructed. and The size of the matrix is , This indicates the width and height of the input image signal, where the width and height of the input image are equal. 1.5) According to formula (5), the input image signal is... By decomposing the matrix, the coefficient matrix can be easily obtained. and conjugate transpose and Finally, the result can be obtained through matrix multiplication. The scale component, horizontal wavelet component, vertical wavelet component, and diagonal wavelet component are denoted as follows: , , , ; 1.6) According to formula (6), when At that time, , , Perform matrix concatenation and output ,when At that time, , , , Perform matrix concatenation and output The final output feature map .

7. The method for detecting ship IMO based on wavelet decomposition and residual neural network according to claim 4, characterized in that: In step 2), by analyzing the feature map Each layer in the algorithm undergoes convolution and weighted averaging to obtain the final feature map output. The steps include: 2.1) According to formulas (7) and (8), the input feature map is processed. After performing convolution or max pooling operations, the width and height of the final output feature map are the same. Same, depth cannot be less than The depth is calculated, and the final output feature map is denoted as... ; 2.2) According to formulas (9), (10), and (11), perform batch normalization and activation operations on each convolution result in 2.1); 2.3) According to formulas (7), (8), (9), (10), and (11), for Perform convolution, batch normalization, and activation operations; the output feature map size is the same. Same, then according to and Weighting in the original input image and after the operation Perform a weighted average; 2.4) Repeat steps 2.1), 2.2), and 2.3) until output. .

8. The method for detecting ship IMO based on wavelet decomposition and residual neural network according to claim 5, characterized in that: In step 3), by... Perform deconvolution, convolution, batch normalization, and activation operations to complete the detection and recognition of ship IMOs in the input image; Includes the following steps: 3.1) Based on formulas (9), (10), (11), (12), and (13), the input... Perform multiple deconvolution operations, denoted as the number of deconvolutions. Furthermore, after each deconvolution, batch normalization and activation operations are performed to ensure that the final output feature map has a width and height that are one-quarter of the original image's width and height, rounded down, and a depth that is no less than 16 times the original image's depth. The final output feature map is denoted as... ; 3.2) According to formulas (7)(8)(9)(10)(11)(14), for Three parallel lines are drawn. The first line is convolved, batch normalized, and activated to obtain a feature map with invariant width and height and a depth of 1. Then, each neuron in this feature map is processed... Activation is performed on each neuron, which is then used to determine whether the image location corresponding to that neuron contains a ship IMO. Convolution, batch normalization, and activation operations are then performed on the other two lines to obtain feature maps with unchanged width and height and a depth of 2. These feature maps are used to predict the center point location and width and height of the ship IMO. Focal loss is used to learn the neural network parameters for the first feature map, while L1 absolute loss or L2 squared loss can be used to learn the parameters for the other two feature maps.

9. A ship identification method based on wavelet decomposition and residual neural network as described in any one of claims 1 to 8, characterized in that: A method for detecting and identifying ship registration numbers based on a ship IMO detection method using wavelet decomposition and residual neural networks.

Images