A ship detection method and device based on three-dimensional information and a storage medium
By enhancing image edge and texture information through the Laplacian algorithm and wavelet transform, and incorporating domain knowledge, the problem of small inter-class differences and large intra-class differences in ship detection is solved, achieving higher detection accuracy and speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI UNIV
- Filing Date
- 2023-03-28
- Publication Date
- 2026-05-19
AI Technical Summary
Existing fine-grained target detection methods suffer from small inter-class differences and large intra-class differences in ship detection, resulting in slow detection speed and insufficient accuracy, especially in the identification of ships on the sea surface.
The Laplacian algorithm is used to generate image gradient maps, which are then combined with wavelet transform to enhance the texture information of the feature maps. Domain knowledge is also incorporated, and the visual feature maps are used for classification and regression to obtain the ship's category and location.
It improves the accuracy and speed of ship detection at sea, overcomes the problem of small differences between categories and large differences within categories, enhances the information dimension of feature maps, and improves the accuracy of detection.
Smart Images
Figure CN116343141B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition technology, and in particular to a ship detection method, equipment and storage medium based on three-dimensional information. Background Technology
[0002] With the development of water transportation, the safety of ship navigation has gradually gained attention, making the identification and inspection of ships, especially those at sea, particularly important. Computer vision has been widely adopted in ship inspection.
[0003] Fine-grained object detection methods are a common approach for ship detection. These methods, building upon general object detection, determine the subcategories of ships through fine-grained identification. Based on their reliance on labeled data, fine-grained object detection can be categorized into strongly supervised and weakly supervised methods. Strongly supervised methods, such as Part R-CNN and Mask-CNN, utilize additional manually labeled information, including local region locations, in addition to the target's category labels and bounding box information during model training. While strong supervision can improve algorithm accuracy to some extent, it further increases the burden of data labeling, and manually labeled component regions may not be the optimal discriminative regions. The second type is the weakly supervised method, such as DFL-CNN and NTS-Net, which relies solely on target category and location labels for fine-grained identification. Weak supervision often employs a localization-identification approach: first, the identifiable target region is located without additional component labeling, and then feature extraction and classification regression are performed on the target region. By locating first and then identifying, identifiable fine-grained target features can be further extracted, thereby achieving accurate fine-grained target classification and identification.
[0004] However, the aforementioned fine-grained target detection methods still suffer from the problem of small inter-class differences and large intra-class differences in ships, making it difficult to accurately detect ships on the sea surface. Specifically, most existing methods locate relevant regions using a sliding window approach, which generates a large number of irrelevant candidate regions and cannot quickly filter out redundant candidate regions while retaining the identifiable target regions. This significantly affects the speed of target detection. Secondly, after locating easily classifiable target regions, learning complex feature representations for identification is also a crucial step. Existing methods directly perform fine-grained classification on the extracted single feature map, failing to refine the features of the target region to be identified. This results in insufficient learning of fine-grained target representations, which also limits the accuracy of fine-grained identification.
[0005] Therefore, there is an urgent need to design a more accurate method for detecting ships at sea, which can overcome the problem that there are small differences between ship categories but large differences within the same category. Summary of the Invention
[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a ship detection method, equipment and storage medium based on three-dimensional information with high ship detection accuracy, which can solve the problem that there are small differences between ship categories and large differences within the same category.
[0007] The objective of this invention can be achieved through the following technical solutions:
[0008] According to a first aspect of the present invention, a ship inspection method based on three-dimensional information is provided, the method comprising the following steps:
[0009] Step S1: Acquire the original high-resolution image I of the ship;
[0010] Step S2: Generate image gradient map I using the Laplacian algorithm. g The first feature map F, which contains image edge information, is generated by convolving it with the original high-resolution image I of the ship.
[0011] Step S3: Employ wavelet transform to enhance the ship texture information on the first feature map F in a hierarchical manner, generating a second feature map N containing both the original information and texture information;
[0012] Step S4: Integrate domain knowledge into the second feature map N to generate a visual feature map F containing category information. t ;
[0013] Step S5: Process the visual feature map F t Classification and regression analysis are performed to obtain the category and location of the vessel to be inspected.
[0014] Preferably, step S2 includes the following sub-steps:
[0015] Step S21: Use the Laplacian algorithm to obtain the image gradient map I from the original high-resolution ship image I. g The expression is:
[0016] I g =lap(I)
[0017] In the formula, lap represents the Laplace algorithm, I g ∈R H*W*1 H and W represent the length and width of the image gradient map, respectively;
[0018] Step S22: Fuse the original high-resolution ship image I with the image gradient map I using the Concat join operation. g The first feature map F, containing image edge information, is generated through convolution operations, and its expression is:
[0019] F = Conv 1×1 (Concat(II g ))
[0020] In the formula, F∈R H*W*C C represents the channel dimension of the image gradient map.
[0021] Preferably, step S3 includes the following sub-steps:
[0022] Step S31: Decouple the first feature map F output in step S2 using wavelet transform, and extract the texture feature D from the feature map. The expression is:
[0023] D = WT(F)
[0024] In the formula, F∈R H*W*C , C is the channel dimension of the first feature map F, and WT is the wavelet transform;
[0025] Step S32: Utilize spatial attention to hierarchically enhance the ship texture information D on the feature map, generating an attention W that highlights the ship texture information, expressed as:
[0026] W = SA(D)
[0027] In the formula, SA stands for Spatial Attention;
[0028] Step S33: Multiply the attention W with the texture information D, and use inverse wavelet transform to obtain the texture feature information E, expressed as:
[0029] E = IWT(W⊙D)
[0030] In the formula, E∈R H*W*C IWT represents the inverse wavelet transform;
[0031] Step S34: Fuse the first feature map F with the texture feature information E to generate a second feature map N containing the original information and texture information.
[0032] Preferably, step S34 specifically involves: fusing the first feature map F and texture feature information E using residual connections to generate a second feature map N containing both original information and texture information, expressed as:
[0033]
[0034] In the formula, N∈R H*W*C , This is a vector concatenation operation.
[0035] Preferably, the wavelet transform in step S3 is the Haar wavelet transform.
[0036] Preferably, step S4 includes the following sub-steps:
[0037] Step S41: Acquire domain knowledge: Use the pre-trained word vector library GloVe model to extract d-dimensional category word vectors w based on fine-grained ship categories. c Generate knowledge of the ship target domain; where d is the dimension of the feature vector and c represents the number of target categories;
[0038] Step S42, Fusing Domain Knowledge: Combine the second feature map N with the category word vector w c The feature vector f at each position on the second feature map N is obtained by fusion;
[0039] Step S43: Obtain feature vectors f for all positions that are related to the encoding and category c. c The expression is:
[0040]
[0041] In the formula, w and h represent each position on the second feature map N;
[0042] Step S44: Repeat f for all categories c The process obtains all feature vectors F related to the category. cs All category-related feature vectors F cs The feature map is fused with the second feature map N to generate a visual feature map F containing category information. t The expression is:
[0043] F t =F cs ⊙N
[0044] In the formula, F t ∈R H*W*C F cs ∈R H*W*C H and W represent the length and width of the image gradient map, respectively, and C is the channel dimension of the first feature map F.
[0045] Preferably, step S42 specifically involves fusing the second feature map N and the category word vector w using a low-rank bilinear pooling method. c The feature vector f at each position on the second feature map N is obtained, and its expression is:
[0046] f = P T tanh((U T N)⊙(V T w c ))+b
[0047] In the formula, P is the intermediate transformation vector, and tanh(·) is the hyperbolic tangent function; These are learnable parameters, where M is the feature vector dimension, and d... sd1 and d2 are the dimensions of the category feature vector and the joint embedding and output, respectively.
[0048] Preferably, step S5 specifically involves: using a target category classifier and a regressor to process the visual feature map F. t Classification and regression are performed to obtain the ship category probability vector p and position (x, y, w, h), where x and y represent the center coordinates of the rectangle, and w and h represent the width and height of the rectangle.
[0049] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement any of the methods described above.
[0050] According to a third aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the methods described herein.
[0051] Compared with the prior art, the present invention has the following advantages:
[0052] This invention utilizes three dimensions of information—image dimension, feature map dimension, and domain knowledge dimension—to enhance and supplement the feature map information of ships, thereby addressing the problem of small differences between ship categories and large differences within a category. This makes it easier to detect ships on the sea surface and improves the accuracy of detecting ships on the sea surface. Attached Figure Description
[0053] Figure 1 This is a flowchart of a ship inspection method based on three-dimensional information according to the present invention;
[0054] Figure 2 The ship image is input for the method of this invention;
[0055] Figure 3 This is an image of the ship inspection results obtained by the method of the present invention. Detailed Implementation
[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0057] Example 1:
[0058] like Figure 1 As shown in the figure, this embodiment presents a ship inspection method based on three-dimensional information, including the following steps:
[0059] Step S1: Acquire the original high-resolution image I of the ship;
[0060] Step S2: Regarding image dimensions: The Laplacian algorithm is used to generate the image gradient map I. g The first feature map F, which contains image edge information, is generated by convolving it with the original high-resolution image I of the ship.
[0061] Step S3: In terms of feature map dimensions: Wavelet transform is used to enhance the ship texture information on the first feature map F in a hierarchical manner, generating a second feature map N containing both original information and texture information;
[0062] Step S4: Integrate domain knowledge into the second feature map N to generate a visual feature map F containing category information. t ;
[0063] Step S5: Process the visual feature map F t Classification and regression analysis are performed to obtain the category and location of the vessel to be inspected.
[0064] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.
[0065] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0066] The processing unit executes the various methods and processes described above, such as methods S1 to S5. For example, in some embodiments, methods S1 to S5 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S5 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S5 by any other suitable means (e.g., by means of firmware).
[0067] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.
[0068] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0069] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0070] Example 2:
[0071] This embodiment is basically the same as Embodiment 1, except that:
[0072] 1. Input images acquired by a high-resolution camera, such as... Figure 2 As shown.
[0073] In this embodiment, see Figure 1 In step S2, regarding image dimensions, the Laplacian algorithm is used to generate an image gradient map to enhance the edge information of the feature map. The process is as follows:
[0074] (2-1) Use the Laplacian algorithm to obtain the image gradient map I from the high-resolution image I obtained in step S1. g The specific calculation formula is as follows:
[0075] I g =lap(I)(1)
[0076] Where lap represents the Laplace algorithm, I g ∈R H*W*1 H and W represent the length and width of the feature map, respectively, as 1024.
[0077] (2-2) Merge the original image I and the image gradient map I using the Concat operation. g Finally, a 1x1 convolution operation is used to generate a feature F containing image edge information. The specific calculation formula is as follows:
[0078] F = Conv 1×1 (Concat(II g (2)
[0079] Where F∈R H*W*C C indicates that the feature map channel dimension is 3, and H and W indicate that the feature map length and width are 1024 and 1024, respectively.
[0080] In this embodiment, see Figure 1 In step S3, wavelet transform is used to enhance the ship texture information on the feature map in a hierarchical manner. The process is as follows:
[0081] (3-1) Decouple the feature map F output from step S2 using Haar wavelet, and extract the texture feature D from the feature map. The calculation formula is as follows:
[0082] D = WT(F) (3)
[0083] Where F∈R H*W*C , WT represents Haar wavelet transform, C indicates that the feature map channel dimension is 3, and H and W indicate that the feature map length and width are 1024, respectively.
[0084] (3-2) Spatial attention is used to hierarchically enhance the ship texture information D on the feature map, and an attention W that highlights the ship texture information is generated. The calculation formula is as follows:
[0085] W=SA(D) (4)
[0086] in, SA represents spatial attention, C indicates that the feature map channel dimension is 3, and H and W indicate that the feature map length and width are 1024 and 1024, respectively.
[0087] (3-3) Multiply the attention W with the texture information D, and use the inverse wavelet transform to recover the texture feature information E that is more obvious than the original feature. The calculation formula is as follows:
[0088] E=IWT(W⊙D) (5)
[0089] Where, E∈R H*W*C IWT represents the Haar wavelet inverse transform, C represents the feature map channel dimension as 3, and H and W represent the feature map length and width as 1024, respectively.
[0090] (3-4) The original information F and texture feature information E are fused using residual connection to form a feature map N containing both the original information and texture information. The calculation formula is as follows:
[0091]
[0092] Where, N∈R H*W*C C indicates that the feature map channel dimension is 3, and H and W indicate that the feature map length and width are 1024 and 1024, respectively.
[0093] In this embodiment, see Figure 1 In step S4, domain knowledge is incorporated as a one-dimensional element into the visual feature map to enhance ship category information. The process is as follows:
[0094] (4-1) Domain knowledge acquisition: Using a pre-trained word vector library (GloVe), the model extracts d-dimensional category word vectors w based on fine-grained ship categories. c This forms knowledge of the ship target domain, where d is 300 and c represents the number of target categories, 14.
[0095] (4-2) Domain knowledge fusion: The corresponding second feature map N and category word vector w are fused using the low-rank bilinear pooling method. c The feature vector f at each position on the second feature map N is formed, and the calculation formula is as follows:
[0096] f = P T tanh((U T N)⊙(V T w c ))+b (7)
[0097] Where tanh(·) is the hyperbolic tangent function. These are learnable parameters. M is set to 2048, d... s Set to 300, and d1 and d2 are the joint embedding and output sizes, set to 1024 and 1024 respectively.
[0098] (4-3) Obtain feature vectors f for all positions that encode and are related to category c. c The calculation formula is as follows:
[0099] fc =∑ w，h f(8)
[0100] Where w and h represent each position on the feature map, and the values of w and h are 0-1024.
[0101] (4-4) Repeatedly obtain f for all categories c The process obtains all feature vectors F related to the category. cs All category-related feature vectors F cs The feature map N output from step S4 is fused to form F, which contains category information. t The calculation formula is as follows:
[0102] F t =F cs ⊙N (9)
[0103] Among them, F t ∈R H*W*C F cs ∈R H*W*C C indicates that the feature map channel dimension is 3, and H and W indicate that the feature map length and width are 1024 and 1024, respectively.
[0104] In this embodiment, see Figure 1 In step S5, a target category classifier and regressor are used to obtain the ship category probability vector p and its position (x, y, w, h), where x and y represent the center coordinates of the rectangle, and w and h represent the width and height of the rectangle. The target probability vector p and position of the ship detection categories (container ship, bulk carrier) are shown below. Figure 3 .
[0105] This embodiment provides a ship detection method based on three-dimensional information, (1) inputting an image acquired by a high-resolution camera, such as... Figure 2 As shown; (2) In terms of image dimension, the Laplacian algorithm is used to generate image gradient maps to enhance the edge information of feature maps; (3) In terms of feature map dimension, wavelet transform is used to enhance the ship texture information on feature maps in a hierarchical manner; (4) Domain knowledge is incorporated into the visual feature map as a one-dimensional element; (5) The visual feature map is classified and regressed to obtain the category and location of the detected ships, as shown in the figure. Figure 3 As shown.
[0106] The method described in this invention utilizes three dimensions of information—image dimension, feature map dimension, and domain knowledge dimension—to enhance and supplement the feature map information of ships, thereby improving the accuracy of ship detection. Compared with traditional methods, the method in this embodiment, utilizing three-dimensional information, makes it easier to detect ships on the sea surface and improves the accuracy of ship detection.
[0107] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A ship inspection method based on three-dimensional information, characterized in that, The method includes the following steps: Step S1: Acquire the original high-resolution image of the ship. ; Step S2: Generate image gradient map using the Laplacian algorithm. Compared with the original high-resolution images of the ship Convolution generates a first feature map containing image edge information. ; Step S3: Employ wavelet transform to enhance the first feature map in a hierarchical manner. Based on the ship texture information, a second feature map containing both the original information and the texture information is generated. ; Step S4: Integrate domain knowledge into the second feature map Generate visual feature maps containing category information. This includes the following sub-steps: Step S41: Acquire domain knowledge: Use the pre-trained word vector library GloVe model to extract d-dimensional category word vectors based on fine-grained ship categories. Generate knowledge of the ship target domain; where d is the dimension of the feature vector and c represents the number of target categories; Step S42, Integrate domain knowledge: Combine the second feature map With category word vectors The second feature map is obtained by fusion. Feature vector at each position ; Step S43: Obtain the code and category for all locations. Feature vectors of relevant information The expression is: In the formula, Representing the second feature map On each position; Step S44: Repeat the acquisition for all categories. The process obtains all category-related feature vectors. All category-related feature vectors With the second feature map The fusion process generates a visual feature map that includes category information. The expression is: In the formula, , , Let C represent the length and width of the image gradient map, respectively, and C be the first feature map. The channel dimension; Step S5: Visual feature map Classification and regression analysis are performed to obtain the category and location of the vessel to be inspected.
2. The ship inspection method based on three-dimensional information according to claim 1, characterized in that, Step S2 includes the following sub-steps: Step S21: Process the original high-resolution image of the ship. Image gradient map obtained using the Laplacian algorithm The expression is: In the formula, This represents the Laplace algorithm. , These represent the length and width of the image gradient map, respectively. Step S22, through The connection operation fuses the original high-resolution images of the ship. Image gradient map The first feature map containing image edge information is generated through convolution operations. The expression is: In the formula, C represents the channel dimension of the image gradient map.
3. The ship inspection method based on three-dimensional information according to claim 1, characterized in that, Step S3 includes the following sub-steps: Step S31: Apply wavelet transform to the first feature map output in step S2. Decoupling and extracting texture features from feature maps The expression is: In the formula, , C is the first feature map The channel dimension, Wavelet transform; Step S32: Utilize spatial attention to hierarchically enhance ship texture information on the feature map. Attention that highlights ship texture information The expression is: In the formula, , Spatial attention; Step S33: Utilize attention With texture information Multiply the samples and use inverse wavelet transform to obtain texture feature information. The expression is: In the formula, , Indicates the inverse wavelet transform; Step S34: Transfer the first feature map With texture feature information The two features are fused to generate a second feature map containing both the original information and the texture information. .
4. The ship inspection method based on three-dimensional information according to claim 3, characterized in that, Step S34 specifically involves fusing the first feature map using residual connections. With texture feature information Generate a second feature map containing both original and texture information. The expression is: In the formula, , This is a vector concatenation operation.
5. The ship inspection method based on three-dimensional information according to claim 3, characterized in that, The wavelet transform in step S3 is the Haar wavelet transform.
6. The ship inspection method based on three-dimensional information according to claim 1, characterized in that, Step S42 specifically involves fusing the second feature map using a low-rank bilinear pooling method. With category word vectors The second feature map is obtained. Feature vector at each position The expression is: In the formula, Let be the intermediate transformation vector, and tanh(⋅) be the hyperbolic tangent function; , , , , are learnable parameters, where, For the feature vector dimension, For the category feature vector dimension, The dimensions for the combined embedding and output.
7. The ship inspection method based on three-dimensional information according to claim 1, characterized in that, Step S5 specifically involves: using a target category classifier and a regressor to analyze the visual feature map. Classification and regression are performed to obtain the probability vector of ship categories. and location ,in, Represents the center coordinates of the rectangle. This represents the width and height of the rectangle.
8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.