Method, system and device for underwater image target detection

Abstract

The present invention discloses a method, system and device for underwater image target detection, including: acquiring an underwater image, preprocessing the underwater image; inputting the preprocessed underwater image into a feature extraction network for feature extraction to obtain underwater image features; sending the preprocessed underwater image into a detection model to obtain a mask of the target in the underwater image; inputting the underwater image features and the mask of the target in the underwater image into a pre-trained generative adversarial model to obtain a pseudo underwater image; sending the pseudo underwater image into the detection model to obtain the position information, mask and category of the target in the underwater image to complete target detection. The present invention can realize underwater image target detection.

Description

Technical Field

[0001] The present invention relates to the field of target detection methods, and in particular to a target detection method, system and device for underwater images. Background Art

[0002] Modern development has put forward higher requirements for the intelligent development of underwater unmanned equipment. Among them, target detection technology is an important supporting technology for underwater equipment environmental perception, target recognition and tracking, and is one of the key technologies for improving the intelligence of underwater equipment. However, due to the complexity of the underwater environment, the development of intelligent underwater target detection technology applied to underwater unmanned equipment is slow. Existing research generally uses traditional image processing methods to detect and identify simple targets in clean waters, but valuable targets often fail in the face of complex ocean backgrounds, which cannot meet the equipment requirements.

[0003] With the development and application of deep learning technology, especially the significant progress in the field of image target detection, new ideas have been provided for the development of intelligent recognition systems for underwater unmanned equipment. Deep learning technology obtains the ability to analyze and process images by large-scale training of a large number of labeled samples. However, the deployment and application of deep learning technology in intelligent recognition systems for underwater unmanned equipment still faces many problems and challenges:

[0004] 1) Due to the nature of the water body, the influence of lighting factors and the presence of microorganisms, underwater images contain large noise and poor image quality.

[0005] 2) There are many kinds of organisms in seawater, and the collected image data conforms to the long-tail distribution, that is, there are some uncommon organisms and sediments and only a small number of samples can be collected, resulting in extremely unbalanced samples, which limits the recognition and generalization capabilities of the deep model.

[0006] 3) Underwater equipment has real-time recognition requirements in actual operations, and the computing resources it carries are limited, which restricts the application and effect of deep learning methods. Summary of the invention

[0007] The purpose of the present invention is to provide a method, system and device for underwater image target detection, aiming to solve the problem of underwater image target detection.

[0008] The present invention provides a method for detecting an underwater image target, comprising:

[0009] S1, acquiring an underwater image, and preprocessing the underwater image;

[0010] S2, inputting the preprocessed underwater image into a feature extraction network to extract features to obtain underwater image features;

[0011] S3, sending the preprocessed underwater image to the detection model to obtain the mask of the target in the underwater image;

[0012] S4, inputting the underwater image features and the mask of the target in the underwater image into a pre-trained generative adversarial model to obtain a pseudo underwater image;

[0013] S5. Send the pseudo underwater image to a detection model to obtain the location information, mask and category of the target in the underwater image to complete target detection.

[0014] The present invention also provides an underwater image target detection system, comprising:

[0015] Acquisition module: used for acquiring underwater images and preprocessing the underwater images;

[0016] Extraction module: used to input the preprocessed underwater image into the feature extraction network for feature extraction to obtain underwater image features;

[0017] Mask module: used to send the preprocessed underwater image to the detection model to obtain the mask of the target in the underwater image;

[0018] A pseudo image module: used for inputting the underwater image features and the mask of the target in the underwater image into a pre-trained generative adversarial model to obtain a pseudo underwater image;

[0019] Target detection module: used to send the pseudo underwater image into the detection model to obtain the location information, mask and category of the target in the underwater image to complete target detection.

[0020] An embodiment of the present invention further provides an underwater image target detection device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above method when executed by the processor.

[0021] An embodiment of the present invention further provides a computer-readable storage medium, on which a program for implementing information transmission is stored, and when the program is executed by a processor, the steps of the above method are implemented.

[0022] By adopting the embodiments of the present invention, underwater target detection can be achieved.

[0023] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 is a flow chart of a method for detecting an underwater image object according to an embodiment of the present invention;

[0026] Figure 2 is a schematic diagram of weak sample target detection in a target detection method for underwater images according to an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of an underwater image target detection system according to an embodiment of the present invention;

[0028] Figure 4 Schematic diagram of an underwater image target detection device according to an embodiment of the present invention.

[0029] Description of reference numerals:

[0030] 310: acquisition module; 320: extraction module; 330: mask module; 340: pseudo image module; 350: target detection module. DETAILED DESCRIPTION

[0031] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0033] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0034] Method Embodiment

[0035] According to an embodiment of the present invention, a method for detecting an underwater image target is provided. Figure 1 FIG. 1 is a flow chart of a method for detecting an underwater image object according to an embodiment of the present invention. Figure 1 As shown, specifically including:

[0036] S1, acquiring an underwater image, and preprocessing the underwater image;

[0037] S2, inputting the preprocessed underwater image into a feature extraction network to extract features to obtain underwater image features;

[0038] S2 specifically includes: inputting the preprocessed underwater image into at least one of ResNet-FPN, MobileNet and ShuffleNet to extract features to obtain underwater image features.

[0039] S3, sending the preprocessed underwater image to the detection model to obtain the mask of the target in the underwater image;

[0040] S3 specifically includes:

[0041] The candidate region p is sent to the domain candidate network RPN to extract the candidate region p where the target may exist. i , and for the candidate region p i Perform preliminary category judgment and box regression to determine the region p i Contains the target, where i represents the i-th image;

[0042] The candidate region p i The coordinates of the underwater image features are input into the ROI align layer to extract the region p i Features i,p ;

[0043] Classification Network Utilization x i,pCalculate the region p containing the target i The category to which the target belongs is determined, and box regression is performed again to determine the location information of the target and generate a mask of the target.

[0044] S4, inputting the underwater image features and the mask of the target in the underwater image into a pre-trained generative adversarial model to obtain a pseudo underwater image;

[0045] S4 specifically includes:

[0046] Separate the mask of the target in the underwater image and the underwater image features to obtain the foreground and background area features;

[0047] The foreground and background area features of the same batch of underwater images are randomly combined to obtain combined features, and the combined features are input into the convolutional network to generate new pseudo underwater images;

[0048] The new pseudo underwater images generated by the generative model are distinguished from the underwater images, the authenticity of the input images is judged, and the pseudo images that conform to the distribution of the original data are obtained.

[0049] S5. Send the pseudo underwater image to a detection model to obtain the location information, mask and category of the target in the underwater image to complete target detection.

[0050] The specific implementation methods are as follows:

[0051] The development of an underwater target detection system, especially a weak sample target detection method for underwater images, so that it can perform various intelligent combat / business tasks in real time is of great significance to the intelligent development of underwater unmanned equipment.

[0052] In order to solve the problem of weak samples, the effect of insufficient effective samples on the deep detection model can be effectively alleviated by using a generative model to generate pseudo data that conforms to the distribution of the original data. Generative adversarial network (GAN) is a generative model based on prior knowledge, which consists of a generative network G and a discriminative network D. G takes prior knowledge or noise as input to generate pseudo samples with the same distribution as real samples. D is usually a binary classifier that judges the input real samples and pseudo samples and judges the generated pseudo samples as false. In order to pass the judgment of the discriminative network D, the generative network G continuously learns parameters to make the generated pseudo samples closer to the real samples, while the discriminative network D can more accurately judge the source of the data. The pseudo samples generated by the generative network G can be used for subsequent training of the deep target detection model to alleviate the effect of insufficient effective samples.

[0053] The technical solution of the embodiment of the present invention is:

[0054] In the first aspect, the embodiments of the present invention provide a weak sample target detection method for underwater. Considering the weak sample problem of underwater images, a detection model based on a generative model is designed to alleviate the limitation of weak samples by generating pseudo samples that conform to the distribution of real underwater image samples and training a deep learning detection model. The detection method includes the following steps:

[0055] 1) Obtain underwater images through underwater optical camera sampling, and obtain images suitable for subsequent detection through data preprocessing;

[0056] Optionally, preprocessing includes:

[0057] At least one of image denoising, image enhancement, and image filtering.

[0058] 2) Input the underwater image features into the pre-trained feature extraction network model to extract the features of the underwater image;

[0059] The feature extraction network includes:

[0060] At least one of ResNet-FPN, MobileNet, and ShuffleNet. Using a lightweight feature extraction network MobileNet or ShuffleNet can effectively reduce computational complexity and enable the deep model to meet real-time requirements.

[0061] Images and labels are obtained from the ImageNet image library, and the feature extraction network is trained to obtain a pre-trained feature extraction network. Pre-training reduces the training difficulty of the model and accelerates the convergence of the model.

[0062] 3) Send the underwater image data to the detection model for training to obtain the image mask, category and location information;

[0063] The detection model is implemented through the Mask-RCNN algorithm.

[0064] 4) Input underwater image features and masks into a pre-trained generative model to separate background and foreground information, and then randomly combine them to generate a pseudo underwater image;

[0065] Generative models include: Generative adversarial networks.

[0066] 5) Sending the generated pseudo underwater image and the original underwater image data to the detection model for training;

[0067] The detection model is implemented through the Mask-RCNN algorithm.

[0068] 6) In the testing phase, the underwater image to be detected is sent to the detection network to detect the target, and the location information and category of the target are output.

[0069] In a second aspect, an embodiment of the present invention provides an underwater weak sample target detection device, the device comprising:

[0070] 1) Image acquisition module, used to acquire underwater images in real time and improve image quality through preprocessing;

[0071] 2) Feature extraction module, used to extract features from underwater images;

[0072] 3) Target generation module, used to generate pseudo samples that conform to the distribution of real underwater image samples to train the deep learning detection model;

[0073] 4) Target detection module, used to detect and classify the position of underwater targets, and output the target's location information and category.

[0074] The embodiments of the present invention provide a method, device, and electronic device for underwater weak sample target detection, which can realize online and real-time detection and positioning of targets under complex ocean backgrounds, and predict target motion.

[0075] Figure 2 is a schematic diagram of weak sample target detection in the target detection method for underwater images according to an embodiment of the present invention. Figure 2 As shown, the detection method comprises the following steps:

[0076] 1) Underwater images are obtained by sampling with an underwater optical camera, and images suitable for subsequent detection are obtained through data preprocessing. The underwater image dataset is sorted and annotated, and divided into a training set D train and the test set D test , set the i-th image x i Contains n targets to be detected, and each target to be detected has a corresponding label {m i,n ,b i,n ,c i,n}, where m i,n is the mask of the target to be detected, b i,n is the candidate box coordinate of the target to be detected, c i,n is the category of the target to be detected.

[0077] 2) Input the underwater image features into the pre-trained feature extraction network model Backbone to extract the features of the underwater image;

[0078] Optionally, the feature extraction network includes:

[0079] One of ResNet, MobileNet, and ShuffleNet. Using a lightweight feature extraction network MobileNet or ShuffleNet can effectively reduce the computational complexity and enable the deep model to meet real-time requirements.

[0080] 3) The underwater image to be detected x i Send it to the detection network to detect the target and send the target location information b i , mask m i And the category c i Output. Figure 2 As shown in Figure 1, the detection model includes a region proposal network RPN, a ROI align layer, and a classification network C, where:

[0081] The region proposal network RPN is used to generate candidate regions p i , extract the regions of interest where the target to be tested may exist in the image, and perform category possibility judgment and box regression operations on each region of interest;

[0082] The ROI align layer maps the candidate box coordinates predicted by RPN (relative to the original input image) to the original image features one by one to extract the candidate region features x i,p .

[0083] The classification network C uses the candidate region features x i,p Calculate the category c of the candidate area i , and perform box regression again to accurately locate the detection box b i , and generate a mask m for the target i .

[0084] 4) Detect the mask m output by the network i The original image features are input into the pre-trained generative adversarial model to generate a pseudo underwater image x′ i ;like Figure 2 As shown, the generative adversarial model includes a separation model M, a generation model G and a discriminant module D, where:

[0085] The separation model M is used to separate the foreground and background region features x i,f ,x i,b , the mask m output by the detection network i and the original image x i Input separation model M to obtain foreground and background region features x i,f ,x i,b , f represents foreground, b represents background;

[0086] x i,f ,x i,b =M(x i ,mi ) (1)

[0087] The generative model G generates the foreground and background region features x of the same batch of underwater images during training. i,f ,x j,b Randomly combine and use the convolutional network to generate a new pseudo underwater image x′ i , whose category is c i .

[0088] x′ i =G(x i,f ,x j,b ) (2)

[0089] The discriminant module D is used to distinguish the samples generated by the generative model G from the original underwater image samples and judge the authenticity of the input image. By jointly training the generative model G and the discriminant model D and confronting each other, a pseudo image that conforms to the distribution of the original data is obtained and output as ture, where j represents the jth image.

[0090] 5) As in 3 above, the pseudo underwater image x′ i Send it to the detection network for training, and the target location information b i , mask m i And the category c i Output.

[0091] 6) During the training phase, the underwater image x i Send it to the generative adversarial model and the detection model for joint training, and use the mask m obtained by detection i Generate pseudo underwater image x′ i The model is further trained to classify objects in the image, generate bounding boxes and segmentation operations, and calculate the loss:

[0092] L=L cls +L box +L mask (3)

[0093] Among them, L is the total loss of the detection model, L cls ,L box ,L mask The corresponding feature maps are used to calculate the loss of target classification, bounding box regression, and the difference between the mask and the true value.

[0094] The objective function V(G,D) of the generative adversarial model is:

[0095]

[0096] The generative adversarial model objective function and L are optimized until the generative adversarial model and the detection model converge.

[0097] 7) In the testing phase, the underwater image x i Input the detection model to classify the objects in the image and generate bounding boxes and segmentation masks.

[0098] The beneficial effects of the embodiments of the present invention are:

[0099] (1) The detection method designed by the embodiment of the present invention is based on a deep learning model. Compared with the traditional image processing algorithm, it can achieve better recognition and detection performance, especially when identifying targets in complex marine environments, it can significantly improve its detection effect.

[0100] (2) The detection model of the embodiment of the present invention uses a generative adversarial network method, so that the detection system can detect weak sample categories with insufficient effective sample data, thereby reducing the degradation of detection performance caused by weak samples.

[0101] (3) The embodiments of the present invention limit the number of network parameters, greatly reduce the time and space complexity of the model, ensure the real-time performance of the overall algorithm, and can achieve online and real-time detection and positioning of targets, making it more applicable to engineering applications.

[0102] (4) The embodiments of the present invention can realize real-time detection and positioning of targets under complex ocean backgrounds, while improving the detection effect and robustness of weak sample target classes.

[0103] System Example

[0104] According to an embodiment of the present invention, a target detection system for underwater images is provided. Figure 3 is a schematic diagram of an underwater image target detection system according to an embodiment of the present invention. Figure 3 As shown, specifically including:

[0105] Acquisition module: used for acquiring underwater images and preprocessing the underwater images;

[0106] Extraction module: used to input the preprocessed underwater image into the feature extraction network for feature extraction to obtain underwater image features;

[0107] The extraction module is specifically used to extract features from at least one of the input ResNet-FPN, MobileNet and ShuffleNet to obtain underwater image features.

[0108] Mask module: used to send the preprocessed underwater image to the detection model to obtain the mask of the target in the underwater image;

[0109] The mask module is specifically used for:

[0110] The candidate region p is sent to the domain candidate network RPN to extract the candidate region p where the target may exist.i , and for the candidate region p i Perform preliminary category judgment and box regression to determine the region p i Contains the target, where i represents the i-th image;

[0111] The candidate region p i The coordinates of the underwater image features are input into the ROI align layer to extract the region p i Features i,p ;

[0112] Classification Network Utilization x i,p Calculate the region p containing the target i The category to which the target belongs is determined, and box regression is performed again to determine the location information of the target and generate a mask of the target.

[0113] A pseudo image module: used for inputting the underwater image features and the mask of the target in the underwater image into a pre-trained generative adversarial model to obtain a pseudo underwater image;

[0114] Target detection module: used to send the pseudo underwater image into the detection model to obtain the location information, mask and category of the target in the underwater image to complete target detection.

[0115] The pseudo image module is specifically used for:

[0116] Separate the mask of the target in the underwater image and the underwater image features to obtain the foreground and background area features;

[0117] The foreground and background area features of the same batch of underwater images are randomly combined to obtain combined features, and the combined features are input into the convolutional network to generate new pseudo underwater images;

[0118] The generated new pseudo underwater image is distinguished from the underwater image, the authenticity of the input image is judged, and a pseudo image that conforms to the distribution of the original data is obtained.

[0119] The embodiment of the present invention is a system embodiment corresponding to the above-mentioned method embodiment. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0120] Device Example 1

[0121] The embodiment of the present invention provides an underwater image target detection device, such as Figure 4 As shown, it includes: a memory 40, a processor 42, and a computer program stored in the memory 40 and executable on the processor 42. When the computer program is executed by the processor, the steps in the above method embodiment are implemented.

[0122] Device Example 2

[0123] An embodiment of the present invention provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by the processor 42, the steps in the above method embodiment are implemented.

[0124] The computer-readable storage medium in this embodiment includes, but is not limited to, ROM, RAM, magnetic disk or optical disk, etc.

[0125] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements of the technical solutions of the embodiments of the present invention do not cause the essence of the corresponding technical solutions to deviate from the scope of this solution.

Claims

1. A method for detecting an underwater image object, characterized in that: include: S1, acquiring an underwater image, and preprocessing the underwater image; S2, inputting the preprocessed underwater image into a feature extraction network to extract features to obtain underwater image features; S3, sending the preprocessed underwater image to the detection model to obtain the mask of the target in the underwater image; S4, inputting the underwater image features and the mask of the target in the underwater image into a pre-trained generative adversarial model to obtain a pseudo underwater image; S5, sending the pseudo underwater image to the detection model to obtain the location information, mask and category of the target in the underwater image to complete target detection; The S4 specifically includes: Separate the mask of the target in the underwater image and the underwater image features to obtain the foreground and background area features; The foreground and background area features of the same batch of underwater images are randomly combined to obtain combined features, and the combined features are input into the convolutional network to generate new pseudo underwater images; The new pseudo underwater images generated by the generative model are distinguished from the underwater images, the authenticity of the input images is judged, and the pseudo images that conform to the distribution of the original data are obtained.

2. The method according to claim 1, characterized in that The S2 specifically includes: inputting the preprocessed underwater image into at least one of ResNet-FPN, MobileNet and ShuffleNet to perform feature extraction to obtain underwater image features.

3. The method according to claim 2, characterized in that The S3 specifically includes: The candidate region p is sent to the domain candidate network RPN to extract the candidate region p where the target may exist. i , and for the candidate region p i Perform preliminary category judgment and box regression to determine the region p i Contains the target, where i represents the i-th image; The candidate region p i The coordinates of the underwater image features are input into the ROI align layer to extract the region p i Features i,p ; Classification Network Utilization x i,p Calculate the area p containing the target i The category to which the target belongs is determined, and box regression is performed again to determine the location information of the target and generate a mask of the target.

4. An underwater image target detection system, characterized in that: include: Acquisition module: used for acquiring underwater images and preprocessing the underwater images; Extraction module: used to input the preprocessed underwater image into the feature extraction network for feature extraction to obtain underwater image features; Mask module: used to send the preprocessed underwater image to the detection model to obtain the mask of the target in the underwater image; A pseudo image module: used for inputting the underwater image features and the mask of the target in the underwater image into a pre-trained generative adversarial model to obtain a pseudo underwater image; Target detection module: used to send the pseudo underwater image into the detection model to obtain the location information, mask and category of the target in the underwater image to complete target detection; The pseudo image module is specifically used for: Separate the mask of the target in the underwater image and the underwater image features to obtain the foreground and background area features; The foreground and background area features of the same batch of underwater images are randomly combined to obtain combined features, and the combined features are input into the convolutional network to generate new pseudo underwater images; The generated new pseudo underwater image is distinguished from the underwater image, the authenticity of the input image is judged, and a pseudo image that conforms to the distribution of the original data is obtained.

5. The system according to claim 4, characterized in that: The extraction module is specifically used to extract features from at least one of the input ResNet-FPN, MobileNet and ShuffleNet to obtain underwater image features.

6. The system according to claim 5, characterized in that: The mask module is specifically used for: The candidate region p is sent to the domain candidate network RPN to extract the candidate region p where the target may exist. i , and for the candidate region p i Perform preliminary category judgment and box regression to determine the region p i Contains the target, where i represents the i-th image; The candidate region p i The coordinates of the underwater image features are input into the ROI align layer to extract the region p i Features i,p ; Classification Network Utilization x i,p Calculate the area p containing the target i The category to which the target belongs is determined, and box regression is performed again to determine the location information of the target and generate a mask of the target.

7. An underwater image target detection device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the underwater image target detection method according to any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by a processor, the steps of the underwater image target detection method according to any one of claims 1 to 3 are implemented.

Images