Target recognition method and device for underwater image, electronic equipment and storage medium
By converting land data into underwater images and performing pre-training and fine-tuning, the problem of insufficient underwater data is solved, improving the performance and detection capabilities of underwater vision tasks, especially for the detection and incremental learning of unknown objects in open environments.
Patent Information
- Application Number
- CN202310774209.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-27
- Publication Date
- 2026-01-23
- Estimated Expiration
- 2043-06-27
AI Technical Summary
Due to the unique nature of the underwater environment, underwater data acquisition is costly and effective underwater datasets are scarce, making underwater visual perception a challenging problem. Furthermore, using terrestrial data directly for pre-training has limited effectiveness and results in poor recognition performance.
The land dataset is converted into underwater images using an underwater image synthesis method. The synthesized underwater images are then used to pre-train the perception model. Finally, the model is fine-tuned using sample underwater images to construct an object detection model.
It improves the performance of underwater vision tasks, increases the model's perception capabilities underwater, reduces training time, and effectively detects unknown objects and has incremental learning capabilities in open underwater environments.
Smart Images
Figure CN116935202B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer vision, and in particular to a target recognition method and device for underwater images, an electronic device and a storage medium. BACKGROUND
[0002] In recent years, with the deepening of marine exploration, underwater robots are widely used in underwater detection and monitoring. As an important environmental perception tool, vision plays a crucial role in underwater operations. In the field of computer vision, most excellent machine learning algorithms require a large amount of data, especially in recent years, the deep learning method has emerged. However, due to the particularity of the underwater environment, the cost of underwater data collection is high, and there are few effective underwater data sets. Therefore, visual perception in underwater environment is still a challenging problem.
[0003] For the problem of lack of underwater data, pre-training and fine-tuning is an effective general solution paradigm. Specifically, the model is first pre-trained on a large-scale dataset, and then fine-tuned on a smaller, task-specific target dataset. Such models achieve state-of-the-art performance on many tasks, such as object detection, semantic segmentation and action recognition. Based on the above considerations, a large number of easily accessible terrestrial public datasets can be used for pre-training in underwater scenarios. However, there is a significant gap in the distribution of land data and underwater data in terms of color, lighting, instance category, etc. Therefore, the effect of directly using land data for pre-training is limited, resulting in poor recognition performance and poor recognition effect of the trained model. SUMMARY
[0004] To solve the problems in the prior art, the embodiments of the present application provide a target recognition method and device for underwater images, an electronic device and a storage medium.
[0005] The present application provides a target recognition method for underwater images, comprising:
[0006] inputting the collected underwater images into a target detection model to obtain a target recognition result output by the target detection model;
[0007] The target detection model is trained by the following method:
[0008] Based on an underwater image synthesis method, the sample land images in the sample land dataset are converted into synthesized underwater images; the sample land dataset includes the sample land images and first labels corresponding to the sample land images;
[0009] Based on the synthesized underwater images and the first labels, a perception model is pre-trained;
[0010] Fine-tune the pre-trained perception model based on the sample underwater domain dataset to obtain the target detection model.
[0011] In some embodiments, the method for synthesizing underwater images based on sample terrestrial images comprises:
[0012] Training the method for synthesizing underwater images based on the sample underwater domain dataset to determine a conversion function;
[0013] Converting the sample terrestrial images into synthesized underwater images based on the conversion function.
[0014] In some embodiments, the pre-training of the perception model based on the synthesized underwater images and the first labels comprises:
[0015] Inputting the synthesized underwater images into the perception model to obtain a first prediction result output by the perception model;
[0016] Determining a first loss function based on the first prediction result and the first labels;
[0017] Iteratively training the perception model based on the first loss function, and obtaining a pre-trained perception model when the first loss function is determined to be minimal.
[0018] In some embodiments, the expression of the parameters of the pre-trained perception model is as follows:
[0019]
[0020] wherein θ represents the parameters of the pre-trained perception model, n l represents the number of sample terrestrial images, represents a sample terrestrial image, represents a first label, SUG represents a conversion function, and M represents a perception model, represents a first loss function.
[0021] In some embodiments, the sample underwater domain dataset comprises sample underwater images and second labels corresponding to the sample underwater images;
[0022] The fine-tuning of the pre-trained perception model based on the sample underwater domain dataset to obtain the target detection model comprises:
[0023] Inputting the sample underwater images into the pre-trained perception model to obtain a second prediction result output by the pre-trained perception model;
[0024] determine a second loss function based on the second prediction result and the second label;
[0025] iteratively train the pre-trained perception model based on the second loss function, and obtain the target detection model when the second loss function is determined to be minimum.
[0026] In some embodiments, the expression of the parameters of the target detection model is as follows:
[0027]
[0028] wherein, θ * represents the parameters of the target detection model, n u represents the number of sample underwater images, represents a sample underwater image, represents a second label, and M' represents a pre-trained perception model, represents a second loss function.
[0029] The application also provides a target recognition device for underwater images, comprising:
[0030] a recognition module configured to input the collected underwater image into a target detection model to obtain a target recognition result output by the target detection model;
[0031] The target detection model is obtained by training in the following manner:
[0032] a sample land image in a sample land data set is converted into a synthesized underwater image based on an underwater image synthesis method; the sample land data set comprises the sample land image and a first label corresponding to the sample land image;
[0033] a perception model is pre-trained based on the synthesized underwater image and the first label;
[0034] The pre-trained perception model is fine-tuned based on a sample underwater data set to obtain the target detection model.
[0035] The application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the target recognition method for underwater images according to any one of the above embodiments when executing the program.
[0036] The application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the target recognition method for underwater images according to any one of the above embodiments.
[0037] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the target recognition method of the underwater image according to any one of the above.
[0038] The underwater image target recognition method, device, electronic equipment and storage medium provided by the application can convert a large number of terrestrial images into synthesized underwater images by using an underwater image synthesis method, thereby pre-training a perception model by using the synthesized underwater images, fine-tuning the perception model by using sample underwater images, obtaining a target detection model, and achieving accurate underwater image target recognition by using the target detection model, improving the performance of underwater visual tasks in the same environment, accelerating convergence and reducing training time. BRIEF DESCRIPTION OF DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.
[0040] Figure 1 is a flowchart of the underwater image target recognition method provided by the embodiment of the application;
[0041] Figure 2 is a schematic diagram of the pre-training method based on underwater synthesized data provided by the embodiment of the application;
[0042] Figure 3 is a structural schematic diagram of the underwater image target recognition device provided by the embodiment of the application;
[0043] Figure 4 is a structural schematic diagram of the electronic equipment provided by the embodiment of the application. DETAILED DESCRIPTION
[0044] In order to make the objects, technical solutions and advantages of the application clearer, the technical solutions in the application will be described clearly and completely below with reference to the drawings in the application. Obviously, the described embodiments are some embodiments of the application, but not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.
[0045] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a particular order or sequence. It should be understood that the terms thus used can be interchanged as appropriate, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second" are generally of a kind and are not limited in number, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates that the objects before and after are in an "or" relationship.
[0046] Due to the particularity of the underwater environment, the cost of underwater data collection is high, and effective underwater data sets are few. Therefore, visual perception of underwater environment is still a challenging problem.
[0047] For the problem of lack of underwater data, pre-training and fine-tuning is an effective general solution paradigm. Specifically, the model is first pre-trained on a large-scale dataset, and then fine-tuned on a smaller, task-specific target dataset. Such models achieve state-of-the-art performance on many tasks, such as object detection, semantic segmentation, and action recognition. Based on the above considerations, a large number of easily accessible terrestrial public datasets can be used for pre-training of underwater scenarios. However, there is a significant gap in distribution between terrestrial data and underwater data in terms of color, illumination, instance class, etc. Therefore, direct pre-training using terrestrial data has limited effect, which requires reducing the difference between the source domain and the target domain to improve the performance of the model in the target domain.
[0048] In addition, when exploring underwater with underwater robots, the real environment faced is an open environment. Open environment perception not only needs to detect known objects contained in the training set, but also needs to detect and incrementally learn unknown objects. Since unknown objects cannot be obtained in the training stage, it is difficult to find a projection space with the smallest difference between the two data distributions.
[0049] By comparing terrestrial data with underwater data, it is necessary to study how to use sufficient terrestrial domain datasets to improve the visual perception ability of a small amount of underwater domain samples. The problem is formally defined as follows:
[0050] Underwater domain is composed of underwater feature space and marginal distribution P u (X u ), that is where X u represents an underwater image dataset;
[0051] Terrestrial domain is composed of terrestrial feature space and marginal distribution P l (X l ) consists of, i.e. where X l represents the land image dataset;
[0052] underwater task consists of the underwater label space and decision function f u , i.e.
[0053] land task consists of the land label space and decision function f l , i.e.
[0054] The input in the underwater domain and the land domain are both RGB images, so their feature spaces are the same, but their marginal distributions are different, i.e.
[0055]
[0056] P u (X u )≠P l (X l )
[0057] The number N u of underwater data and the number N l of land data satisfy:
[0058] 0<N u <<N l
[0059] In addition, since the samples (cows, horses, etc.) in the land environment are different from the samples (starfish, sea urchins, etc.) in the underwater environment, each underwater category needs to have at least one labeled data to enable the model to obtain the ability to predict the underwater label space:
[0060]
[0061]
[0062] In the formula, represents the underwater data label set of the kth category.
[0063] In order to solve the problem of poor underwater visual perception performance caused by insufficient underwater samples, the data set in the land domain is used to assist the perception model in the underwater domain to improve the perception ability of the model in the underwater environment. That is, the optimization goal of the problem is:
[0064]
[0065] In the formula, represents a loss function, Y P represents a decision function f u a set of predicted labels, Y u a set of real labels of underwater data, f l is a decision function trained with a land data set.
[0066] When exploring underwater with an underwater robot, the real environment faced is an open environment. The open world object detection problem is taken as the research object of open environment perception for analysis.
[0067] The open world object detection task of unknown class separable contains three parts:
[0068] (1) The detector M of detecting known class objects O known and unknown class objects O unk ;
[0069] (2) The labeling process L labels the unknown data at time t , and expands the known class set
[0070] (3) The incremental learning process I expands the new class objects O new learned by labeling, and avoids forgetting the old class objects O old learned before.
[0071] Among them, the labeling process relies on manual marking to obtain labels, so only the training of the auxiliary optimization detector and the incremental learning process are needed. Therefore, the optimization goal of improving the visual perception ability of the underwater open environment by using land data is:
[0072]
[0073] Among them, the known class objects O known are divided into old class objects O old and new class objects O new , M u represents the detector of the underwater domain, M l represents the detector of the land domain, UC-mAP is used to evaluate the model's detection of unknown classes, mAP represents the performance index of the model's detection of known classes, and λ1, λ2, λ3 are weight factors.
[0074] Among them, the first term represents the detection ability of the model to unknown objects; the second term represents the detection ability to old classes, that is, the ability to reduce the forgetting of old knowledge by the model incremental learning; and the third term represents the detection ability to new classes, that is, the ability of the model to expand new knowledge.
[0075] Through the above analysis of the open environment perception, when using the land dataset to assist the underwater domain, the detection of unknown underwater objects and the incremental learning ability of the model need to be considered. In order to achieve this goal, the two domains need to be as identical as possible.
[0076] Assuming that there is a transformation function φ(·) that can convert the land domain dataset into an underwater distribution, that is:
[0077] P l (φ(X l ))=P u (X u )
[0078] Since the unknown object X unk and the data required for incremental learning X new come from the underwater distribution, by analogy, it satisfies:
[0079] P l (φ(X l ))=P u (X unk )=P u (X new )
[0080] Using φ(·) to transform the land domain dataset can make the model not only obtain more foreground data to improve the extraction ability of foreground features, but also maintain the perception ability of the underwater domain, so as to consider the detection of unknown underwater objects and underwater incremental learning.
[0081] The optimization goal of the present application is to minimize the distribution of the land domain and the distribution of the underwater domain, that is:
[0082]
[0083] In the formula, represents the difference loss between the two distributions.
[0084] According to the above analysis, the embodiment of the present application constructs a pre-training paradigm for improving underwater visual perception ability using land data, that is, an underwater domain pre-training (UDP) scheme based on synthetic data enhancement, which mainly includes pre-training and fine-tuning stages. Pre-training and fine-tuning have always been the main paradigm for training deep neural networks in computer vision, and the pre-training weights learned on large labeled datasets are usually used to initialize downstream tasks.
[0085] Figure 1 is a flowchart of the target recognition method for underwater images provided by the embodiment of the present application, as shown in Figure 1 The target recognition method for underwater images provided by the embodiment of the present application comprises:
[0086] Step 101, input the collected underwater image into a target detection model to obtain a target recognition result output by the target detection model;
[0087] The target detection model is obtained by training in the following manner:
[0088] Based on an underwater image synthesis method, a sample land image in a sample land domain data set is converted into a synthesized underwater image; the sample land domain data set includes the sample land image and a first label corresponding to the sample land image;
[0089] Based on the synthesized underwater image and the first label, a perception model is pre-trained;
[0090] Based on a sample underwater domain data set, the pre-trained perception model is fine-tuned to obtain the target detection model.
[0091] It should be noted that the execution subject of the target recognition method for the underwater image provided by the present application can be an electronic device, a component in the electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Illustratively, the mobile electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc., and the present application is not limited in this regard.
[0092] In step 101, the collected underwater image is input into a target detection model to obtain a target recognition result output by the target detection model;
[0093] The target detection model is obtained by training in the following manner:
[0094] Based on an underwater image synthesis method, a sample land image in a sample land domain data set is converted into a synthesized underwater image; the sample land domain data set includes the sample land image and a first label corresponding to the sample land image;
[0095] Based on the synthesized underwater image and the first label, a perception model is pre-trained;
[0096] Fine-tune the pre-trained perception model based on the sample underwater domain data set to obtain the target detection model.
[0097] The collected underwater image is the image data to be recognized, and the collected underwater image is input into the target detection model to obtain the recognition result of the underwater image.
[0098] Before performing underwater target recognition, the perception model needs to be trained to obtain the target detection model. The perception model can be Faster R-CNN, Grid R-CNN, SSD, YOLOv3, Deformable DETR or YOLOX, etc.
[0099] First, the sample data set needs to be obtained, which includes a sample land domain data set and a sample underwater domain data set.
[0100] The sample land domain data set includes sample land images and first labels corresponding to the sample land images. The first labels include object position information and class information in the target detection task.
[0101] The sample underwater domain data set includes sample underwater images and second labels corresponding to the sample underwater images. The second labels include object position information and class information in the target detection task.
[0102] The sample land domain data set has a large amount of data and is easy to obtain, but there is a significant gap in color, illumination, instance category, etc. between land data and underwater data. Therefore, direct pre-training using land data has limited effect, so it is necessary to reduce the difference between the source domain and the target domain to improve the performance of the model in the target domain.
[0103] Specifically, the underwater image synthesis method can be used to convert the sample land images in the sample land domain data set into synthesized underwater images, and then the synthesized underwater images and the first labels are used to pre-train the perception model to obtain the pre-trained perception model.
[0104] And the sample underwater domain data set is used to fine-tune the pre-trained perception model to obtain the target detection model.
[0105] The underwater image target recognition method provided by the embodiment of the application converts a large number of land images into synthesized underwater images by using an underwater image synthesis method, thereby pre-training the perception model using the synthesized underwater images, and fine-tuning the perception model using the sample underwater images to obtain the target detection model. Through the target detection model, accurate underwater image target recognition can be achieved, the performance of underwater visual tasks in the same environment is improved, convergence is accelerated, and training time is reduced.
[0106] In some embodiments, the method for synthesizing underwater images based on the sample land image in the sample land domain data set comprises:
[0107] The method for synthesizing underwater images is trained based on the sample underwater domain data set to determine a conversion function;
[0108] The sample land image is converted into a synthesized underwater image based on the conversion function.
[0109] First, the method for synthesizing underwater images is self-supervised trained using the sample underwater domain data set to obtain a conversion function φ(·):
[0110] φ(x)=SUG(x)
[0111] In the formula, SUG(·) represents the self-supervised trained method for synthesizing underwater images.
[0112] The conversion function obtained by the method for synthesizing underwater images can eliminate the domain difference between the underwater data set and the land data set, and convert the land image into an underwater image in the underwater style, that is, a large synthesized underwater data set can be generated using a large amount of easily obtained land domain data set for pre-training.
[0113] The method for identifying underwater image targets provided by the embodiment of the application trains the method for synthesizing underwater images using the sample underwater domain data set to determine the conversion function, so that the sample land image is converted into a synthesized underwater image using the conversion function, the domain difference between the underwater data set and the land data set can be eliminated, and a large training data is provided for pre-training of the perception model.
[0114] In some embodiments, the method for pre-training the perception model based on the synthesized underwater image and the first label comprises:
[0115] The synthesized underwater image is input into the perception model to obtain a first prediction result output by the perception model;
[0116] A first loss function is determined based on the first prediction result and the first label;
[0117] The perception model is iteratively trained based on the first loss function, and the pre-trained perception model is obtained when the first loss function is determined to be minimum.
[0118] In some embodiments, the expression of the parameters of the pre-trained perception model is as follows:
[0119]
[0120] Wherein, θ represents the parameters of the pre-trained perception model, nl denotes the number of sample land images, denotes a sample land image, denotes a first label, SUG denotes a conversion function, and M denotes a perception model, denotes a first loss function.
[0121] The synthesized underwater image is input into the perception model to obtain a first prediction result output by the perception model, and a first loss function is constructed according to a difference between the first prediction result and the first label.
[0122] The first loss function is optimized by iteratively training the perception model, so that the perception model is pre-trained under the condition that the first loss function is minimized.
[0123] Specifically, the perception model M is pre-trained using the synthesized underwater image and the corresponding first label, and the expression of the parameter θ of the pre-trained perception model is as follows:
[0124]
[0125] wherein θ denotes the parameter of the pre-trained perception model, n l denotes the number of sample land images, denotes a sample land image, denotes a first label, SUG denotes a conversion function, and M denotes a perception model, denotes a first loss function.
[0126] The target recognition method for underwater images provided by the embodiment of the application pre-trains a perception model using a large number of synthesized underwater images to obtain a pre-trained perception model, thereby improving the recognition accuracy of the perception model.
[0127] In some embodiments, the sample underwater domain data set includes sample underwater images and second labels corresponding to the sample underwater images;
[0128] The pre-trained perception model is fine-tuned based on the sample underwater domain data set to obtain the target detection model, including:
[0129] The sample underwater image is input into the pre-trained perception model to obtain a second prediction result output by the pre-trained perception model;
[0130] A second loss function is determined based on the second prediction result and the second label;
[0131] The pre-trained perception model is iteratively trained based on the second loss function, and the target detection model is obtained under the condition that the second loss function is minimized.
[0132] In some embodiments, the expression of the parameters of the target detection model is as follows:
[0133]
[0134] wherein θ * represents the parameters of the target detection model, n u represents the number of sample underwater images, represents a sample underwater image, represents a second label, M' represents a pre-trained perception model, represents a second loss function.
[0135] After obtaining the pre-trained perception model using the synthesized underwater images, the pre-trained perception model can be fine-tuned using real underwater images, i.e., sample underwater images in the sample underwater domain dataset, to improve the recognition accuracy of the perception model.
[0136] The sample underwater images are input into the pre-trained perception model to obtain a second prediction result output by the pre-trained perception model. A second loss function is constructed according to the difference between the second prediction result and the second label.
[0137] The pre-trained perception model is iteratively trained to optimize the second loss function, so that the fine-tuned perception model, i.e., the final target detection model, is obtained under the condition that the second loss function is minimized.
[0138] Specifically, for the perception model M' with pre-training parameters θ, the pre-trained perception model M' is fine-tuned using sample underwater images and corresponding second labels, and the optimal parameters are found by minimizing the loss function, and the parameters θ * of the target detection model are as follows:
[0139]
[0140] wherein θ * represents the parameters of the target detection model, n u represents the number of sample underwater images, represents a sample underwater image, represents a second label, M' represents a pre-trained perception model, represents a second loss function.
[0141] The target recognition method for underwater images provided by the embodiments of the present application fine-tunes the pre-trained perception model using sample underwater images to obtain a target detection model, further improving the recognition accuracy of the target detection model.
[0142] Figure 2is a schematic diagram of a pre-training method based on underwater synthetic data provided by an embodiment of the present application, as shown Figure 2 During pre-training, the model iteratively adjusts its parameters to minimize the loss function and obtains a set of optimal model parameters. Thanks to the transformation function, the pre-trained model learns the knowledge of underwater distribution. In order to fine-tune the real underwater image, directly use this pre-trained model and further adjust its parameters to obtain the fine-tuned parameters * , so that the model obtains the ability to identify underwater samples.
[0143] The specific algorithm is as follows:
[0144] Input: land domain data set
[0145] Input: underwater domain data set
[0146] Output: trained model parameters *
[0147]
[0148]
[0149] The embodiments of the present application verify the method on various underwater visual perception tasks and various models, and visualize the distribution of the land data set transformed into underwater distribution and the real underwater data set.
[0150] The embodiments of the present application verify the method on underwater target detection and underwater open world target detection tasks using the URPC2018 data set. Based on the URPC2018 data set, an underwater open world target detection data set is constructed. The data set is divided into two tasks, and each task contains 2 new categories.
[0151] As shown in Table 1, the new categories in task 1 are “holothurian” and “echinus”, and the new categories in task 2 are “scallop” and “starfish”. Consistent with the foregoing, the known categories in task 2 include new categories and categories in task 1. The test set in task 1 and task 2 directly uses the test set of URPC2018.
[0152] Table 1 Underwater Open World Target Detection Data Set
[0153]
[0154] Six target detectors are used to verify the proposed method in underwater target detection tasks, which are Faster R-CNN, Grid R-CNN, SSD, YOLOv3, Deformable DETR and YOLOX.
[0155] Among them, Faster R-CNN, Grid R-CNN and Deformable DETR are two-stage detectors, and the rest are single-stage detectors. The first four are anchor-based, and the last two are anchor-free.
[0156] Specifically, Faster RCNN is a typical representative of two-stage target detection models and the first fully differentiable model. The first stage generates regions of interest, and the second stage is used to determine target classes and perform frame regression.
[0157] Grid R-CNN improves Faster R-CNN by changing the frame regression in the second stage to use key point-based prediction to determine the position of the target frame.
[0158] SSD completes the classification and positioning of the target in a single-stage manner, significantly improving the real-time performance of the detection.
[0159] YOLOv3 improves the detection speed and is as accurate as SSD, but three times faster.
[0160] DETR applies transformer to target detection, replacing the current manually designed model and achieving good results.
[0161] Deformable DETR improves the slow convergence speed and limited feature space resolution in DETR.
[0162] YOLOX is an anchor-free detector used to improve the performance of the detector in the presence of significant differences in object size.
[0163] All detectors are based on MMDetection.
[0164] Baseline is a detector trained and tested directly on the underwater dataset.
[0165] VOCP refers to pre-training with the Pascal VOC dataset and fine-tuning with the underwater dataset.
[0166] UDP is pre-trained after converting the Pascal VOC dataset to an underwater distribution. All training and testing are performed on 4 NVIDIA GeForce RTX 3090 GPUs.
[0167] Table 2 shows the best detection results of six detectors under three training strategies. After pre-training, all methods can be improved, especially for small objects, for example, the mAP of Deformable DETR increases from 18.1% to 21.3%. s from 18.1% to 21.3%.
[0168] Since there are more small objects underwater, improving the performance of small objects can effectively improve the overall performance, so the mAP of this method is significantly improved, and it can be improved by 10.4% by pre-training using the Pascal VOC dataset. Moreover, using the UDP strategy, the mAP is further improved to 39.6%. The experiments of YOLOv3 show that using the underwater image synthesis method for pre-training is more effective than the method based on the land dataset.
[0169] Table 2 Quantitative results of underwater target detection
[0170]
[0171]
[0172] In order to qualitatively analyze the influence of UDP on underwater target detection, the detection results of YOLOX with the highest quantitative score are visualized.
[0173] Through experiments, it is verified that Baseline and VOCP will miss detection. Baseline will mis-detect the background as ``holothurian". Compared with other algorithms, UDP has the best effect in underwater target detection, and can realize correct detection of most objects, and only miss one small object ``echinus". The positioning of the model under the UDP strategy is better. These evaluations can verify the effectiveness of the proposed method for improving the situation of underwater detection.
[0174] In order to explore the effect of UDP in the underwater open environment, the embodiment of the application uses UC-Det as the detector to compare the underwater open environment detection performance under three pre-training strategies. Since the unknown class has only two categories, the number is small, so the UC-Det without adding the UCR module is used.
[0175] As shown in Table 3, UDP has obvious promotion on UC-mAP and UC-Recall of unknown classes, which shows that more unknown objects are detected. UDP performs poorly on WI and A-OSE, which shows that when more foreground objects (known classes and unknown classes) are detected, unknown objects also have a greater probability of being detected as known objects. By comparing the detection indicators of unknown objects, it can be seen that VOCP and Baseline have poor foreground detection ability. For the detection of known objects, UDP can improve the mAP index by 8.06%. From the results in task 2, it can be seen that UDP can greatly improve the catastrophic forgetting of old classes in the incremental learning process.
[0176] Table 3 Quantitative results of underwater open world object detection
[0177]
[0178]
[0179] Through experiments, in task 1, Baseline and VOCP are consistent with the results of quantitative analysis, and have poor foreground detection ability and cannot effectively detect unknown objects. Under the same setting, Baseline also has poor detection ability for known objects. However, UDP also misdetected a known object as an unknown class. In task 2, in addition to the obvious missing detection phenomenon of Baseline, VOCP and UDP both realize incremental learning and can detect new classes and old classes.
[0180] In view of the poor performance of the visual model in the underwater open environment, the application proposes an underwater domain pre-training method based on underwater synthetic data enhancement.
[0181] First, from the perspective of transfer learning, the characteristics of the open world object detection task are analyzed, and it is concluded that the distribution of the source domain data should be migrated to the target domain to improve the detection performance of known objects and unknown objects.
[0182] Secondly, based on the above analysis, the underwater image synthesis method is used as a conversion function to convert terrestrial data into underwater distribution, so as to reduce the distribution difference between the source domain data and the target domain data.
[0183] Finally, a scheme for underwater domain pre-training using terrestrial data sets is proposed, which further improves the recognition performance of underwater visual tasks. The experimental results on different underwater visual perception tasks and different perception models show that the proposed method can effectively improve the performance of visual perception models. In addition, in the underwater open world object detection task, the proposed UDP can effectively improve the ability of the model to detect unknown objects and known objects and incremental learning.
[0184] The target recognition method for underwater images provided by the embodiment of the present application can improve the performance of underwater visual tasks in the same environment, accelerate convergence and reduce training time. The detection performance of known objects and unknown objects in the underwater open world target detection task is greatly improved, and the catastrophic forgetting after incremental learning is greatly alleviated.
[0185] The target recognition device for underwater images provided by the present application is described below. The target recognition device for underwater images described below can be mutually corresponding with the target recognition method for underwater images described above.
[0186] Figure 3 is a structural schematic diagram of the target recognition device for underwater images provided by the embodiment of the present application, as Figure 3 indicated, the target recognition device for underwater images provided by the embodiment of the present application comprises:
[0187] The recognition module 310 is configured to input the collected underwater image into the target detection model to obtain a target recognition result output by the target detection model.
[0188] The target detection model is obtained by training in the following manner:
[0189] Based on the underwater image synthesis method, the sample land image in the sample land data set is converted into a synthesized underwater image; the sample land data set comprises the sample land image and a first label corresponding to the sample land image.
[0190] Based on the synthesized underwater image and the first label, the perception model is pre-trained.
[0191] Based on the sample underwater domain data set, the pre-trained perception model is fine-tuned to obtain the target detection model.
[0192] It should be noted that the target recognition device for underwater images provided by the embodiment of the present application can realize all the method steps realized by the target recognition method for underwater images provided by the above embodiment, and can achieve the same technical effects. The same parts and beneficial effects in this embodiment as the method embodiment will not be described in detail.
[0193] Optionally, based on the underwater image synthesis method, the sample land image in the sample land data set is converted into a synthesized underwater image, comprising:
[0194] Based on the sample underwater domain data set, the underwater image synthesis method is trained to determine a conversion function.
[0195] Based on the conversion function, the sample land image is converted into a synthesized underwater image.
[0196] Optionally, the pre-training of the perception model based on the synthesized underwater image and the first label comprises:
[0197] inputting the synthesized underwater image into the perception model to obtain a first prediction result output by the perception model;
[0198] determining a first loss function based on the first prediction result and the first label;
[0199] iteratively training the perception model based on the first loss function, and obtaining a pre-trained perception model in a case where the first loss function is determined to be minimum.
[0200] Optionally, an expression of a parameter of the pre-trained perception model is as follows:
[0201]
[0202] wherein θ represents the parameter of the pre-trained perception model, n l represents a number of sample land images, represents a sample land image, represents the first label, SUG represents a conversion function, and M represents the perception model, represents the first loss function.
[0203] The sample underwater domain data set comprises a sample underwater image and a second label corresponding to the sample underwater image;
[0204] The fine-tuning of the pre-trained perception model based on the sample underwater domain data set to obtain the target detection model comprises:
[0205] inputting the sample underwater image into the pre-trained perception model to obtain a second prediction result output by the pre-trained perception model;
[0206] determining a second loss function based on the second prediction result and the second label;
[0207] iteratively training the pre-trained perception model based on the second loss function, and obtaining the target detection model in a case where the second loss function is determined to be minimum.
[0208] Optionally, an expression of a parameter of the target detection model is as follows:
[0209]
[0210] wherein θ * represents the parameter of the target detection model, n u represents a number of sample underwater images, representing a sample underwater image, representing a second label, M' represents a pre-trained perception model, representing a second loss function.
[0211] Figure 4 An example of an entity structure diagram of an electronic device is shown as Figure 4 The electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 complete mutual communication through the communications bus 440. The processor 410 can invoke a logical instruction in the memory 430 to execute an underwater image target recognition method, which includes inputting an acquired underwater image into a target detection model to obtain a target recognition result output by the target detection model; the target detection model is obtained by training in the following manner: based on an underwater image synthesis method, converting a sample land image in a sample land data set into a synthesized underwater image; the sample land data set includes the sample land image and a first label corresponding to the sample land image; based on the synthesized underwater image and the first label, a perception model is pre-trained; based on a sample underwater data set, the pre-trained perception model is fine-tuned to obtain the target detection model.
[0212] In addition, the logical instructions in the memory 430 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.
[0213] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program being stored in a non-transitory computer-readable storage medium, and the computer program being executable by a processor to enable a computer to perform the target recognition method of the underwater image provided by the above method, the method comprising: inputting an acquired underwater image into a target detection model to obtain a target recognition result output by the target detection model; and training the target detection model by: converting a sample land image in a sample land domain data set into a synthesized underwater image based on an underwater image synthesis method, wherein the sample land domain data set comprises the sample land image and a first label corresponding to the sample land image; pre-training a perception model based on the synthesized underwater image and the first label; and fine-tuning the pre-trained perception model based on a sample underwater domain data set to obtain the target detection model.
[0214] In another aspect, the present application also provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executable by a processor to implement the target recognition method of the underwater image provided by the above method, the method comprising: inputting an acquired underwater image into a target detection model to obtain a target recognition result output by the target detection model; and training the target detection model by: converting a sample land image in a sample land domain data set into a synthesized underwater image based on an underwater image synthesis method, wherein the sample land domain data set comprises the sample land image and a first label corresponding to the sample land image; pre-training a perception model based on the synthesized underwater image and the first label; and fine-tuning the pre-trained perception model based on a sample underwater domain data set to obtain the target detection model.
[0215] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement it without creative labor.
[0216] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0217] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A target recognition method for underwater images, characterized in that, include: The acquired underwater images are input into the target detection model to obtain the target recognition result output by the target detection model; The target detection model is trained in the following way: Based on an underwater image synthesis method, sample land images in a sample land domain dataset are converted into synthesized underwater images; the sample land domain dataset includes the sample land images and the first labels corresponding to the sample land images. Based on the synthesized underwater image and the first label, the perception model is pre-trained. The target detection model is obtained by fine-tuning the pre-trained perception model based on the sample underwater domain dataset. The pre-training of the perception model based on the synthesized underwater image and the first label includes: The synthesized underwater image is input into the perception model to obtain the first prediction result output by the perception model; Based on the first prediction result and the first label, a first loss function is determined; The perception model is iteratively trained based on the first loss function, and a pre-trained perception model is obtained when the first loss function is minimized. The parameters of the pre-trained perception model are expressed as follows: Where θ represents the parameters of the pre-trained perception model, and n l Indicates the number of sample land images. Represents sample land images, The first label is represented by SUG, the transition function by M, and the perceptual model by M. Represents the first loss function; The sample underwater domain dataset includes sample underwater images and second labels corresponding to the sample underwater images; The target detection model is obtained by fine-tuning the pre-trained perception model based on the sample underwater domain dataset, including: The underwater image of the sample is input into the pre-trained perception model to obtain the second prediction result output by the pre-trained perception model; Based on the second prediction result and the second label, determine the second loss function; The pre-trained perception model is iteratively trained based on the second loss function, and the target detection model is obtained when the second loss function is minimized. The expressions for the parameters of the target detection model are as follows: Where, θ * n represents the parameters of the object detection model. u This indicates the number of underwater images in the sample. Indicates underwater images of the sample. M' represents the second label, and M′ represents the pre-trained perceptual model. This represents the second loss function; The first and second labels include object location information and category information in the target detection task.
2. The underwater image target recognition method according to claim 1, characterized in that, The underwater image synthesis method converts sample land images from a sample land domain dataset into synthesized underwater images, including: The underwater image synthesis method is trained based on the aforementioned underwater domain dataset to determine the transformation function; Based on the transformation function, the sample land image is converted into a synthetic underwater image.
3. A target recognition device for underwater images, characterized in that, include: The recognition module is used to input the acquired underwater images into the target detection model and obtain the target recognition result output by the target detection model. The target detection model is trained in the following way: Based on an underwater image synthesis method, sample land images in a sample land domain dataset are converted into synthesized underwater images; the sample land domain dataset includes the sample land images and the first labels corresponding to the sample land images. Based on the synthesized underwater image and the first label, the perception model is pre-trained. The target detection model is obtained by fine-tuning the pre-trained perception model based on the sample underwater domain dataset. The identification module is specifically used for: The synthesized underwater image is input into the perception model to obtain the first prediction result output by the perception model; Based on the first prediction result and the first label, a first loss function is determined; The perception model is iteratively trained based on the first loss function, and a pre-trained perception model is obtained when the first loss function is minimized. The parameters of the pre-trained perception model are expressed as follows: Where θ represents the parameters of the pre-trained perception model, and n l Indicates the number of sample land images. Represents sample land images, The first label is represented by SUG, the transition function by M, and the perceptual model by M. Represents the first loss function; The underwater domain dataset includes underwater images of the samples and second labels corresponding to the underwater images; the recognition module is specifically used for: The underwater image of the sample is input into the pre-trained perception model to obtain the second prediction result output by the pre-trained perception model; Based on the second prediction result and the second label, determine the second loss function; The pre-trained perception model is iteratively trained based on the second loss function, and the target detection model is obtained when the second loss function is minimized. The expressions for the parameters of the target detection model are as follows: Where, θ * n represents the parameters of the object detection model. u This indicates the number of underwater images in the sample. Indicates underwater images of the sample. M' represents the second label, and M′ represents the pre-trained perceptual model. This represents the second loss function; The first and second labels include object location information and category information in the target detection task.
4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the underwater image target recognition method as described in any one of claims 1 or 2.
5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the underwater image target recognition method as described in any one of claims 1 or 2.
6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the underwater image target recognition method as described in any one of claims 1 or 2.