Target detection method, system and equipment based on virtual and real data and medium

Abstract

The invention discloses a target detection method, system and device based on virtual and real data and a medium, and relates to the field of target detection, and the method comprises the steps: carrying out the target detection of a sea scene image through a target detection model, and obtaining the prediction types of a plurality of sea surface targets and a plurality of candidate boxes; the target detection model is obtained by supervising and training a student model through a feature map obtained by a teacher model, the teacher model is a trained neural network model, and the student model is obtained by performing knowledge distillation on the feature map of the teacher model through a multi-valued mask; and adopting a non-maximum suppression algorithm to screen the plurality of candidate frames according to the prediction categories to obtain a prediction frame of the sea surface target, and taking the prediction category of the sea surface target corresponding to the prediction frame as the category of the sea surface target. According to the invention, the speed and accuracy of target detection in a complex marine environment can be improved.

Description

Technical Field

[0001] The present application relates to the field of target detection, and in particular to a target detection method, system, device and medium based on virtual and real data. Background Art

[0002] In the prior art, in order to build a target detection model with superior performance and high detection accuracy in a real ocean scene, it is often necessary to stack parameters and design a complex neural network structure. Due to the increase in the width and depth of the neural network structure, the complexity and parameter redundancy of the target detection model are also greatly increased, resulting in the marine target detection model being too cumbersome and difficult to deploy on low-resource and low-power devices such as unmanned ships and drones. In addition, due to the increase in model calculation amount and training time, the real-time performance of target detection is reduced, making it difficult to meet the real-time performance requirements of the intelligent unmanned system cluster for target perception. Therefore, the target detection model in the ocean scene has the problems of complex network structure and slow detection speed. Summary of the invention

[0003] The purpose of this application is to provide a method, system, device and medium for target detection based on virtual and real data, which can improve the speed and accuracy of target detection in complex marine environments. To achieve the above purpose, this application provides the following solutions:

[0004] In a first aspect, the present application provides a method for detecting a target based on virtual and real data, comprising:

[0005] The target detection model is used to perform target detection on the ocean scene image to obtain predicted categories of multiple sea surface targets and multiple candidate boxes; the target detection model is obtained by supervised training of the student model through the feature map obtained by the teacher model, the teacher model is a trained neural network model, and the student model is obtained by performing knowledge distillation on the feature map of the teacher model through a multi-value mask;

[0006] A non-maximum suppression algorithm is used to screen the multiple candidate boxes according to the prediction category to obtain a prediction box of the sea surface target, and the prediction category of the sea surface target corresponding to the prediction box is used as the category of the sea surface target; wherein the prediction box represents the location information of the sea surface target, and the categories of the sea surface targets include cruise ships, container ships, bulk carriers, sailboats and islands and reefs.

[0007] Furthermore, the training process of the target detection model specifically includes:

[0008] Constructing a training set, wherein the training set includes a plurality of training data, wherein the training data includes ocean scene sample pictures and labels of the ocean scene sample pictures; the labels include categories of sea surface targets in the ocean scene sample pictures and location information of the sea surface targets; the ocean scene pictures include real ocean scene sample pictures and virtual ocean scene sample pictures;

[0009] Inputting the training data into the teacher model to obtain a feature map of the teacher model; inputting the training data into the student model to obtain a feature map of the student model;

[0010] Determine a loss function based on the feature graph of the teacher model and the feature graph of the student model;

[0011] The loss function is used to update the parameters of the student model to obtain a trained student model, and the trained student model is used as the target detection model.

[0012] Further, the loss function includes feature distillation loss and classification distillation loss;

[0013] Determining a loss function according to the feature graph of the teacher model and the feature graph of the student model specifically includes:

[0014] Determining the feature distillation loss according to the feature map of the teacher model, the feature map of the student model, and the multi-value mask;

[0015] Determining the cosine similarity loss according to the first candidate box in the feature map of the student model and the second candidate box in the feature map of the teacher model; determining the cross entropy loss according to the prediction result of the category of the sea surface target in the first candidate box and the prediction result of the category of the sea surface target in the second candidate box;

[0016] The classification distillation loss is determined based on the cross entropy loss and the cosine similarity loss.

[0017] Furthermore, the expression of the characteristic distillation loss is:

[0018]

[0019] Among them, L FPN is the feature distillation loss, H and W represent the height and width of the feature map respectively, and M w，h is a multi-valued mask of W×H. h，w and T h，w are the feature maps of the student model and the teacher model, φ adapt is the feature adaptation function.

[0020] Furthermore, the expression of the classification distillation loss is:

[0021]

[0022] Among them, L cls is the classification distillation loss, L cosine is the cosine similarity loss, α and β are hyperparameters for balancing different distillation losses. The teacher model and the student model generate k candidate boxes for a training data respectively. For the jth candidate box, the jth candidate box in the teacher model and the student model is converted into a feature vector through convolution operation. and Use linear transformation to obtain the prediction results of the category of the sea surface target in the jth candidate frame in the teacher model and the student model respectively and

[0023] Furthermore, the generation process of the multi-value mask includes:

[0024] Obtain a feature map of the student model, wherein the feature map includes multiple candidate boxes.

[0025] Calculate the distance intersection-and-union ratio between each candidate frame and the real frame to obtain multiple distance intersection-and-union ratio values.

[0026] Add the distance intersection and union ratios greater than the first threshold to obtain a mask;

[0027] The mask is normalized and mapped to a preset range to obtain the multi-value mask.

[0028] Further, the non-maximum suppression algorithm is used to screen the plurality of candidate frames according to the prediction category to obtain a prediction frame of the sea surface target, and the prediction category of the sea surface target corresponding to the prediction frame is used as the category of the sea surface target, including:

[0029] Sorting the plurality of candidate boxes according to the confidence levels of the candidate boxes to obtain a first list;

[0030] According to the size of the confidence, select the candidate boxes corresponding to the confidence from large to small as the reference candidate boxes;

[0031] Traversing the first list, for each reference candidate box, calculating an intersection-and-union ratio between the reference candidate box and all candidate boxes in the first list except the reference candidate box, to obtain a plurality of intersection-and-union ratios;

[0032] Determine whether the intersection-and-union ratio satisfies a repetition condition, and if so, delete the candidate box corresponding to the intersection-and-union ratio to obtain a second list; the second list includes multiple candidate boxes; the repetition condition is that the intersection-and-union ratio is greater than a second threshold;

[0033] Classifying the candidate boxes in the second list according to the predicted category of the sea surface target to obtain a classification result, wherein each predicted category in the classification result includes at least one candidate box and the confidence of the candidate box;

[0034] The candidate frame with the highest confidence is used as the predicted frame of the sea surface target, and the predicted category of the sea surface target corresponding to the candidate frame with the highest confidence is used as the category of the sea surface target.

[0035] In a second aspect, the present application provides a target detection system based on virtual and real data, and the target detection system based on virtual and real data includes:

[0036] A detection module is used to perform target detection on an ocean scene image using a target detection model to obtain predicted categories of multiple sea surface targets and multiple candidate boxes; the target detection model is obtained by supervising and training a student model using a feature map obtained by a teacher model, the teacher model is a trained neural network model, and the student model is obtained by performing knowledge distillation on the feature map of the teacher model using a multi-value mask;

[0037] The detection result output module is used to use a non-maximum suppression algorithm to filter the multiple candidate boxes according to the predicted category to obtain a predicted box of a sea surface target, and use the predicted category of the sea surface target corresponding to the predicted box as the category of the sea surface target; wherein the predicted box represents the location information of the sea surface target, and the categories of the sea surface targets include cruise ships, container ships, bulk carriers, sailboats and islands and reefs.

[0038] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-mentioned methods for target detection based on virtual and real data.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the above-mentioned methods for target detection based on virtual and real data.

[0040] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0041] The present application provides a method, system, device and medium for target detection based on virtual and real data. A trained neural network model is used as a teacher model, and a multi-value mask is used to perform knowledge distillation on the feature map of the teacher model to obtain a student model, thereby simplifying the network structure of the student model and improving the calculation speed of the target detection model. At the same time, the feature map obtained by the teacher model is used to supervise the student model for training, which can improve the detection accuracy of the student model for target detection. Further, a non-maximum suppression algorithm is used to obtain multiple candidate frames from the student model and the candidate frames are screened according to the prediction category. The screened prediction frame and the prediction category corresponding to the prediction are used as the prediction frame and category of the sea surface target, thereby optimizing the detection result of the sea surface target again. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 This is an application environment diagram of a target detection method based on virtual and real data in one embodiment of the present application;

[0044] Figure 2 This is a flow chart of a target detection method based on virtual and real data in one embodiment of the present application;

[0045] Figure 3 A flowchart of a training process of a target detection model provided in one embodiment of the present application;

[0046] Figure 4 A flowchart of a process for generating a multi-value mask provided in one embodiment of the present application;

[0047] Figure 5 A schematic diagram of functional modules of a target detection system based on virtual and real data provided by an embodiment of the present application; Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0050] The target detection method based on virtual and real data provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the ocean scene picture to be processed to the server 104. After the server 104 receives the ocean scene picture to be processed, for the ocean scene picture to be processed, the server 104 uses the target detection model to perform target detection on the ocean scene picture, obtains multiple predicted categories and multiple candidate frames of sea surface targets, and uses the non-maximum suppression algorithm to screen the multiple candidate frames according to the predicted category to obtain the predicted frame of the sea surface target, and uses the predicted category of the sea surface target corresponding to the predicted frame as the category of the sea surface target. The server 104 can feedback the obtained category and prediction frame of the sea surface target to the terminal 102. In addition, in some embodiments, the target detection method for virtual and real data can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly perform target detection processing on the ocean scene picture to be processed, or the server 104 can obtain the ocean scene picture to be processed from the data storage system and perform target detection processing on the ocean scene picture to be processed.

[0051] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0052] In an exemplary embodiment, Figure 2 As shown, the present application provides a target detection method based on virtual and real data, comprising the following steps:

[0053] S1. Use a target detection model to perform target detection on an ocean scene image to obtain predicted categories of multiple sea surface targets and multiple candidate boxes; the target detection model is obtained by supervised training of a student model through a feature map obtained by a teacher model, the teacher model is a trained neural network model, and the student model is obtained by performing knowledge distillation on the feature map of the teacher model through a multi-value mask.

[0054] like Figure 3 As shown, the training process of the target detection model specifically includes the following steps:

[0055] S11, constructing a training set, wherein the training set includes a plurality of training data, wherein the training data includes ocean scene sample pictures and labels of the ocean scene sample pictures; the labels include the categories of sea surface targets in the ocean scene sample pictures and the location information of the sea surface targets; the ocean scene pictures include real ocean scene sample pictures and virtual ocean scene sample pictures, specifically including the following steps:

[0056] Step 1: Build a maritime virtual unmanned system simulation verification environment based on Unity3D to obtain virtual sample images of marine scenes with varied scenes, multiple angles, and all-round multi-scales.

[0057] Specifically, Unity3D is used as a rapid development platform for virtual unmanned maritime systems to design and encode the kinematic logic, physical logic, particle effects, and collision effects of the modeled model, and the visible light image sensor is simulated.

[0058] Visible light sensor simulation is based on Unity3D's light system and simulated visible light camera components. Different camera component parameters are set for different types of unmanned systems to achieve different perception ranges and accuracies. Based on this, the visible light sensor can obtain more realistic virtual sample images of ocean scenes, and its perception data quality is strongly correlated with the entity and environment simulation.

[0059] Based on the simulation virtual engine, different virtual scenes at sea are constructed, and the weather of complex scenes at sea can be switched at any time to enhance the simulation degree of virtual scenes at sea. Based on the powerful weather simulation system of Unity3D, data collection under different lighting conditions is realized for virtual scenes at sea, such as sunny days, cloudy days, dusk days and other types of lighting effects, to ensure the diversity of sample images of virtual scenes at sea from the perspective of visual appearance. At the same time, in order to further increase the diversity of multi-scale target features in sample images of virtual scenes at sea, color random scripts are added to all sea surface targets, and sea surface targets move randomly in the scene, thereby generating rich multi-scale target combination data, further improving the diversity of sample images of virtual scenes at sea.

[0060] A typical maritime virtual unmanned system simulation environment is built based on Unity3D, including a series of entities such as cruise ships, container ships, bulk carriers, sailboats, islands and reefs, and other ships, and all entities are equipped with exclusive physical characteristics and kinematic models. Multiple types of ships are added to the virtual maritime scene and random navigation is set. The labels of the sample images of the virtual maritime scene are obtained in real time based on the automatic labeling tool. The labels include the categories of sea surface targets in the sample images of the ocean scene and the location information of the sea surface targets, providing sufficient multi-scale virtual images of ocean scenes for the training of target detection models, in which the location information of the sea surface targets is represented by the labeling box.

[0061] Step 2: Enhance the real sample images of ocean scenes and the virtual sample images of ocean scenes through hybrid data enhancement methods such as random scaling, random flipping, padding, mosaic data enhancement and MixUp (random mixed image algorithm), and combine the two to obtain feature-enhanced virtual and real data.

[0062] Specifically, the random scaling strategy randomly scales the image and annotation box; the random flipping strategy randomly flips the image and annotation box horizontally, vertically, and diagonally; and the filling strategy fills the image to a fixed size or to a size that is divisible by a specified number.

[0063] Mosaic data enhancement is to randomly select 4 images and stitch them together. Each selected image is used as a sub-image. After stitching, simply integrate the labels in the sub-images as the labels of the stitched image. Specifically, first, create an empty image of twice the size and determine the center point of the image stitching; scale and stitch different images according to the four directions of upper left, upper right, lower left and lower right; stitch the labels of the four images together.

[0064] Mixup data enhancement is to randomly select two pictures, superimpose the two pictures pixel by pixel in a random ratio, use the selected two pictures as two sub-pictures, and then simply integrate the labels in the sub-pictures as the labels of the superimposed picture.

[0065] Based on step 1 and step 2, a virtual-real fusion six-category marine data set is constructed, including 22,854 real sample images of marine scenes and 18,617 virtual sample images of marine scenes. The real sample image set of marine scenes is divided into a training set (18,283 images) and a test set (4,571 images) according to a 4:1 ratio, and the validation set is consistent with the test set; the virtual sample image set of marine scenes is constructed into a training set (14,893 images) and a test set (3,724 images) according to a 4:1 ratio, and the validation set is consistent with the test set. Among them, the real sample images of marine scenes are annotated with annotation boxes by an annotation tool, and the annotation tool can be the LabelImg tool.

[0066] S12. Input the training data into the teacher model to obtain the feature map of the teacher model; input the training data into the student model to obtain the features of the student model, and determine the loss function according to the feature map of the teacher model and the feature map of the student model.

[0067] Specifically, the feature map obtained by the teacher model is used to select the distillation information to guide the student model training and perform multi-value mask-guided knowledge distillation.

[0068] like Figure 4 As shown, the generation process of multi-value masks includes the following steps:

[0069] Step 121: Obtain a feature map of the student model, wherein the feature map includes multiple candidate boxes.

[0070] A sample image of an ocean scene in the training data is input to the student model and the teacher model. Both the student model and the teacher model will generate feature maps of different scales (length and width). Therefore, one training data will correspond to multiple feature maps. Each sample image of an ocean scene corresponds to a file in a preset format, where the preset format can be a json format. The file in the preset format annotates the real frame of the sample image of the ocean scene and the category of the ocean entity in the real frame. The sample image of the ocean scene is input to the student model to generate k candidate frames, where k is a positive integer.

[0071] Step 122: Calculate the distance intersection over union (DIOU) between each of the candidate boxes and the real box to obtain multiple DIOU values.

[0072] Specifically, the DIOU of the real box and k candidate boxes are calculated respectively to obtain a W×H×K DIOU graph D m , D m The value of each point on corresponds to the DIOU between the k candidate boxes and the real box at that position, where W and H represent the width and length of the feature map.

[0073] The DIOU calculation formula is as follows:

[0074]

[0075] Among them, ρ 2 (b,b gt ) is a point b in the candidate box and a point b in the real box gt , C is the diagonal length of the minimum closed box surrounding the candidate box and the real box, and IOU is the intersection-over-union ratio.

[0076] Intersection over Union (IOU) is the area of ​​the overlap between the candidate box A and the real box B divided by the combined area of ​​the candidate box A and the real box B. The calculation formula of IOU is as follows:

[0077]

[0078] Step 123: Add the DIOU values ​​greater than the first threshold to obtain a mask.

[0079] Specifically, set the first threshold ψ, and set D m The DIOU values ​​that are greater than the first threshold ψ are added together to obtain a mask.

[0080] Step 124: normalize and map the mask to a preset range to obtain the multi-value mask.

[0081] The mask obtained in step 123 is mapped to the interval [0, 1] through the normalized value to obtain the final W×H multi-value mask M∈[0, 1] H×W .

[0082] The loss function in step 12 includes feature distillation loss and classification distillation loss. The loss function is determined according to the feature graph of the teacher model and the feature graph of the student model, specifically including:

[0083] (1) Determining the feature distillation loss according to the feature map of the teacher model, the feature map of the student model, and the multi-value mask.

[0084] The expression of the characteristic distillation loss is:

[0085]

[0086] Among them, LFPN is the feature distillation loss, H and W represent the height and width of the feature map respectively, and M w，h is a multi-valued mask of W×H. h，w and T h，w are the feature maps of the student model and the teacher model, φ adapt is the feature adaptation function.

[0087] Since the feature maps generated by the teacher model and the student model may have size mismatches, a feature adaptation layer is added after the feature map of the student model to align the features. Specifically, the feature adaptation function φ is used. adapt .

[0088] (2) determining the cosine similarity loss according to the first candidate box in the feature map of the student model and the second candidate box in the feature map of the teacher model; determining the cross entropy loss according to the prediction result of the category of the sea surface target in the first candidate box and the prediction result of the category of the sea surface target in the second candidate box;

[0089] (3) Determine the classification distillation loss based on the cross entropy loss and the cosine similarity loss.

[0090] The expression of the classification distillation loss is:

[0091]

[0092] Among them, L cls is the classification distillation loss, L cosine is the cosine similarity loss, α and β are hyperparameters for balancing different distillation losses. The teacher model and the student model generate k candidate boxes for a training data respectively. For the jth candidate box, the jth candidate box in the teacher model and the student model is converted into a feature vector through convolution operation. and Use linear transformation to obtain the prediction results of the category of the sea surface target in the jth candidate frame in the teacher model and the student model respectively and

[0093] S13. Using the loss function to update the parameters of the student model to obtain a trained student model, and using the trained student model as the target detection model.

[0094] The training set of virtual and real data is used as training samples to train the teacher model with large parameters and complex structure until convergence. The teacher model has superior performance and high detection accuracy, but its model complexity is extremely high, and there are limitations such as long training time, large storage space, and high hardware cost.

[0095] The distillation information from the classification head feature selection of the teacher model weights guides the student model training and performs relationship-based knowledge distillation to obtain a high-precision, lightweight, and real-time student model.

[0096] S2. Use a non-maximum suppression algorithm to filter the multiple candidate boxes according to the prediction category to obtain a prediction box of the sea surface target, and use the prediction category of the sea surface target corresponding to the prediction box as the category of the sea surface target; wherein the prediction box represents the location information of the sea surface target, and the categories of the sea surface targets include cruise ships, container ships, bulk carriers, sailboats and islands and reefs.

[0097] For step 121, the student model generates a prediction result of the category of the sea surface target for each ocean scene sample image. Pick The maximum value among them is taken as the confidence of the candidate box. The category with the largest median value is used as the predicted category of the candidate box. According to the predicted category, the candidate boxes of different scales are merged into one set, and different candidate box sets corresponding to different categories are obtained. The non-maximum suppression algorithm (NMS) is applied to each set to remove overlapping candidate boxes and retain the optimal candidate box.

[0098] Step S2 specifically includes the following steps:

[0099] S21, sorting the plurality of candidate boxes according to the confidence of the candidate boxes to obtain a first list. All candidate boxes are sorted in descending order according to the confidence to generate a first list L.

[0100] S22. According to the size of the confidence, select candidate boxes corresponding to the confidence from large to small as reference candidate boxes.

[0101] S23, traversing the first list, and for each reference candidate box, calculating an intersection-and-union ratio between the reference candidate box and all candidate boxes in the first list except the reference candidate box, to obtain a plurality of intersection-and-union ratios.

[0102] Specifically, the candidate box B1 with the highest confidence is selected from L as the benchmark candidate box, and the IOU between all candidate boxes except B1 and B1 is calculated.

[0103] Select the candidate box B2 with the second highest confidence from L as another benchmark candidate box, and calculate the IOU of all candidate boxes except B2 with B1.

[0104] The above process is repeated to traverse the first list L until all candidate boxes in L have been used as reference candidate boxes.

[0105] S24, determining whether the IoU ratio satisfies a repetition condition, and if so, deleting the candidate box corresponding to the IoU ratio to obtain a second list; the second list includes multiple candidate boxes; the repetition condition is that the IoU ratio is greater than a second threshold.

[0106] After each reference candidate box is selected in step S23, the IOU of all candidate boxes except the reference candidate box and the reference candidate box is calculated to determine whether the IOU meets the duplication condition. If so, the candidate box corresponding to the IOU is deleted to obtain a second list. The second list retains the candidate box with the highest confidence and removes the candidate boxes that are too similar to it.

[0107] S25. Classify the candidate boxes in the second list according to the predicted categories of the sea surface targets to obtain classification results, wherein each predicted category in the classification results includes at least one candidate box and the confidence of the candidate box.

[0108] S26. Using the candidate frame with the highest confidence as the predicted frame of the sea surface target, and using the predicted category of the sea surface target corresponding to the candidate frame with the highest confidence as the category of the sea surface target.

[0109] For example, if there are 2 prediction categories for sea surface targets: c1 and c2. After NMS, there are 5 candidate boxes left. Among them, 3 candidate boxes have a prediction category of c1, and their confidences are [0.6, 0.9, 0.8] respectively. The candidate box with a confidence of 0.9 is taken as the final prediction box, and its prediction category is c1; 2 candidate boxes have a prediction category of c2, and their confidences are [0.7, 0.8] respectively. The candidate box with a confidence of 0.8 is taken as the final prediction box, and its prediction category is c2.

[0110] The average precision (AP) consists of the area enclosed by the Precision-Recall curve and the coordinates, which represents the average of the highest accuracy rates at different recall rates (generally, the AP is calculated for each category separately). The better the performance, the higher the ratio. 50 It means that the IOU threshold for determining positive and negative samples is 0.5, AP S 、AP M and AP L The three indicators can measure the detection accuracy of small, medium and large objects. This embodiment adopts the evaluation criteria of the image dataset (Microsoft Common Objects in Context, MS COCO). Instead of using a fixed IOU threshold, it takes the average value between 0.5 and 0.95 on multiple IOU thresholds, and uses mean average precision (mAP), AP 50 、AP 75 、AP S 、AP M and AP LAs the evaluation standard. The calculation process of the evaluation index AP mainly involves true positive (TruePositive, TP), true negative (TrueNegative, TN), false positive (FalsePositive, FP), false negative (FalseNegative, FN), positive (Positive) and negative (Negative), accuracy (Precision) and recall (Recall) and other indicators. Among them, TP represents the positive sample that is detected correctly, and the IOU of the detection prediction box and the real box ≥ the detection threshold, that is, the number of predicted bounding boxes with correct classification and correct bounding box coordinates; FP represents the original positive sample, but it is mistakenly detected as a negative sample, which is divided into two cases: one is that the IOU of the predicted box and the real box ≥ the detection threshold, but the predicted category is wrong, and the other is that the IOU of the predicted box and the real box < the detection threshold; FN represents the original negative sample, but it is mistakenly detected as a positive sample; TN represents the negative sample that is detected correctly. The calculation method of accuracy and recall is shown in the following formula:

[0111]

[0112] Table 1 shows the comparative experimental results of different target detection models on six types of marine virtual-real fusion datasets. All models use the Stochastic Gradient Descent (SGD) optimizer or the Adaptive Moment Estimation (Adam) optimizer, and the training rounds are set to 3x, where x is a positive integer. In this embodiment, the rounds are set to 36.

[0113] The different target detection models are: Cascade Region-based Convolutional Neural Network (Cascade R-CNN), Fully Convolutional One-Stage (FCOS), Sparse Region-based Convolutional Neural Network (Sparse R-CNN), Generalized focal loss (GFL) and target detector (Varifocalnet, VFNET).

[0114] Table 1

[0115]

[0116] From the results in Table 1, it can be seen that the target detection method based on virtual and real data proposed in this application can greatly improve the accuracy of multi-scale target detection, which proves that the virtual and real group collaborative perception method can make up for the limitations of incomplete single perception data, thereby improving the marine scene perception ability of unmanned system clusters. In addition, due to the fusion of virtual and real data, the detection accuracy of the six types of targets at sea is relatively balanced, and the A of each category of target detection is P50 The value difference is not large, which reduces the difference in detection accuracy of various categories of targets in the maritime unmanned system cluster.

[0117] In an exemplary embodiment, Figure 5 As shown, the present application provides a target detection system based on virtual and real data, and the target detection system based on virtual and real data includes:

[0118] The detection module 41 is used to perform target detection on the ocean scene image using the target detection model to obtain the predicted categories of multiple sea surface targets and multiple candidate boxes; the target detection model is obtained by supervising the student model through the feature map obtained by the teacher model, the teacher model is a trained neural network model, and the student model is obtained by performing knowledge distillation on the feature map of the teacher model using a multi-value mask;

[0119] The detection result output module 42 is used to use a non-maximum suppression algorithm to filter the multiple candidate boxes according to the predicted category to obtain a predicted box of the sea surface target, and use the predicted category of the sea surface target corresponding to the predicted box as the category of the sea surface target; wherein the predicted box represents the location information of the sea surface target, and the categories of the sea surface targets include cruise ships, container ships, bulk carriers, sailboats and islands and reefs.

[0120] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store various operating data of the transportation equipment. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a target detection method based on virtual and real data is implemented.

[0121] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0122] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0123] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0124] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0125] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0126] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0127] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A target detection method based on virtual and real data, characterized in that: The target detection method based on virtual and real data includes: The target detection model is used to perform target detection on the ocean scene image to obtain predicted categories of multiple sea surface targets and multiple candidate boxes; the target detection model is obtained by supervised training of the student model through the feature map obtained by the teacher model, the teacher model is a trained neural network model, and the student model is obtained by performing knowledge distillation on the feature map of the teacher model through a multi-value mask; A non-maximum suppression algorithm is used to screen the multiple candidate boxes according to the prediction category to obtain a prediction box of the sea surface target, and the prediction category of the sea surface target corresponding to the prediction box is used as the category of the sea surface target; wherein the prediction box represents the location information of the sea surface target, and the categories of the sea surface targets include cruise ships, container ships, bulk carriers, sailboats and islands and reefs.

2. The target detection method based on virtual and real data according to claim 1, characterized in that: The training process of the target detection model specifically includes: Constructing a training set, wherein the training set includes a plurality of training data, wherein the training data includes ocean scene sample pictures and labels of the ocean scene sample pictures; the labels include categories of sea surface targets in the ocean scene sample pictures and location information of the sea surface targets; the ocean scene pictures include real ocean scene sample pictures and virtual ocean scene sample pictures; Inputting the training data into the teacher model to obtain a feature map of the teacher model; inputting the training data into the student model to obtain a feature map of the student model; Determine a loss function based on the feature graph of the teacher model and the feature graph of the student model; The loss function is used to update the parameters of the student model to obtain a trained student model, and the trained student model is used as the target detection model.

3. The target detection method based on virtual and real data according to claim 2, characterized in that: The loss function includes feature distillation loss and classification distillation loss; Determining a loss function according to the feature graph of the teacher model and the feature graph of the student model specifically includes: Determining the feature distillation loss according to the feature map of the teacher model, the feature map of the student model, and the multi-value mask; Determine a cosine similarity loss according to a first candidate box in the feature map of the student model and a second candidate box in the feature map of the teacher model; determine a cross entropy loss according to a prediction result of a category of a sea surface target in the first candidate box and a prediction result of a category of a sea surface target in the second candidate box; The classification distillation loss is determined based on the cross entropy loss and the cosine similarity loss.

4. The target detection method based on virtual and real data according to claim 3 is characterized in that: The expression of the characteristic distillation loss is: Among them, L FPN is the feature distillation loss, H and W represent the height and width of the feature map respectively, and M w，h is a multi-valued mask of W×H. h，w and T h，w are the feature maps of the student model and the teacher model, φ adapt is the feature adaptation function.

5. The target detection method based on virtual and real data according to claim 3, characterized in that: The expression of the classification distillation loss is: Among them, L cls is the classification distillation loss, L cosine is the cosine similarity loss, α and β are hyperparameters for balancing different distillation losses. The teacher model and the student model generate k candidate boxes for a training data respectively. For the jth candidate box, the jth candidate box in the teacher model and the student model is converted into a feature vector through convolution operation. and Use linear transformation to obtain the prediction results of the category of the sea surface target in the jth candidate frame in the teacher model and the student model respectively and 6. The target detection method based on virtual and real data according to claim 3, characterized in that: The generation process of the multi-value mask includes: Obtain a feature map of the student model, wherein the feature map includes multiple candidate boxes. Calculate the distance intersection-and-union ratio between each candidate frame and the real frame to obtain multiple distance intersection-and-union ratio values. Add the distance intersection and union ratios greater than the first threshold to obtain a mask; The mask is normalized and mapped to a preset range to obtain the multi-value mask.

7. The target detection method based on virtual and real data according to claim 1, characterized in that: The non-maximum suppression algorithm is used to screen the plurality of candidate frames according to the prediction category to obtain a prediction frame of the sea surface target, and the prediction category of the sea surface target corresponding to the prediction frame is used as the category of the sea surface target, including: Sorting the plurality of candidate boxes according to the confidence levels of the candidate boxes to obtain a first list; According to the size of the confidence, select the candidate boxes corresponding to the confidence from large to small as the reference candidate boxes; Traversing the first list, for each reference candidate box, calculating an intersection-and-union ratio between the reference candidate box and all candidate boxes in the first list except the reference candidate box, to obtain a plurality of intersection-and-union ratios; Determine whether the intersection-and-union ratio satisfies a repetition condition, and if so, delete the candidate box corresponding to the intersection-and-union ratio to obtain a second list; the second list includes multiple candidate boxes; the repetition condition is that the intersection-and-union ratio is greater than a second threshold; Classifying the candidate boxes in the second list according to the predicted category of the sea surface target to obtain a classification result, wherein each predicted category in the classification result includes at least one candidate box and the confidence of the candidate box; The candidate frame with the highest confidence is used as the predicted frame of the sea surface target, and the predicted category of the sea surface target corresponding to the candidate frame with the highest confidence is used as the category of the sea surface target.

8. A target detection system based on virtual and real data, characterized in that: The target detection system based on virtual and real data includes: A detection module is used to perform target detection on an ocean scene image using a target detection model to obtain predicted categories of multiple sea surface targets and multiple candidate boxes; the target detection model is obtained by supervising and training a student model using a feature map obtained by a teacher model, the teacher model is a trained neural network model, and the student model is obtained by performing knowledge distillation on the feature map of the teacher model using a multi-value mask; The detection result output module is used to use a non-maximum suppression algorithm to filter the multiple candidate boxes according to the predicted category to obtain a predicted box of a sea surface target, and use the predicted category of the sea surface target corresponding to the predicted box as the category of the sea surface target; wherein the predicted box represents the location information of the sea surface target, and the categories of the sea surface targets include cruise ships, container ships, bulk carriers, sailboats and islands and reefs.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the target detection method based on virtual and real data described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the target detection method based on virtual and real data described in any one of claims 1 to 7 are implemented.

Images