A drone re-identification method based on deep learning
Through a deep learning-based drone re-identification method, by matching ship and cargo feature vectors, combined with a feature index library and a lightweight network model, the problem of real-time drone ship identification is solved, and efficient and accurate ship re-identification is achieved.
Patent Information
- Application Number
- CN202211182639.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-27
- Publication Date
- 2025-10-14
- Estimated Expiration
- 2042-09-27
AI Technical Summary
Ship targets in drone aerial images are small, have large scale variations, and are highly similar. Existing re-identification methods find it difficult to achieve real-time recognition and accurately distinguish between new and old ships on drones with limited computer computing power.
A deep learning-based drone re-identification method is adopted. Feature vectors are extracted through the network models of the ship body and the cargo body, and compared with the feature index library. Multiple recognition thresholds and distance thresholds are set, and lightweight network models such as MobileNetV1 and HNSW32 are used for feature matching to achieve real-time recognition.
The detection speed and accuracy of drone re-identification are significantly improved, enabling real-time operation on low-computing power devices and effectively distinguishing between similar types of ships.
Smart Images
Figure CN115578653B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to a deep learning-based unmanned aerial vehicle (UAV) re-identification method and belongs to the image detection field. BACKGROUND
[0002] Traditional ship identification methods include an automatic identification system (AIS), a shore-based radar monitoring system, a closed-circuit television monitoring system (CCTV), and manual cruising. The AIS system requires positioning equipment to be installed on a target ship, and cannot identify non-cooperative targets. The shore-based radar system has poor identification capability for small ships. Both of them lack video image information and need to obtain the video image information through the CCTV system. Meanwhile, the view angle obtained by the camera is relatively limited due to the locations of the port, bridge crane and shore base.
[0003] The UAV-borne sensor is used to re-identify the ship target in the channel by using aerial images, which can effectively improve the water traffic supervision capability of the maritime department. Effective and accurate maritime ship re-identification can supervise the state of the ship, and is very important for improving the safety of ship navigation.
[0004] However, the UAV aerial image has the difficulties of small ship target and large scale variation, and it is difficult to identify the ship target. In addition, the high similarity between ships further increases the difficulty of re-identification, resulting in insufficient accuracy of the existing re-identification method.
[0005] Most of the existing ship identification methods are based on convolutional neural networks. Some researchers improve the R-CNN method, extract the network of the region of interest, and assist the position matching of the regression frame to improve the recall rate of the method. Some researchers input the classification confidence of the Faster R-CNN into the Bayesian feature fusion method, judge according to the joint probability distribution of the continuous frames of the video, and evaluate the target detection and classification results.
[0006] However, the computer algorithm power of the UAV itself is limited, and the load is small. The above method has high requirements for the computer algorithm power, and it is difficult to realize the real-time identification (more than 15 FPS) of the target ship by applying the above method to the UAV. Meanwhile, for the newly added unknown ship type, the method itself needs to be adjusted according to the ship features, which will consume a lot of algorithm power and time. For the ship that disappears from the field of view and then appears again, the above method cannot determine the identity of the ship.
[0007] Due to the above reasons, the inventors have made intensive research and proposed a deep learning-based UAV re-identification method to solve the above problems. SUMMARY
[0008] In order to overcome the above problems, the inventors have made intensive research and proposed a deep learning-based UAV re-identification method, which comprises the following steps:
[0009] S1. The drone takes pictures of the water surface and extracts the ship area through the ship main body network model;
[0010] S2. Identify the ship area through the ship re-identification network model and obtain the ship feature vector;
[0011] S3. Compare the ship feature vector with the feature vector in the ship feature index library to determine the identification ID of the ship.
[0012] In a preferred embodiment, in S1, the ship cargo area is also extracted through the cargo body network model.
[0013] In a preferred embodiment, the ship main body network model is a single-stage detection network model.
[0014] In a preferred embodiment, in S2, the cargo area on board the ship is also identified by a cargo re-identification network model to obtain a cargo feature vector.
[0015] In a preferred embodiment, in S3, the feature index library is expanded.
[0016] In a preferred embodiment, in S3, the ship feature vector is compared with the feature vector in the ship feature index library to obtain the recognition confidence of each ship, including the following situations:
[0017] Case 1: The ship identification confidence level is not higher than the first identification threshold T1:
[0018] Expand the ship feature index library and cargo feature index library;
[0019] Case 2: The ship identification confidence level is higher than the first-level identification threshold T1 and lower than the second-level identification threshold T2:
[0020] Compare the cargo feature vector with the feature vectors in the cargo feature index library to obtain the confidence level of all cargo identification;
[0021] When the confidence level of any cargo identification is higher than the third-level identification threshold G1, and the distance between the center of the corresponding cargo area and the ship area is less than the distance threshold K, the corresponding ship feature vector identification ID is marked to the ship. Otherwise, the ship feature index library and the cargo feature index library are expanded.
[0022] Case 3: The ship identification confidence level is not lower than the second identification threshold T2:
[0023] Mark the corresponding ship feature vector identification ID to the ship;
[0024] The expansion of the ship feature index library refers to adding the ship feature vector to the ship feature index library and assigning a unique identification ID to the ship feature vector.
[0025] The expansion of the cargo feature index library refers to adding the corresponding cargo feature vector to the cargo feature index library, and the corresponding refers to that the distance between the center of the ship cargo area and the center of the ship area is less than the distance threshold K.
[0026] In a preferred embodiment, in S3, before the ship feature index library and the cargo feature index library are expanded, the ship is also photographed multiple times, and the angles of each time of photographing are different.
[0027] The pictures obtained by multiple times of photographing are input into the ship main body network model to extract multiple ship areas, and the multiple ship areas are input into the ship re-identification network model to obtain multiple ship feature vectors.
[0028] The pictures obtained by multiple times of photographing are input into the cargo main body network model to extract multiple ship cargo areas, and the multiple ship cargo areas are input into the cargo re-identification network model to obtain multiple cargo feature vectors.
[0029] When the ship feature index library is expanded, the multiple ship feature vectors are all expanded to the ship feature index library.
[0030] When the cargo feature index library is expanded, the multiple cargo feature vectors are all expanded to the cargo feature index library.
[0031] In a preferred embodiment, in S3, the HNSW32 method is used for comparison of the feature vectors.
[0032] In a preferred embodiment, in S2, the ship re-identification network model is a MobileNetV1 re-identification network.
[0033] In a preferred embodiment, when the MobileNetV1 re-identification network is trained, the training loss function L3 is set as:
[0034]
[0035] Wherein N is the total number of pictures input into the MobileNetV1 re-identification network in a batch, i represents different pictures, is the ship feature vector of the ith picture, n represents the total number of sample categories in the index library, j represents a sample category, y i represents the sample category of the ith picture, θ j represents the feature vector of the jth category picture, s and m are hyperparameters.
[0036] The present application has the beneficial effects including:
[0037] (1) The unmanned aerial vehicle re-identification method based on deep learning provided by the present application significantly improves the detection speed and can realize real-time operation on low-power devices on board.
[0038] (2) The unmanned aerial vehicle re-identification method based on deep learning provided by the present application can distinguish similar types of ships and has high re-identification accuracy. BRIEF DESCRIPTION OF DRAWINGS
[0039] Figure 1 The figure shows a schematic diagram of the unmanned aerial vehicle re-identification method based on deep learning according to a preferred embodiment of the present application. DETAILED DESCRIPTION
[0040] The present application will be further described in detail below by means of the accompanying drawings and examples. Through these descriptions, the features and advantages of the present application will become clearer and more explicit.
[0041] The word "exemplary" used herein means "serving as an example, embodiment or illustration". Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0042] The present application provides an unmanned aerial vehicle re-identification method based on deep learning, comprising the following steps:
[0043] S1, the unmanned aerial vehicle takes pictures of the water surface to obtain pictures, and extracts the ship area through a ship main body network model;
[0044] S2, the ship area is identified through a ship re-identification network model to obtain a ship feature vector;
[0045] S3, the ship feature vector is compared with the feature vectors in the ship feature index library to determine the identification ID of the ship.
[0046] Further, when there are multiple ship feature vectors in the picture, they are compared with the ship feature index library in turn. If there is no identical feature in the ship feature index library, the ship feature vector is added to the ship feature index library and is assigned an identification ID,
[0047] If there is an identical feature in the ship feature index library, the corresponding identification ID is marked to the ship, realizing the re-identification of the ship.
[0048] In the present invention, a drone equipped with a visual sensor is used to realize the re-identification of the identity of the target ship. Compared with the traditional ship monitoring and identification method, the drone ship re-identification has a wider field of view, a more flexible viewing angle, and a wider recognition coverage. It can obtain ship images at different angles and heights as needed, and can realize real-time monitoring of the target ship.
[0049] Furthermore, when a ship disappears from view and reappears, it is necessary to confirm the identity of the ship to determine whether it is a new ship or a previously disappeared ship, that is, re-identification of the ship.
[0050] During actual testing, it was found that although the above method can achieve re-identification of ships, due to the small number of ship types and the high similarity between ships of the same type, it is impossible to accurately determine whether a ship is newly appeared or has appeared before based solely on the ship feature vector. In the recognition confidence range of 60% to 80%, the re-identification error rate is high.
[0051] According to the present invention, in S1, the ship cargo area is also extracted through the cargo body network model.
[0052] Furthermore, the ship cargo area refers to the area where items carried on the deck, such as a group of containers, buildings above the deck, etc. are located.
[0053] This invention innovatively separates ship re-identification into vessel identification and cargo identification. The inventors discovered that drone images of ships show varying angles, heights, and dimensions, leading to a high degree of similarity between different ships. By incorporating cargo identification, re-identification is performed beyond the already high similarity between the ship bodies, thereby increasing re-identification accuracy.
[0054] Furthermore, since the ship and the cargo on board are identified separately, there is no mutual interference between the two, which can further increase the accuracy of re-identification.
[0055] According to the present invention, a distance threshold K is further provided. When the pixel distance between the center point of the ship area and the center point of the ship cargo area is less than the distance threshold K, the ship cargo is considered to belong to the ship.
[0056] Preferably, the threshold value ranges from 80 to 200.
[0057] In the present invention, the ship main body network model is a single-stage detection network model. The single-stage detection network model has a simple structure and requires low computing power. It is suitable for rapid detection by drone onboard computers. Since the ship edge is clear, the detection accuracy is relatively high. The extraction accuracy of the single-stage detection network model is generally high. Preferably, the single-stage detection network model is YoloX.
[0058] Furthermore, the cargo body network model is also a single-stage detection network model, which is also suitable for rapid detection by drone onboard computers. Although the single-stage detection network model is not very accurate in extracting shipborne cargo areas and is prone to marking non-shipborne cargo areas as shipborne cargo areas, the background color of the image is a relatively pure water surface, and the shipborne cargo area is rarely lost, resulting in the low detection accuracy brought by the single-stage detection network model and does not affect the accuracy of re-identification.
[0059] In a preferred embodiment, in S2, the cargo area on board the ship is also identified by a cargo re-identification network model to obtain a cargo feature vector.
[0060] According to the present invention, in S3, the feature index library is further expanded.
[0061] Traditional re-identification uses image collections as re-identification index libraries. However, there is no comprehensive image collection ship re-identification index library in the existing ship identification field. In addition, using image collections as re-identification index libraries requires preparing a large number of images, and the comparison process requires high computer computing power, which cannot be used directly on drones.
[0062] In the present invention, a feature index library is used as the re-identification index library, and feature vectors are used to replace traditional images, thereby reducing the computing power during comparison, while reducing storage space and improving storage and reading efficiency, so that the index library can be expanded quickly and in real time, thereby realizing real-time ship monitoring.
[0063] Furthermore, the expansion is achieved through vector concatenation, that is, the new feature vector is directly connected to the end of the feature index library to achieve expansion.
[0064] In a preferred embodiment, in S3, when the feature index library is expanded, pictures with only background environments but no ships are also added. These pictures can effectively improve the accuracy of ship re-identification and reduce false detections.
[0065] In a preferred embodiment, in S3, the ship feature vector is compared with the feature vector in the ship feature index library to obtain the recognition confidence of each ship, including the following situations:
[0066] Case 1: The ship identification confidence level is not higher than the first identification threshold T1:
[0067] Expand the ship feature index library and cargo feature index library;
[0068] Case 2: The ship identification confidence level is higher than the first-level identification threshold T1 and lower than the second-level identification threshold T2:
[0069] Compare the cargo feature vector with the feature vectors in the cargo feature index library to obtain the confidence level of all cargo identification;
[0070] When the confidence level of any cargo identification is higher than the third-level identification threshold G1, and the distance between the center of the corresponding cargo area and the ship area is less than the distance threshold K, the corresponding ship feature vector identification ID is marked to the ship. Otherwise, the ship feature index library and the cargo feature index library are expanded.
[0071] Case 3: The ship identification confidence level is not lower than the second identification threshold T2:
[0072] Mark the corresponding ship feature vector identification ID to the ship;
[0073] The expansion of the ship feature index library refers to adding the ship feature vector to the ship feature index library and assigning a unique identification ID to the ship feature vector;
[0074] The expansion of the cargo feature index library refers to adding the corresponding cargo feature vector to the cargo feature index library, and the correspondence refers to that the distance between the center of the cargo area on board and the center of the ship area is less than the distance threshold K.
[0075] In the present invention, by setting the first, second and third re-identification thresholds, and using the ship feature vector and the cargo feature vector to perform re-identification, the accuracy of re-identification is greatly improved.
[0076] In a preferred embodiment, the first re-identification threshold T1 is 40-60%, preferably 50%; the second re-identification threshold T2 is 80-95%, preferably 90%; and the third re-identification threshold G1 is 60-90%, preferably 70%.
[0077] In a more preferred embodiment,
[0078] In S3, before expanding the ship feature index library and the cargo feature index library, the ship is photographed multiple times, each photographed at a different angle, preferably 20 times or more.
[0079] Input the pictures obtained by multiple photos into the ship main body network model to extract multiple ship areas, and input the multiple ship areas into the ship re-identification network model to obtain multiple ship feature vectors.
[0080] Input the images obtained by multiple photos into the cargo subject network model to extract multiple cargo areas on board, and input the multiple cargo areas into the cargo re-identification network model to obtain multiple cargo feature vectors;
[0081] When expanding the ship feature index library, multiple ship feature vectors are expanded to the ship feature index library.
[0082] When the cargo feature index library is expanded, multiple cargo feature vectors are expanded to the cargo feature index library.
[0083] The above settings improve the accuracy and comprehensiveness of the feature index library, thereby improving the accuracy of re-identification.
[0084] According to a preferred embodiment of the present invention, the HNSW32 method is used to compare feature vectors. HNSW (Hierarchical Navigable Small World) is a classic retrieval method and will not be described in detail in the present invention.
[0085] In a preferred embodiment, the ship re-identification network model is a MobileNetV1 re-identification network.
[0086] In a preferred embodiment, the cargo re-identification network model is a MobileNetV1 re-identification network.
[0087] MobileNetV1 is a lightweight convolutional neural network that splits the traditional convolutional neural network into depth-wise separable convolution and point-by-point convolution in its structure. Compared with traditional convolution operations, the model parameters and computational complexity are greatly reduced, making it more suitable for application on drone-mounted computers.
[0088] In a preferred embodiment, the image of the ship area is resized to a 224×224 3-channel image and then input into the ship re-identification network model, which then outputs a ship feature vector of length 512. This size is compatible with both recognition accuracy and computing power.
[0089] In a preferred embodiment, when training the MobileNetV1 re-identification network, the training loss function L3 is set to:
[0090]
[0091] Among them, N is the total number of images input into the MobileNetV1 re-identification network in a batch, i represents different images, is the ship feature vector of the i-th picture, n represents the total number of sample categories in the index library, j represents the sample category, that is, the type of ship, y i Indicates the sample category of the i-th picture, θ j Represents the feature vector of the j-category image, s and m are hyperparameters.
[0092] The training loss function of the traditional MobileNetV1 re-identification network is generally a Softmax function. Experiments have found that the traditional training loss function cannot effectively distinguish the inter-class features between different ships. In this paper, the training loss function is improved to overcome the above defects, so that the model can more accurately distinguish the features between different ships and improve the accuracy of re-identification of different types of ships.
[0093] Example
[0094] Example 1
[0095] In a certain river channel, a drone was used to conduct a re-identification experiment on ships on the river. The drone flew back and forth within a range of 3km to identify whether the ship within the range was a new ship.
[0096] The re-identification process includes the following steps:
[0097] S1. The drone takes pictures of the water surface and extracts the ship area through the ship main body network model;
[0098] S2. Identify the ship area through the ship re-identification network model and obtain the ship feature vector;
[0099] S3. Compare the ship feature vector with the feature vector in the ship feature index library to determine the identification ID of the ship.
[0100] Among them, in S1, the ship cargo area is also extracted through the cargo body network model.
[0101] The network model of the ship and the network model of the cargo are both YoloX.
[0102] In S2, the cargo area on board is also identified through the cargo re-identification network model to obtain the cargo feature vector.
[0103] The ship re-identification network model and the cargo re-identification network model are both MobileNetV1 re-identification networks.
[0104] When training the MobileNetV1 re-identification network, set the training loss function L3 to:
[0105]
[0106] In S3, the ship feature vector is compared with the feature vector in the ship feature index library to obtain the recognition confidence of each ship, including the following situations:
[0107] Case 1: The ship identification confidence level is not higher than the first identification threshold T1:
[0108] Expand the ship feature index library and cargo feature index library;
[0109] Case 2: The ship identification confidence level is higher than the first-level identification threshold T1 and lower than the second-level identification threshold T2:
[0110] Compare the cargo feature vector with the feature vectors in the cargo feature index library to obtain the confidence level of all cargo identification;
[0111] When the confidence level of any cargo identification is higher than the third-level identification threshold G1, and the distance between the center of the corresponding cargo area and the ship area is less than the distance threshold K, the corresponding ship feature vector identification ID is marked to the ship. Otherwise, the ship feature index library and the cargo feature index library are expanded.
[0112] Case 3: The ship identification confidence level is not lower than the second identification threshold T2:
[0113] Mark the corresponding ship feature vector identification ID to the ship;
[0114] The expansion of the ship feature index library refers to adding the ship feature vector to the ship feature index library and assigning a unique identification ID to the ship feature vector;
[0115] The expansion of the cargo feature index library refers to adding the corresponding cargo feature vector to the cargo feature index library, and the correspondence refers to that the distance between the center of the cargo area on board and the center of the ship area is less than the distance threshold K.
[0116] Among them, the HNSW32 method is used to compare the feature vectors.
[0117] In S3, before expanding the ship feature index library and cargo feature index library, the ship is photographed 20 times, each time at a different angle.
[0118] The pictures obtained from 20 photos are input into the ship main body network model to extract multiple ship areas, and the multiple ship areas are input into the ship re-identification network model to obtain multiple ship feature vectors.
[0119] The images obtained from the 20 photos are input into the cargo subject network model to extract multiple cargo regions on board. The multiple cargo regions on board are then input into the cargo re-identification network model to obtain multiple cargo feature vectors.
[0120] When expanding the ship feature index library, multiple ship feature vectors are expanded to the ship feature index library.
[0121] When the cargo feature index library is expanded, multiple cargo feature vectors are expanded to the cargo feature index library.
[0122] The average frame recognition per second FPS is 38.9%, and the number of correct recognition frames accounts for 96.23% of the total recognition frames.
[0123] In the description of the present application, it should be noted that the terms "upper", "lower", "inner", "outer", "front", "back" and the like indicate the orientation or positional relationship in the working state of the present application, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third", "fourth" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.
[0124] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.
[0125] The above describes the present application in combination with the preferred embodiments, but these embodiments are only exemplary and serve only to illustrate. On this basis, various substitutions and improvements can be made to the present application, which all fall within the scope of protection of the present application.
Claims
1. A drone re-identification method based on deep learning, characterized in that: The following steps are involved: S1. The drone takes pictures of the water surface and extracts the ship area through the ship main body network model; S2. Identify the ship area through the ship re-identification network model and obtain the ship feature vector; S3. Compare the ship feature vector with the feature vector in the ship feature index library to determine the identification ID of the ship; In S3, the ship feature vector is compared with the feature vector in the ship feature index library to obtain the recognition confidence of each ship, including the following situations: Case 1: The ship identification confidence level is not higher than the first identification threshold T1: Expand the ship feature index library and cargo feature index library; Case 2: The ship identification confidence level is higher than the first-level identification threshold T1 and lower than the second-level identification threshold T2: Compare the cargo feature vector with the feature vectors in the cargo feature index library to obtain the confidence level of all cargo identification; When the confidence level of any cargo identification is higher than the third-level identification threshold G1, and the distance between the center of the corresponding cargo area and the ship area is less than the distance threshold K, the corresponding ship feature vector identification ID is marked to the ship. Otherwise, the ship feature index library and the cargo feature index library are expanded. Case 3: The ship identification confidence level is not lower than the second identification threshold T2: Mark the corresponding ship feature vector identification ID to the ship; The expansion of the ship feature index library refers to adding the ship feature vector to the ship feature index library and assigning a unique identification ID to the ship feature vector; The expansion of the cargo feature index library refers to adding the corresponding cargo feature vector to the cargo feature index library, and the correspondence refers to the distance between the center of the cargo area on board and the center of the ship area being less than a distance threshold K; In S3, before expanding the ship feature index library and the cargo feature index library, the ship is photographed multiple times, each time at a different angle. Input the pictures obtained by multiple photos into the ship main body network model to extract multiple ship areas, and input the multiple ship areas into the ship re-identification network model to obtain multiple ship feature vectors. Input the images obtained by multiple photos into the cargo subject network model to extract multiple cargo areas on board, and input the multiple cargo areas into the cargo re-identification network model to obtain multiple cargo feature vectors; When expanding the ship feature index library, multiple ship feature vectors are expanded to the ship feature index library. When expanding the cargo feature index library, multiple cargo feature vectors are expanded to the cargo feature index library; In S3, the HNSW32 method is used to compare feature vectors.
2. The drone re-identification method based on deep learning according to claim 1, characterized in that: In S1, the ship cargo area is also extracted through the cargo body network model.
3. The drone re-identification method based on deep learning according to claim 1, characterized in that: The ship main body network model is a single-stage detection network model.
4. The drone re-identification method based on deep learning according to claim 1, characterized in that In S2, the cargo area on board is also identified through the cargo re-identification network model to obtain the cargo feature vector.
5. The drone re-identification method based on deep learning according to claim 1, characterized in that: In S3, the feature index library is expanded.
6. The drone re-identification method based on deep learning according to claim 1, characterized in that: In S2, the ship re-identification network model is the MobileNetV1 re-identification network.
7. The deep learning-based drone re-identification method according to claim 6, characterized in that: When training the MobileNetV1 re-identification network, set the training loss function L3 to: Among them, N is the total number of images input into the MobileNetV1 re-identification network in a batch, i represents different images, is the ship feature vector of the i-th picture, n represents the total number of sample categories in the index library, j represents the sample category, and y i Indicates the sample category of the i-th picture, θ j Represents the feature vector of the j-category image, s and m are hyperparameters.