Array camera based real-time tracking of coastlines and vessels

Abstract

The application provides a kind of array camera-based coastline ship real-time tracking method, which comprises determining the small lens number and the overlapping area of the image data collected between each adjacent small lens according to the small lens distribution of array camera;All ships to be tracked in the small lens are preliminarily labeled rectangular frame TE;Each labeled rectangular frame TE ship forms a detection frame TF, and a set of Box (x, y, w, h) and a set of feature vectors P are obtained;The intersection-over-union (IOU) of the detection frame TF and the rectangular frame TE of the ship to be tracked is compared, and the ship to be tracked with the intersection-over-union (IOU) reaching the set value is tracked;The ship tracking sequence TraceQueue is obtained;The current small lens sends the signal of the ship to be tracked to the corresponding adjacent small lens through socket communication, and continues to track the ship in the adjacent small lens. The position information and the feature vector of the ship are combined to determine whether two targets in different cameras are the same, thereby improving the accuracy of target tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ship trajectory tracking methods, in particular to a coastal ship real-time tracking method based on an array camera. BACKGROUND

[0002] The coastline is generally long, and it is impossible to monitor the entire coastline with a single lens. Multiple lenses need to work together to complete the monitoring task. The ship tracking method is the core of the coastal monitoring system. The existing monitoring system can only detect ships and cannot track target ships.

[0003] One difficulty in ship tracking is the tracking of the border. When the ship target moves from one lens to another, it is necessary to determine whether the ships in the two lenses are the same ship according to the appearance characteristics, behavior characteristics and other factors of the ship. The technology for determining the same ship according to the characteristics is not mature enough, especially in the special environment of the coastline. When the ship target crosses multiple lenses, it becomes very difficult to accurately track the trajectory of each ship.

[0004] Chinese Patent Publication No. CN111476827A discloses a target tracking method, system, electronic device and storage medium. The target tracking method uses the working principle of world coordinates for tracking. The position and size of the detection bounding box of the same target in different cameras will differ even if world coordinates are used. Therefore, there is an error in determining the unique world coordinates of each target, especially when two targets are very close.

[0005] Chinese Patent Publication No. CN108051777A discloses a target tracking method, device and electronic equipment. The target tracking method uses the working principle of sensors for tracking. The hardware requirements are high, but the tracking accuracy of the target by the sensor is poor.

[0006] Therefore, it is necessary to provide a coastal ship real-time tracking method based on an array camera to overcome the above problems. SUMMARY

[0007] The purpose of the present application is to provide a coastal ship real-time tracking method based on an array camera. The method uses the structural characteristics of the camera itself, combines ship position information and feature vectors to determine whether two targets in different cameras are the same, thereby improving the accuracy of target tracking.

[0008] To achieve the above purpose, the present application adopts a technical scheme: a coastal ship real-time tracking method based on an array camera, comprising the following steps:

[0009] The first step is to determine the small lens number and the overlapping area of the image data collected by each adjacent small lens according to the small lens distribution of the array camera.

[0010] The second step is to preliminarily label a rectangular frame TE of the to-be-tracked ship in all the small lenses.

[0011] The third step is to use the ship detection model to detect the image data of the labeled rectangular frame TE of the ship using the N small lenses of the array camera, form a detection frame TF for each labeled rectangular frame TE of the ship, obtain a set of Box(x, y, w, h) and a set of feature vectors P, and store them.

[0012] The fourth step is to compare the intersection-over-union (IOU) of the detection frame TF and the rectangular frame TE of the to-be-tracked ship in each small lens having the labeled rectangular frame TE of the ship, and track the ship whose IOU reaches a set value among the labeled to-be-tracked ships.

[0013] The fifth step is to track the rectangular frame TE using the sailing information and appearance information of the ship, and obtain a ship tracking sequence TraceQueue.

[0014] The sixth step is that when the ship sails to the edge of the small lens, the current small lens sends the tracking information of the to-be-tracked ship to the corresponding adjacent small lens through socket communication, and the tracking of the ship is continued on the adjacent small lens, and the process returns to the fifth step, and the cycle continues until the ship sails out of the monitoring range of the array camera.

[0015] Compared with the prior art, the method has the advantages that the construction characteristics of the camera itself are utilized, the position information and the feature vector of the ship are combined to determine whether two targets in different cameras are the same, and the accuracy of target tracking is improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows:

[0017] Figure 1 is a flowchart of the method for tracking the coastline ship in real time based on the array camera according to the present application.

[0018] Figure 2 is a structural schematic diagram of the overlapping area formed when the array camera collects images. DETAILED DESCRIPTION

[0019] The application will be described in greater detail with reference to the accompanying drawings, in which:

[0020] Referring to Figure 1 The application is a coastline ship real-time tracking method based on an array camera, which comprises the following steps:

[0021] In the first step, the number of small lenses and the overlapping area between adjacent small lenses are determined according to the distribution of small lenses in the array camera.

[0022] Specifically, a plurality of small lens arrays form an array camera. In this embodiment, 12 small lenses are taken as an example, and in other embodiments, any one of 3-100 small lenses can be taken as an example. The small lenses are numbered in order, and the images obtained by two adjacent small lenses are overlapped to form an overlapping area. The width of the overlapping area is determined according to the actual use condition, and is generally within 10% of the width of the small lens.

[0023] In the second step, all the ships to be tracked are preliminarily labeled with a rectangular frame TE in the small lenses.

[0024] Among all the obtained ship images, the ships considered to be the ships to be tracked are respectively assigned with a rectangular frame TE, and one or more rectangular frames TE are drawn in the monitoring picture to frame the M ships a-1, a-2, …, a-M to be tracked.

[0025] A large amount of data (including ship images) is collected, and the ships are manually labeled and respectively assigned with a rectangular frame TE. The labeled ships may or may not be the ships to be tracked, and the number of ships labeled with the rectangular frame TE is greater than the actual number of ships to be tracked.

[0026] As Figure 2 shown, it is a structural diagram of the overlapping area formed when the small lenses in the array camera collect images. The diagram of the array of 12 small lenses is shown, the adjacent small lenses of small lens A1 are A2 and A5; the adjacent small lenses of small lens A2 are A1, A3 and A6; and the adjacent small lenses of small lens A6 are A2, A5, A7 and A10. The overlapping area can be seen by both small lens A1 and small lens A2. The overlapping area between every two small lenses is not the same.

[0027] In the third step, a ship detection model is used to detect the image data of the labeled rectangular frame TE ships using N small lenses of the array camera, each labeled rectangular frame TE ship forms a detection frame TF, and a set of Box(x, y, w, h) and a set of feature vectors P are obtained, and the ship detection frame TF Box and the feature vector P are stored.

[0028] Specifically, in the S31 step, a large number of images containing ships are collected to obtain a ship sample set, which is sent to a convolutional neural network model optimized for ship targets by an SSD algorithm for training to obtain a network model CModel that can be used to identify ships.

[0029] In the S32 step, the CModel is transplanted to a TensorRT inference engine at the front end of a small lens, and each network layer of the CModel is converted into a network layer recognizable by the TensorRT inference engine at the front end of the small lens by comparing the network layers of the TensorRT inference engine. For network layers that cannot be recognized by the inference engine, a custom network layer is needed to tell the inference engine how to implement the layer. After transplantation, the network model CModel becomes CModel2.

[0030] In the S33 step, the CModel2 after transplantation is used to send the image data of the labeled rectangular frame TE ship collected by the small lens to form a detection frame TF, obtain a set of Box(x, y, w, h) of the detection frame TF and a set of feature vectors P, and detect the position of the labeled rectangular frame TE ship.

[0031] Wherein, x, y, w, h represent the horizontal coordinate, vertical coordinate, width and height of the top-left corner point of the detection frame, respectively; P represents the feature vector corresponding to the detection frame TF of the ship; the feature vector includes but is not limited to the shape features (color, category, etc.) of the ship.

[0032] The feature vector P is calculated as follows: first scale the detection frame TF into a 64*64 matrix, multiply it by a 64*1 ship detection matrix factor, and obtain a 64-dimensional feature vector P.

[0033] In the fourth step, in each small lens with a labeled rectangular frame TE ship, the intersection-over-union (IOU) of the detection frame TF and the rectangular frame TE of the ship to be tracked is compared, and the ship with an intersection-over-union (IOU) reaching a set value in the labeled ship to be tracked is tracked.

[0034] Only the detection frame TF with an intersection-over-union (IOU) very close to 1 is tracked.

[0035] Specifically, since the distance moved by the ship in a short time is limited, first remove those detection frames TF located outside a certain range around the rectangular frame TE of the ship to be tracked.

[0036] In the fifth step, the rectangular frame TE is tracked using the navigation information and appearance information of the ship to obtain a ship tracking sequence TraceQueue.

[0037] The tracking sequence TraceQueue refers to a set of detection boxes TF with the same ship number, and contains a ship number key and a set of detection boxes TFBox with the same key value.

[0038] Specifically, the rectangular box TE to be tracked is subjected to navigation prediction, the ship historical tracking sequence TraceQueue is used to predict the prediction rectangular box TE prediction representing the position of the ship at the current t1 moment, and the detection box TF set at the current moment t1 of the current small lens is matched, the matching mode is: first, remove those detection boxes TF located outside a specific range around T prediction, and then use the KM algorithm to find the optimal matching result from the remaining detection boxes TF. For the matched detection box TF target, the historical navigation information of the ship target is updated. The navigation information is initially considered to be static.

[0039] The navigation information includes but is not limited to the speed and direction of the ship navigation.

[0040] Obtaining appearance information: according to the feature vector P in the current small lens detection box TFBox, the Euclidean distance between the feature vector P and the feature vector P of the plurality of ship targets with the same ship number key in the tracking sequence TraceQueue is calculated, an average value is taken, and the average Euclidean distance is the smallest. It is considered to be the closest ship.

[0041] The addition of appearance information is to deal with the situation that the ship is blocked. When the ship is blocked, the tracking box obtained according to the navigation information may not be the ship to be tracked, so it is necessary to judge whether the ship is blocked according to the appearance information, and continue to track the ship after being blocked.

[0042] The sixth step is that when the ship travels to the edge of the small lens, the current small lens sends the information of the ship to be tracked to the corresponding adjacent small lens through socket communication, and continues to track the ship on the adjacent small lens, and goes to the fifth step, and the cycle is repeated until the ship leaves the monitoring range of the array camera.

[0043] The information of the ship to be tracked includes: the current small lens number, the ship number to be tracked, the coordinate converted in the adjacent small lens, the historical navigation information, the appearance information, etc.

[0044] The present application utilizes the characteristics of the array camera, and is very convenient for handling the cross-border head when tracking the ship in real time.

[0045] The array camera composed of multiple small lenses has a very wide monitoring range and a very long monitoring distance. The application can track the running track of each ship in a large scene across lenses based on the array camera.

[0046] Of course, those skilled in the art should recognize that the above embodiments are only used to illustrate the present application, and are not used as a limitation to the present application, as long as the changes and modifications of the above embodiments are within the scope of the spirit of the present application, they will fall within the scope of the claims of the present application.

Claims

1. A method for real-time tracking of ships along a coastline based on an array camera, characterized in that, It includes the following steps: The first step is to determine the lens number and the overlapping area where the image data acquired by adjacent lenses overlap, based on the distribution of the small lenses of the array camera. The second step is to initially mark the ships to be tracked among all the ships obtained from the small lens with rectangular boxes TE. The third step involves using a ship detection model and employing N small lenses from an array camera to detect the image data of ships with labeled rectangular boxes TE. Each labeled rectangular box TE forms a detection box TF, and a set of Box(x,y,w,h) and a set of feature vectors P are obtained and stored. The fourth step is to compare the intersection-union ratio (IOU) of the detection box (TF) and the rectangle TE of the ship to be tracked in each small shot of the ship with the labeled rectangle TE, and to track the ships whose IOU reaches the set value among the labeled ships to be tracked. The fifth step involves using the ship's navigation and appearance information to track the bounding box TE and obtain the ship tracking sequence TraceQueue. In the sixth step, when the ship reaches the edge of the small lens, the current small lens sends the information of the ship to be tracked to the corresponding adjacent small lens via socket communication. The adjacent small lens continues to track the ship, and then proceeds to the fifth step. This cycle continues until the ship leaves the monitoring range of the array camera.

2. The method for real-time tracking of ships along a coastline based on an array camera as described in claim 1, characterized in that, The third step mentioned above includes: In step S31, images containing ships are collected to obtain a ship sample set, which is then fed into the SSD algorithm to train a convolutional neural network model optimized for ship targets, resulting in a network model Cmodel that can be used to identify ships. In step S32, CModel is ported to the TensorRT inferencer at the front end of the small lens. By comparing the network layers of the TensorRT inferencer, each layer of the CModel network is converted into a network layer that the TensorRT inferencer at the front end of the small lens can recognize. For network layers that the inferencer cannot recognize, custom network layers need to be defined to tell the inferencer how to implement the layer. After the porting is completed, the network model CModel becomes CModel2. In step S33, the ship image data with labeled rectangular boxes TE collected by the small lens is sent into the ported CModel2 to form the detection box T.

3. The method for real-time tracking of ships along a coastline based on an array camera as described in claim 1, characterized in that, For the bounding box TE to be tracked, navigation prediction is performed. The predicted bounding box TE representing the ship's position at the current time t1 is predicted using the ship's historical tracking sequence TraceQueue. This prediction is then matched with the set of detection boxes TF at the current time t1 in the current small shot.

4. The method for real-time tracking of ships along a coastline based on an array camera as described in claim 3, characterized in that, The matching method is as follows: first, remove the detection boxes (TFs) that are outside a specific range around the prediction of T, and then use the KM algorithm to find the optimal matching result from the remaining detection boxes (TFs).

5. The method for real-time tracking of ships along a coastline based on an array camera as described in claim 1, characterized in that, In the fifth step above, the method for obtaining appearance information is as follows: based on the feature vector P in the Box of the current small lens detection box TF, calculate the Euclidean distance between the feature vector P and the feature vectors P of the most recently appearing multiple ship targets for each ship number key in the tracking sequence TraceQueue, take an average value, and the ship with the smallest average Euclidean distance is considered to be the closest ship.

Images