Automatic ship tracking method based on video image tracking and photoelectric linkage

Abstract

The invention discloses an automatic ship tracking method based on video image tracking and photoelectric linkage. The method comprises the following steps: S1, acquiring ship images and making a ship data set; s2, constructing a ship identification model and training the ship identification model; s3, identifying the monitored ship, and obtaining an identification tracking result; and S4, tracking the selected object based on the obtained identification tracking result. Through a ship automatic tracking function based on video image and photoelectric linkage, real-time dynamic information of a ship target in a water area can be mastered, automatic tracking recording is carried out on a focused ship, and decision-making assistance is provided for commanding and dispatching. The on-duty efficiency and accuracy of front-line on-duty personnel are greatly improved, and an effective auxiliary means for unified supervision, organization and coordination is provided for business personnel or cooperative members to jointly execute tasks such as water area supervision and the like.

Description

Technical Field

[0001] The invention relates to the technical field of ship tracking, in particular to an automatic ship tracking method based on video image tracking and photoelectric linkage. Background Art

[0002] With the advancement of video surveillance technology, cameras are increasingly being used in areas that require target surveillance, such as airports, highways, and ship channels, effectively supplementing traditional surveillance measures such as radar and AIS. However, in the field of situational awareness on the sea and rivers, the PTZ camera is mainly guided by the gun, and the target is tracked through the gun-ball linkage. There will be setbacks in the tracking process, and the ship target tracking will be delayed or even run out of the screen. There is a lack of autonomous recognition, perception, and autonomous locking of the ship target. Summary of the invention

[0003] In order to solve the above problems, the present invention provides a method for automatic ship tracking based on video image tracking and optoelectronic linkage, comprising the following steps: S1. collecting ship images and making a ship data set; S2. building a ship recognition model and training the ship recognition model; S3. identifying the monitored ship and obtaining the recognition and tracking results; S4. tracking the selected object based on the obtained recognition and tracking results.

[0004] Furthermore, the S1 step specifically includes the following sub-steps: S11. Collecting 20 types of ship pictures and performing image preprocessing; S12. Obtaining the ship target coordinate information and ship type information corresponding to each picture and exporting them as a recognition model training data set; S13. Collecting ship target videos, converting the videos into continuous pictures, and extracting the corresponding ship target coordinate information and ship type information according to steps S11 and S12 and exporting them as a feature extractor training data set.

[0005] Furthermore, the S2 step specifically includes the following sub-steps: S21. Use PyTorch to load the ship data set and configure relevant training parameters to optimize the training indicators; S22. Use the average precision mean and loss indicators to evaluate the training quality of the ship recognition model.

[0006] Furthermore, the relevant training parameters specifically include: batch-size, subdivisions, learning-rate, width, classes, epochs; the training indicators specifically include: GPU resource utilization, training time, and model accuracy.

[0007] Furthermore, the step S22 specifically includes the following steps: measuring the accuracy of the ship identification model by calculating the precision and recall rate of the ship identification model under different ship types and taking the average value; judging the training stability of the ship identification model by monitoring the changes in the loss index value.

[0008] Furthermore, the identification and tracking result specifically includes: target frame coordinates, target type, target ID, and predicted future center point coordinates of the target.

[0009] Further, the step S4 specifically includes the following sub-steps: S41. Continuously obtain the target frame coordinates of the identified target ship and the predicted future target frame coordinates, and record the target frame center point as P1 and the future target frame center point as P2; S42. Calculate the horizontal deflection angle △θ required for tracking the target under the current zoom magnification Z based on the pixel point horizontal distance W and pixel point vertical distance H between point P2 and the video center point P0 H and the vertical deflection angle △θ V ; S43. Adjust the parameters of the PTZ camera to track the target ship; S44. Calculate and adjust the zoom ratio Z of the PTZ camera n , zoom and track the target ship; in the above, the predicted future target frame coordinates are specifically: after 200ms, calculated at a frame rate of 25, that is, the target frame coordinates after 5 frames.

[0010] Furthermore, in the step S42, the horizontal deflection angle Δθ required for tracking the target under the current zoom magnification Z is calculated. H and the vertical deflection angle △θ V The calculation formula is: H = (W / H res ) × FOV' H ; △θ V =(H / V res ) × FOV' V , where V res Indicates the number of vertical pixels of the current camera image resolution; H res Indicates the number of horizontal pixels of the current camera image resolution; FOV' V = FOV V / Z,FOV' H = FOV H / Z,FOV V FOV H Respectively represent the vertical field of view and horizontal field of view of the ship monitoring camera. When the zoom magnification Z changes, the latest zoom magnification is Zn. At the Zn magnification, the horizontal field of view is FOV' H , vertical field of view is FOV'V .

[0011] Furthermore, in the step S44, the zoom factor Z is calculated. n The calculation formula of Z is: n = W res / (5•W O ), where W res Indicates the pixel width at the current resolution, W O Indicates the width of the target box recognized in the initial state.

[0012] Furthermore, the adjustment and control of the pan-tilt camera is specifically as follows: a multi-dimensional PID controller is introduced to control the deflection of the pan-tilt camera, and a linear interpolation is used to control the zoom ratio for PTZ control.

[0013] The present invention provides a method for automatic ship tracking based on video image tracking and photoelectric linkage, which has the following beneficial effects: The present invention can grasp the real-time dynamic information of ship targets in the waters through the ship automatic tracking function based on video images and photoelectric linkage, automatically track and record the ships of key concern, and provide decision-making assistance for command and dispatch. It greatly improves the duty efficiency and accuracy of front-line duty personnel, and provides an effective auxiliary means of unified supervision, organization, and coordination for various business personnel or collaborative members to jointly perform tasks such as water supervision. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0015] Figure 1 A flow chart of the method provided by the present invention; Figure 2 The change of Loss value in the training provided by the present invention; Figure 3 This is a schematic diagram of the optimized PTZ camera tracking control process provided by the present invention. DETAILED DESCRIPTION

[0016] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0017] The following is a detailed description of the implementation method of the present invention in conjunction with the accompanying drawings. Only some embodiments are described, not all embodiments. For the purpose of clarity, representations and descriptions that are not related to the present invention are omitted in the drawings and descriptions.

[0018] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below. Obviously, the implementation cases described are part of the embodiments of the present invention, not all of the embodiments, and cannot be understood as limiting the scope of the implementation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0019] The training is based on a large number of ship images. The training data consists of image data collected from real scenes, covering 20 types of ships such as fishing boats, speedboats, and passenger and vehicle ferries, and the training data has been augmented. The real-time detection framework of ship targets uses YoloV4-Tiny, and the target tracking algorithm uses DeepSORT. DeepSORT processes the output results of the YoloV4-Tiny ship detection model, assigns an ID to each target for tracking, and continuously predicts the position of the target; the monitoring software performs PID calculations based on the predicted position of the target, and outputs PTZ values ​​to smoothly control the ball camera to achieve continuous tracking and monitoring of the target.

[0020] like Figure 1 As shown, the present invention provides a method for automatic ship tracking based on video image tracking and optoelectronic linkage, comprising the following steps: S1. collecting ship images and making a ship data set; S2. building a ship recognition model and training the ship recognition model; S3. identifying the ship under monitoring and obtaining the recognition and tracking results; S4. tracking the selected object based on the obtained recognition and tracking results.

[0021] Step S1 specifically includes the following sub-steps: S11. Collect 20 types of ship images and perform image preprocessing to unify the image size to meet the needs of the dataset.

[0022] S12. Obtain the ship target coordinate information and ship type information corresponding to each image and export them as a recognition model training data set. The collected 20 types of ship images are labeled using Labelmg software to obtain the ship target coordinate information and ship type information corresponding to each image, and the labeled information is exported to generate a data set for ship recognition model training.

[0023] S13. Collect the ship target video, use FFmpeg software to convert the video into continuous pictures, and extract the corresponding ship target coordinate information and ship type information according to steps S11 and S12 and export them as a feature extractor training data set, which is used to train the DeepSORT feature extractor data set.

[0024] Step S2 specifically includes the following sub-steps: S21. Use PyTorch to load the ship data set and configure relevant training parameters to optimize the training indicators; wherein the relevant training parameters specifically include: batch-size, subdivisions, learning-rate, width, pretrained, gpu, data-dir, classes, train-label-path, epochs; the training indicators specifically include: GPU resource utilization, training time, and model accuracy.

[0025] S22. Use the average precision mean and loss index to evaluate the training quality of the ship recognition model: by calculating the precision and recall of the ship recognition model under different ship types and taking the average value to measure the accuracy of the ship recognition model; by monitoring the changes in the loss index value, the training stability of the ship recognition model is judged. The recognition and tracking results specifically include: target frame coordinates, target type, target ID, and predicted future center point coordinates of the target.

[0026] The mean average precision (mAP) and loss (Loss) indicators evaluate the quality of model training. mAP is an indicator for comprehensively evaluating the detection performance of the model. The accuracy of the model is measured by calculating the precision (Precision) and recall (Recall) of the model under different ship types and taking the average value; Loss is a value that measures the difference between the model prediction and the true label. By monitoring the change of the Loss value, it can be determined whether the training process is stable. Figure 2 As shown in the figure, mAP reaches 98%, Loss decreases rapidly in the early stage of training, and then gradually stabilizes. The model training results are in line with expectations, and the ship target recognition model file (file format pt) is obtained.

[0027] Use PyTorch to load the feature extractor training dataset, configure the training parameters, and obtain the DeepSORT feature extractor weight file (file format pt) after training. Enabling the feature extractor in DeepSORT can significantly improve the tracking performance.

[0028] Connect to the real-time video stream of the gimbal camera, call YoloV4 to process each frame of video image, and obtain the image recognition result (RecRet), which includes the target frame coordinates and target type information. Input RecRet into DeepSORT to obtain the recognition and tracking result (ObjRet), which includes the target frame coordinates, target type, target ID, and the predicted target's future (200ms later, calculated at a frame rate of 25, that is, 5 frames later) center point coordinates P2 (x2, y2).

[0029] Step S4 specifically includes the following sub-steps: S41. Continue to obtain the target frame coordinates of the identified target ship and the predicted future target frame coordinates (after 200ms, calculated at a frame rate of 25, i.e., the target frame coordinates after 5 frames), and record the center point of the target frame as P1 (x1, y1), and the center point of the future target frame as P2 (x2, y2).

[0030] S42. Calculate the horizontal deflection angle △θ required to track the target under the current zoom magnification Z based on the horizontal pixel distance W and the vertical pixel distance H between point P2 and the video center point P0. H and the vertical deflection angle △θ V :△θ H = (W / H res ) × FOV' H ; △θ V = (H / V res ) × FOV' V , where V res Indicates the number of vertical pixels of the current camera image resolution; H res Indicates the number of horizontal pixels of the current camera image resolution; FOV' V = FOV V / Z,FOV' H = FOV H / Z,FOV V FOV H Respectively represent the vertical field of view and horizontal field of view of the ship monitoring camera. When the zoom magnification Z changes, the latest zoom magnification is Zn. At the Zn magnification, the horizontal field of view is FOV' H , vertical field of view is FOV' V .

[0031] S43. Adjust the parameters of the PTZ camera to track the target ship. H Input the horizontal direction PID controller to obtain the horizontal deflection angle variable P1 required to drive the gimbal camera to track smoothly; V Input the vertical direction PID controller to obtain the vertical deflection angle variable T1 required to drive the gimbal camera to track smoothly; the new P value of the gimbal camera is the current P value + P1, recorded as P', and the new T value is the current T value + T1, recorded as T'. Call the gimbal camera API to set the P value and T value to P' and T' respectively to complete the camera's tracking of the target.

[0032] S44. Calculate and adjust the zoom ratio Z of the gimbal camera n , zoom and track the target ship: Z n = W res / (5•W O ), where Wres Indicates the pixel width at the current resolution, W O Indicates the width of the target frame identified in the initial state. In the ship target tracking business scenario, because the distance between the ship and the camera is far, the ship target is too small, which will make it impossible to effectively identify the ship information such as the ship number, so it is also necessary to automatically adjust the zoom magnification Z value so that the ship target can be clearly displayed in the picture. The current algorithm temporarily determines that the target frame width needs to be kept at 1 / 5 of the picture width. The pixel width at the current resolution is W res , in the initial state (Z=1), the width of the target box recognized is W O , calculate the target Z value (Z n ), call the gimbal camera API to set the Z value and complete the camera's zoom tracking of the target.

[0033] Continue to receive the ship target identification and tracking results input by the identification module, repeat the above PTZ control steps to keep the pan-tilt camera tracking the target until exiting the tracking state (PT parameters exceed the tracking threshold or the monitoring software cancels the tracking command).

[0034] The adjustment and control of the PTZ camera is specifically as follows: a multi-dimensional PID controller is introduced to control the deflection of the PTZ camera, and the zoom ratio is controlled by linear interpolation for PTZ control.

[0035] PID controller is a common feedback mechanism in control systems, used to adjust and control the system output to make it close to the target value. PID stands for Proportional, Integral, and Derivative control. Each part corrects the error in a different way. The output formula of PID controller is as follows: u(t) = K p ∙e(t) + K i ∙ +K d ∙ .

[0036] Using the predicted future (200ms later, 25 frames per second, or 5 frames per second) target frame center coordinates P2 (x2, y2) from the recognition results to guide the PTZ camera's steering can solve the problem of the camera center always lagging behind the moving target when using the target's current position to guide the steering. Calculate the horizontal deflection angle △θ from P2 to the camera screen center point P0 (x0, y0) H , calculate the vertical deflection angle △θ from P2 to P0 V , create two PID controllers, the horizontal PID controller converts △θ H As the horizontal error input e x, the horizontal deflection angle variable P1 required for smooth tracking of the PTZ camera is calculated; the vertical direction PID controller converts △θ V As the vertical error input e y , calculate the vertical deflection angle variable T1 required for smooth tracking of the PTZ camera: control x = K p,x ∙e x + K i,x ∙ +K d,x ∙ ; control y = K p,y ∙e y + K i,y ∙ +K d,y ∙ .

[0037] To prevent the PID output from overshooting, causing drastic changes in the image and tracking failure, the PID output needs to be limited. Set the maximum deflection angle P according to the actual situation. max and T max :P max =PIX h × ; T max =PIX v × ; Where, PIX h : Maximum horizontal deflection allowed in pixels; PIX v : Maximum vertical deflection allowed in pixels; FOV h : The camera's initial horizontal field of view; FOV v : Camera initial vertical field of view; Z: Current zoom ratio; PIX w : The pixel width of the current image (such as 1920 pixels); PIX h : The pixel height of the image (such as 1080 pixels).

[0038] K p The parameters need to be adjusted dynamically as the current Z value changes to prevent the output of invalid values ​​that are too large.

[0039] To prevent PID from frequently starting control and causing continuous screen jitter, you need to set the effective threshold and failure threshold for PID. When the calculated position error is less than the failure threshold, PID will no longer control until the position error exceeds the effective threshold again, then PID will be reactivated. The setting unit of the threshold is pixel distance.

[0040] When setting the Z value, you need to determine the Z to be set nIs the difference between the current value and Z0 greater than the smoothing step value Z? s (Currently set to 1 in the software). If it exceeds this value, an interpolation algorithm is required to generate the intermediate Z values ​​(Z1, Z2…Z n ), and set it up step by step to complete smooth scaling and prevent target loss.

[0041] The complete tracking control process of the optimized PTZ camera is as follows: Figure 3 As shown in the figure, W is the horizontal pixel distance between the center point P2 (x2, y2) of the target prediction position and the center point P0 (x0, y0) of the camera screen, H is the vertical pixel distance between the center point P2 (x2, y2) of the target prediction position and the center point P0 (x0, y0) of the camera screen, and ObjRet is the target recognition and tracking result continuously input by the recognition module.

[0042] The present invention can grasp the real-time dynamic information of ship targets in the waters through the ship automatic tracking function based on video images and photoelectric linkage, automatically track and record the ships of key concern, and provide decision-making assistance for command and dispatch. It greatly improves the duty efficiency and accuracy of front-line duty personnel, and provides an effective auxiliary means of unified supervision, organization, and coordination for various business personnel or collaborative members to jointly perform tasks such as water supervision.

[0043] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A ship automatic tracking method based on video image tracking and photoelectric linkage, characterized in that: The following steps are involved: S1. Collect ship images and create ship datasets; S2. construct a ship identification model and train the ship identification model; S3. Use the PTZ camera to capture the ship video information and identify the monitored ship to obtain the identification and tracking results; S4. Track the selected object based on the obtained recognition and tracking results, and adjust the gimbal camera in real time.

2. The method for automatic ship tracking based on video image tracking and photoelectric linkage according to claim 1 is characterized in that: The S1 step specifically includes the following sub-steps: S11. Collect 20 types of ship images and perform image preprocessing; S12. Obtain the ship target coordinate information and ship type information corresponding to each image and export it as a recognition model training data set; S13. Collect the ship target video, convert the video into continuous pictures, and extract the corresponding ship target coordinate information and ship type information according to steps S11 and S12 and export them as a feature extractor training data set.

3. The method for automatic ship tracking based on video image tracking and photoelectric linkage according to claim 1, characterized in that: The S2 step specifically includes the following sub-steps: S21. Use PyTorch to load the ship dataset and configure relevant training parameters to optimize the training indicators; S22. Evaluate the training quality of the ship recognition model using mean average precision and loss metrics.

4. The method for automatic ship tracking based on video image tracking and photoelectric linkage according to claim 3 is characterized in that: The relevant training parameters specifically include: batch-size, subdivisions, learning-rate, width, classes, and epochs; the training indicators specifically include: GPU resource utilization, training time, and model accuracy.

5. The method for automatic ship tracking based on video image tracking and photoelectric linkage according to claim 3 is characterized in that: The step S22 specifically includes the following steps: measuring the accuracy of the ship identification model by calculating the precision and recall rate of the ship identification model under different ship types and taking the average value; and judging the training stability of the ship identification model by monitoring the change of the loss index value.

6. The method for automatic ship tracking based on video image tracking and photoelectric linkage according to claim 1, characterized in that: The identification and tracking results specifically include: target frame coordinates, target type, target ID, and predicted future center point coordinates of the target.

7. The method for automatic ship tracking based on video image tracking and photoelectric linkage according to claim 6 is characterized in that: The S4 step specifically includes the following sub-steps: S41. Continuously obtain the target frame coordinates and the predicted future target frame coordinates of the identified target ship, and record the center point of the target frame as P1 and the center point of the future target frame as P2; S42. Calculate the horizontal deflection angle △θ required to track the target under the current zoom magnification Z based on the horizontal pixel distance W and the vertical pixel distance H between point P2 and the video center point P0. H and the vertical deflection angle △θ V ; S43. Adjust the parameters of the PTZ camera to track the target ship; S44. Calculate and adjust the zoom ratio Z of the gimbal camera n , zoom and track the target ship; In the above, the predicted future target frame coordinates are specifically: 200ms later, calculated at a frame rate of 25, that is, the target frame coordinates 5 frames later.

8. The method for automatic ship tracking based on video image tracking and photoelectric linkage according to claim 7 is characterized in that: In the step S42, the horizontal deflection angle Δθ required for tracking the target under the current zoom magnification Z is calculated. H and the vertical deflection angle △θ V The calculation formula is: H = (W / H res ) × FOV' H ; △θ V = (H / V res ) × FOV' V , where V res Indicates the number of vertical pixels of the current camera image resolution; H res Indicates the number of horizontal pixels of the current camera image resolution; FOV' V = FOV V / Z,FOV' H = FOV H / Z,FOV V FOV H Respectively represent the vertical field of view and horizontal field of view of the ship monitoring camera. When the zoom magnification Z changes, the latest zoom magnification is Zn. At the Zn magnification, the horizontal field of view is FOV' H , vertical field of view is FOV' V .

9. The method for automatic ship tracking based on video image tracking and photoelectric linkage according to claim 7, characterized in that: In step S44, the zoom factor Z is calculated. n The calculation formula of Z is: n = W res / (5·W O ), where W res Indicates the pixel width at the current resolution, W O Indicates the width of the target box recognized in the initial state.

10. The method for automatic ship tracking based on video image tracking and photoelectric linkage according to claim 7, characterized in that: The adjustment and control of the pan-tilt camera is specifically as follows: a multi-dimensional PID controller is introduced to control the deflection of the pan-tilt camera, and a linear interpolation is used to control the zoom ratio for PTZ control.

Images