Ship gear identification method and device based on target tracking and re-identification, and medium

Abstract

The invention relates to a ship gear identification method and device based on target tracking and re-identification, and a medium. The method comprises the following steps: obtaining ship information; target identification is carried out based on YOLO5s introduced with an extended Kalman filtering algorithm, a ship trajectory is tracked from ship entry, and a ship detection frame of a trajectory tracking result and ship basic information are superposed on a ship real-time video; if the ship trajectory is lost due to shielding in the ship trajectory tracking process, determining a lost target trajectory based on a Deep OC-SORT algorithm and a Redis re-recognition technology; in the ship trajectory tracking process, if it is recognized that the ship does not move for continuous preset time, it is judged that ship gear stopping is completed, the ship stopping position is obtained, and a ship actual gear diagram is formed; and judging whether the ship is scheduled according to the ship scheduling graph, the actual scheduling graph and the basic information of the ship, and if the ship is not scheduled, giving an alarm. Compared with the prior art, the method has the advantages that accurate positioning of the ship position can be achieved, and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of ship management, and in particular to a ship gear recognition method, device and medium based on target tracking and re-recognition. Background Art

[0002] Ship gearing refers to the orderly docking of ships in the lock chamber when passing through the lock. Generally, the lock will formulate a corresponding ship scheduling and gearing plan based on the ship's lock reporting situation. By comparing the scheduling and gearing plan with the actual gearing situation, the lock staff can know whether the ship has been geared according to the scheduling. For ships that are not geared according to the gearing diagram, the staff can promptly notify the ship to adjust. At present, most of the patents related to ship gearing are mainly focused on the optimization of ship scheduling and gearing methods. For example, patent CN110110403B proposes a scheduling and gearing method suitable for one-way continuous passing ships, but does not propose a method for identifying the actual gearing of the ship. If the actual gearing of the ship is inconsistent with the scheduling gearing, manual comparison and judgment are required, which is time-consuming and labor-intensive. Summary of the invention

[0003] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a ship gear identification method, equipment and medium based on target tracking and re-identification, aiming to track the ship passing through the lock and identify the actual gear position of the ship, so as to realize rapid detection and early warning of whether the ship is in the scheduled gear.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] According to a first aspect of the present invention, a ship gear identification method based on target tracking and re-identification is provided, the method comprising the following steps:

[0006] Obtaining ship information, including basic ship information, ship dispatch diagram, ship pictures and real-time ship video;

[0007] Target recognition is performed based on YOLO5s with the extended Kalman filter algorithm. The ship's trajectory is tracked from the moment it enters the lock, and the basic information of the ship and the ship's picture are matched. The ship detection frame and basic information of the trajectory tracking result are superimposed on the real-time video of the ship.

[0008] If occlusion occurs during the ship trajectory tracking process, resulting in the loss of the ship trajectory, the Deep OC-SORT algorithm is used to predict the position of the next frame or the next few frames of the target lost in the real-time video, and the lost target is searched near the predicted position; when one or more targets are found near the predicted position, the IoU calculation is performed between the content in the ship detection frame obtained by trajectory tracking and the Redis re-identification technology. When the IoU value is greater than the preset threshold, the found target and the target in the corresponding ship detection frame are regarded as the same target;

[0009] During the ship trajectory tracking process, if it is identified that the ship has not moved for a preset time, the ship gear shifting and docking is judged to be completed, the ship's docking position is obtained and the actual ship gear shifting map is formed;

[0010] Determine whether the ship is scheduled according to the ship scheduling diagram, actual gear diagram and basic ship information. If the ship is not scheduled according to the scheduling, an alarm will be issued.

[0011] As a preferred technical solution, the target recognition based on YOLO5s introduced with extended Kalman filter algorithm specifically includes the following steps:

[0012] Initialize the state of the extended Kalman filter tracker: use the frame where YOL0v5s target detection fails as the third frame of the extended Kalman filter tracker, use the detection values ​​of the first two frames as the initial state of the tracker, and initialize the state vector and error covariance matrix of the second frame;

[0013] State prediction: predict the current state vector and state error covariance matrix based on the previous state;

[0014] Calculate the extended Kalman gain: define the observation vector, calculate the extended Kalman gain through the daily matrix and the prior results obtained in the prediction stage;

[0015] Update: Update the state vector and error covariance matrix according to the extended Kalman gain to achieve continuous tracking of the target;

[0016] Candidate region creation and matching: Based on the position of the prior system state bounding box of the extended Kalman filter, the region is expanded to obtain the candidate region, and the difference value hash algorithm is used to match the candidate region, and the candidate region with the smallest difference is selected as the observation value.

[0017] As a preferred technical solution, the state vector x and error covariance matrix P of the second frame are as follows:

[0018]

[0019] in and Represents the coordinates of the upper left corner and lower right corner of the bounding box on the x-axis and y-axis respectively. The subscript m represents the observed value, and the subscript 2 of m represents the second frame. and To calculate the result, ΔT is the time taken from the first frame to the second frame, cent(x) and cent(y) represent the centroid coordinates of the bounding box on the x-axis and y-axis, respectively.

[0020] As a preferred technical solution, the extended Kalman filter updates the state vector and the error covariance matrix according to the extended Kalman gain:

[0021]

[0022] Among them, K k is the extended Kalman gain, is the prior state vector, is the prior error covariance matrix, H is the Jacobian matrix, and Z k is the observation vector, Represents the nonlinear transformation of the state vector to the measurement vector:

[0023]

[0024] As a preferred technical solution, the candidate area is created by moving the position of the prior system state bounding box of the extended Kalman filter in eight directions, the directions being 45° to the upper left, 45° to the upper right, 45° to the lower right, 45° to the lower left, and directly above, directly below, directly to the right, and directly to the left. The degree of movement is represented by the intersection-over-union ratio IoU, the bounding box of the prior system state is represented by E, the bounding box after the move is represented by F, and the intersection-over-union ratio IoU is defined as:

[0025]

[0026] The degree of movement is determined by presetting the intersection-and-union ratio value, and the expansion areas in eight directions are obtained. Combined with the prior system state bounding box position, nine candidate areas are obtained.

[0027] As a preferred technical solution, the use of the difference value hash algorithm to match candidate regions is specifically as follows:

[0028] Reduce the expanded candidate area image to n×(n-1), where n is a preset value;

[0029] Convert the reduced image into a grayscale image, assign 1 to each point in each row where the previous pixel value is greater than the next pixel value, and 0 otherwise, to generate a hash value;

[0030] Calculate the Hamming distance between the candidate region and the image in the target bounding box of the previous frame. The smaller the distance, the smaller the difference in the images. Select the candidate region with the smallest difference as the observation value.

[0031] As a preferred technical solution, the Deep OC-SORT algorithm performs the following steps:

[0032] Get the scaling rotation matrix Mt=s t R t and the translation matrix T t ,in s t represents the scaling matrix, R t represents the rotation matrix;

[0033] Introduce the scaling rotation matrix and translation matrix into three components of the OC-SORT algorithm: OOS, OCM and OCR:

[0034] For the OOS component, the Kalman filter is updated along a linear interpolation path starting from the last known measurement, which is given by [x c ,y c , a, s], the first two items are the center of the bounding box, and the center of the bounding box is c<-M t c+T t Transformed to obtain, where a is the area and s is the aspect ratio;

[0035] For the OCM component, let p1, p2 be the upper left and lower right corners of the bounding box. OCM uses the last Δt=3 bounding box to calculate the bounding box angular velocity. At each time step t, the transformation p is applied to the bounding box. i ←M t p i +T t ;

[0036] For the OCR component, for the last seen bounding box position in OCR, at each time step t, apply p i ←M t p i +T t to adjust its position;

[0037] Under the OC-SORT algorithm, the Kalman state is The Kalman state is adjusted as follows by camera motion compensation:

[0038]

[0039] As a preferred technical solution, low detector confidence is used to identify image degradation caused by occlusion or blur. tis the tracklet appearance embedding at time t, the standard EMA is:

[0040] e t =α t e t-1 +(1-α t ) new

[0041] e new is the appearance of the matching detection added to the model, α t Defined as:

[0042]

[0043] Among them, s det is the detector confidence, σ is the detection confidence threshold used to filter noisy detections, and α f is a fixed value;

[0044] Adaptive weighting: Use the standard cosine similarity between track and box embeddings to get an M×N appearance cost matrix A c , where M and N are the number of trajectories and detections, respectively, and A c [m, n] represents the item at the intersection of the mth row and the nth column, which is combined with the IoU cost matrix to obtain C = I c +a w A c , and the weighting factor w b (m, n) is added to the global a w In the above example, let T m Indicates tracking, d n Indicates detection, when T m When only one box has a high similarity score, in A c [m,:] increases the appearance weight if the detection d n has only one distinct association with a track, then the same operation is applied to A c Columns, using z diff To measure the discriminability of the detection box-track pair, z diff Defined as the difference between the highest and second highest values ​​in a row or column:

[0045]

[0046] Among them, ∈ is a hyperparameter;

[0047] According to z diff The weighting factors are derived, resulting in the final cost matrix C:

[0048]

[0049] C[m,n]=IoU[m,n]+[a w +w b (m,n)]A c [m,n].

[0050] The missing target search is achieved by minimizing the cost matrix C.

[0051] According to a second aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0052] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described above when executed by a processor.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] (1) The present invention provides a ship gear identification method based on target tracking and re-identification technology, which tracks the ship's trajectory from the time the ship enters the lock until the ship docks and locks the ship's position, forming an actual ship gear map. By comparing the actual gear map with the scheduled gear map, it is confirmed whether the ship is in the scheduled gear. For ships with mismatched gear coordinates, an alarm is automatically issued, saving manpower and time for manual gear identification and improving efficiency.

[0055] (2) During the ship tracking process, if a ship cannot be continuously tracked due to being blocked from behind, the Deep OC-SORT algorithm of the present invention predicts the possible position of the blocked ship in several frames and retrieves the lost ship through target re-identification technology. Redis re-identification can quickly and accurately compare the lost target with the re-identified target to assist in the one-to-one retrieval of the lost ship.

[0056] (3) The present invention adds an alarm mechanism, which issues an alarm to ships that fail to follow the scheduling schedule, reminding ship business personnel to handle the situation in a timely manner, thereby realizing the linkage of the entire process of ship passing through the lock, and thoroughly improving the scheduling efficiency of the lock and the efficiency of ship passing through the lock. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of the method of the present invention;

[0058] Figure 2 is a flow chart of a ship trajectory tracking algorithm of the present invention;

[0059] Figure 3 It is a schematic diagram of the system structure of the present invention;

[0060] Figure 4 is the actual trajectory of the target in one embodiment;

[0061] Figure 5 is a change in the distance between the target and the observation origin in one embodiment;

[0062] Figure 6 The change in the orientation of the target and the observation origin in one embodiment;

[0063] Figure 7 A comparison between the actual value and the estimated value of the target x-coordinate in an embodiment;

[0064] Figure 8 A comparison between the actual value and the estimated value of the target y coordinate in an embodiment;

[0065] Fig. 9 A comparison between the actual value and the estimated value of the target x-direction velocity in an embodiment;

[0066] Fig.10 A comparison between the actual value and the estimated value of the target y-direction velocity in an embodiment;

[0067] Fig.11 A comparison between the actual trajectory and the estimated trajectory of a target in an embodiment;

[0068] Fig.12 is an extended Kalman filter position estimation bias in one embodiment;

[0069] Fig.13 An extended Kalman filter velocity estimation bias in one embodiment;

[0070] Fig.14 A comparison between a real trajectory and an estimated trajectory after changing the initial state estimation in an embodiment;

[0071] Fig.15 The position and velocity deviation of the target tracking after changing the initial estimation state in one embodiment;

[0072] Fig.16 The actual trajectory and estimated trajectory of the target tracking after adding the time point in one embodiment;

[0073] Fig.17 In one embodiment, the position and speed deviation of the target tracking after adding the time point;

[0074] Fig.18 is a target real trajectory after increasing process noise in one embodiment;

[0075] Fig.19 The position and velocity deviation of the target tracking after increasing the process noise in one embodiment;

[0076] Fig. 20A comparison of target estimation and actual distance and azimuth changes after increasing measurement noise in an embodiment;

[0077] Fig.21 A comparison between the actual and estimated trajectories of a target after increasing the measurement noise in one embodiment. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0079] Example 1

[0080] This embodiment provides a ship gear recognition method based on target tracking and re-recognition, such as Figure 1 As shown, the method comprises the following steps:

[0081] S1, obtaining ship information, wherein the ship information includes basic ship information, ship dispatch diagram, ship picture and ship real-time video.

[0082] S2, based on the YOLO5s with extended Kalman filter algorithm, performs target recognition, tracks the ship's trajectory from the moment it enters the lock, and uses AR video technology to match the ship's basic information and ship pictures, superimposing the ship detection frame and basic ship information of the trajectory tracking results on the ship's real-time video.

[0083] like Figure 2 As shown, the YOLO5s algorithm that introduces the extended Kalman filter algorithm performs the following steps:

[0084] S21, initialize the state of the extended Kalman filter tracker: use the frame where the YOL0v5s target detection fails as the third frame of the extended Kalman filter tracker, use the detection values ​​of the first two frames as the initial state of the tracker, and initialize the state vector x and error covariance matrix P of the second frame:

[0085]

[0086] in and Represents the coordinates of the upper left corner and lower right corner of the bounding box on the x-axis and y-axis respectively. The subscript m represents the observed value, and the subscript 2 of m represents the second frame. and To calculate the result, ΔT is the time taken from the first frame to the second frame, cent(x) and cent(y) represent the centroid coordinates of the bounding box on the x-axis and y-axis, respectively.

[0087]

[0088] S22, state prediction: predict the current state vector and state error covariance matrix based on the previous state.

[0089]

[0090] in is the prior state vector, is the prior error covariance matrix.

[0091] S23, calculating the extended Kalman gain, which can determine how much influence the measured quantity has on the system state prediction.

[0092] Define the observation vector:

[0093]

[0094] in and The calculation formula is as follows:

[0095]

[0096] For the extended Kalman filter, the relationship between the state vector and the measurement vector is nonlinear. Due to this nonlinear relationship, the H matrix is ​​different from the traditional Kalman filter. The H matrix is ​​a Jacobian matrix.

[0097]

[0098] The extended Kalman gain K is calculated using the daily matrix and the prior results obtained in the prediction phase. k :

[0099]

[0100] S24, Update: Update the state vector and error covariance matrix according to the extended Kalman gain to achieve continuous tracking of the target:

[0101]

[0102] in Represents the nonlinear transformation of the state vector to the measurement vector.

[0103]

[0104] Appropriate Q and R are conducive to improving the performance of the extended Kalman filter. In the Japanese target tracking, Q and R rely on empirical adjustment. Therefore, we conducted three sets of experiments to optimize the performance of the extended Kalman filter tracker. In each set of experiments, the value of Q remained fixed, while R was changed according to the diagonal elements taking 1, 0.1 or 0.01 values, and multiple experiments were conducted. The final selected Q and R values ​​are as follows:

[0105]

[0106] S25, candidate region creation and matching: region expansion is performed based on the position of the prior system state bounding box of the extended Kalman filter to obtain candidate regions, and the difference value hash algorithm is used to match the candidate regions, and the candidate region with the smallest difference is selected as the observation value.

[0107] According to the position of the prior system state bounding box of the extended Kalman filter, it is moved in eight directions, namely 45° to the upper left, 45° to the upper right, 45° to the lower right, 45° to the lower left, and directly above, directly below, directly to the right, and directly to the left. The degree of movement is represented by the intersection over union (IoU). The bounding box of the prior system state is represented by E, and the bounding box after the movement is represented by F. The intersection over union (IoU) is defined as:

[0108]

[0109] The degree of movement is determined by presetting the intersection-over-union ratio to 0.8, and the expansion areas in eight directions are obtained. Nine candidate areas are obtained by combining the prior system state bounding box positions.

[0110] Reduce the expanded candidate area image to n×(n-1), where n is a preset value;

[0111] Convert the reduced image into a grayscale image, assign 1 to each point in each row where the previous pixel value is greater than the next pixel value, and 0 otherwise, to generate a hash value;

[0112] Calculate the Hamming distance between the candidate region and the image in the target bounding box of the previous frame. The smaller the distance, the smaller the difference in the images. Select the candidate region with the smallest difference as the observation value.

[0113] S3, if occlusion occurs during the ship trajectory tracking process and causes the ship trajectory to be lost, based on the Deep OC-SORT algorithm, the position of the next frame or the next few frames of the lost target in the real-time video is predicted, and the lost target is searched near the predicted position.

[0114] The Deep OC-SORT algorithm performs the following steps:

[0115] Get the scaling rotation matrix M t =s t Rt and the translation matrix T t ,in s t represents the scaling matrix, R t represents the rotation matrix;

[0116] Since the lost target prediction is prone to quadratic mixing errors over time, the scaling rotation matrix and the translation matrix are introduced into the three components of the OC-SORT algorithm: OOS (online smoothing), OCM (momentum), and OCR (recovery):

[0117] For the OOS component, the Kalman filter is updated along a linear interpolation path starting from the last known measurement, which is given by [x c ,y c , a, s], the first two items are the center of the bounding box, and the center of the bounding box is c<-M t c+T t Transformed to obtain, where a is the area and s is the aspect ratio;

[0118] For the OCM component, let p1, p2 be the upper left and lower right corners of the bounding box. OCM uses the last Δt=3 bounding box to calculate the bounding box angular velocity. At each time step t, the transformation p is applied to the bounding box. i ←M t p i +T t ;

[0119] For the OCR component, for the last seen bounding box position in OCR, at each time step t, apply p i ←M t p i +T t to adjust its position;

[0120] Under the OC-SORT algorithm, the Kalman state is The Kalman state is adjusted as follows by camera motion compensation:

[0121]

[0122] Using low detector confidence to identify image degradation due to occlusion or blur, let e t is the tracklet appearance embedding at time t, the standard EMA is:

[0123] e t =α t e t-1 +(1-α t ) new

[0124] enew is the appearance of the matching detection added to the model, α t Defined as:

[0125]

[0126] Among them, s det is the detector confidence, σ is the detection confidence threshold used to filter noisy detections, and α f is a fixed value, which is set to 0.95 in this embodiment.

[0127] Adaptive weighting: Use the standard cosine similarity between track and box embeddings to get an M×N appearance cost matrix A c , where M and N are the number of trajectories and detections, respectively, and A c [m, n] represents the item at the intersection of the mth row and the nth column, which is combined with the IoU cost matrix to obtain C = I c +a w A c , and the weighting factor w b (m, n) is added to the global a w In the above example, let T m Indicates tracking, d n Indicates detection, when T m When only one box has a high similarity score, in A c [m,:] increases the appearance weight if the detection d n has only one distinct association with a track, then the same operation is applied to A c Columns, using z diff To measure the discriminability of the detection box-track pair, z diff Defined as the difference between the highest and second highest values ​​in a row or column:

[0128]

[0129] Among them, ∈ is a hyperparameter;

[0130] According to z diff The weighting factors are derived, resulting in the final cost matrix C:

[0131]

[0132] C[m,n]=IoU[m,n]+[a w +w b (m,n)]A c [m,n].

[0133] The missing target search is achieved by minimizing the cost matrix C.

[0134] S4, after one or more targets are found near the predicted position, IoU calculation is performed on the contents in the ship detection frame obtained by Redis re-identification technology and trajectory tracking. When the IoU value is ≥ 0.8, the found target and the target in the corresponding ship detection frame are regarded as the same target.

[0135] S5, during the ship trajectory tracking process, if it is recognized that the ship has not moved for 10 frames, it is determined that the ship gear shifting and docking is completed, the ship's docking position is obtained and the actual ship gear shifting map is formed.

[0136] S6, judging whether the ship is scheduled according to the ship scheduling diagram, the actual scheduling diagram and the basic information of the ship, and if the ship is not scheduled according to the scheduling, an alarm is issued.

[0137] Example 2

[0138] The above is an introduction to the method embodiment. The following is a further explanation of the solution of the present invention through a system embodiment.

[0139] like Figure 3 As shown, the system includes:

[0140] Data storage module: connected with the algorithm processing module, the ship positioning module and the lock control system, used to store relevant data and send data information to relevant modules;

[0141] Algorithm processing module: used to perform the following steps: obtain ship information, which includes basic ship information, ship scheduling diagram, ship pictures and real-time video of the ship; perform target recognition based on YOLO5s that introduces the extended Kalman filter algorithm, track the ship trajectory from the time the ship enters the lock, match the basic ship information and ship pictures, and superimpose the ship detection frame and basic ship information of the trajectory tracking result on the real-time video of the ship; if occlusion occurs during the ship trajectory tracking process and causes the ship trajectory to be lost, then based on the Deep OC-SORT algorithm, predict the position of the next frame or the next few frames of the target lost in the real-time video, and search for the lost target near the predicted position; when one or more targets are found near the predicted position, the IoU calculation is performed on the content of the ship detection frame obtained by Redis re-identification technology and trajectory tracking. When the IoU value is greater than the preset threshold, the found target and the target in the corresponding ship detection frame are regarded as the same target; during the ship trajectory tracking process, if it is recognized that the ship has not moved for a preset time, it is judged that the ship gear docking is completed, and the actual gear map of the ship is formed;

[0142] Ship positioning module: used to obtain the ship's berthing position when judging that the ship has completed berthing;

[0143] Lock control system: used to compare the gear map according to the scheduling gear map sent by the data storage module, the actual gear map, and the basic information of the ship, to determine whether the ship is scheduled according to the scheduling gear map. If the ship does not follow the scheduling gear map, an automatic alarm will be issued to remind the staff.

[0144] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0145] Example 3

[0146] This embodiment performs simulation on the method described in Embodiment 1 to verify the effectiveness of the present invention.

[0147] 1. Basic results display

[0148] Configure and use the extended Kalman filter to complete the filter estimation of the system model. When drawing the curve here, the range of time n is determined to be 60. In order to make a better evaluation of the model at the beginning, the initial state is set to be the same as the extended Kalman filter state. Figure 4 In the figure, the horizontal axis is r x [N], the vertical axis is r y [N], by Figure 4 It can be seen that the target's motion trajectory is roughly a straight line. The target state equation also shows that the target is in uniform linear motion, and its x- and y-axis coordinate values ​​are proportional to the speed. The speed remains unchanged. The target is in uniform linear motion, and will be subject to random interference from noise, causing fluctuations. The reason is that a certain amount of process noise is added to its state equation.

[0149] Figure 5 is the distance change between the target and the observation origin, the horizontal axis is n, and the vertical axis is Figure 6 is the change in the azimuth between the target and the observation origin, the horizontal axis is n, and the vertical axis is according to Figure 5 , 6 It can be seen that as the target moves, the azimuth angle is increasing, and it can be seen that the target is gradually moving upward. EKF also has a good tracking effect on the movement of the target.

[0150] Figure 7 The horizontal axis is n, and the vertical axis is r determined by the state equation. y [N], vertical axis 2 is estimated by EKF Figure 8 Comparison between the true value and estimated value of the target y coordinate, the horizontal axis is n, the vertical axis 1 is ry[N] determined by the state equation, and the vertical axis 2 is estimated by EKF according to Figure 7, 8 ,Comparing the x and y coordinate values ​​between the state value after EKF filtering and the target's true state, we can see that, regardless of the x or y coordinate value, the EKF estimation curve can track the x and y coordinate values ​​of the target well. However, as time goes by, the tracking effect of EKF gradually deteriorates.

[0151] Fig. 9 Comparison between the actual and estimated values ​​of the target x-direction velocity. The horizontal axis is n, and the vertical axis is v determined by the state equation. k [N], vertical axis 2 is estimated by EKF Fig.10 Comparison between the actual and estimated values ​​of the target y-direction velocity, with n on the horizontal axis and v on the vertical axis determined by the state equation. r [N], vertical axis 2 is estimated by EKF according to Fig. 9 , 10 It can be seen that the estimation errors of the speed in the x and y directions are kept within 0.5 within 60 time points, and the estimated values ​​within the 60 time points are relatively close to the true values.

[0152] 2. Error Analysis

[0153] From the tracking trajectory Fig.11 It can be seen from the figure that the initial estimation effect of the EKF algorithm is good, but the estimation effect is average, which can be said to be poor. This is mainly because the state is four-dimensional information, while the observation is only two-dimensional angle information, and the angle has a nonlinear relationship with x and y in the state. It is very difficult for the EKF algorithm to get a good result. As time goes by, the Kalman estimate will gradually deviate from the true trajectory of the target.

[0154] from Fig.12 and 13 It can be seen that after several iterations of the algorithm, the error of the EKF estimation becomes larger and larger, and finally the EKF algorithm is almost divergent. In a nonlinear system, it is very difficult to achieve continuous tracking of the target based on the initial state and observation information.

[0155] 3. Parameter adjustment

[0156] In this embodiment, some parameters are changed and some typical curve changes are selected for analysis.

[0157] 1. Change the initial state

[0158] The initial state of the extended Kalman filter is s[0|0] = [15, 5, -0.2, 0.2] T Change to s[0|0] = [5,5,0,0] T, and the initial state is still s[0]=[15,5,-0.2,0.2] T .observe Fig.14 , 15 It can be seen that it is difficult to achieve continuous tracking of the target in a nonlinear system. For this target tracking model, the system is very dependent on the initial state, so the selection of the initial state is very important for target tracking.

[0159] 2. Increase the time range

[0160] The time point value is increased from 60 to 100. When the time range value is increased, Fig.16 It can be seen that the estimated values ​​gradually disperse, and the estimated value of the target y-axis speed is no longer so accurate. Fig.17 It can be seen that as the number of time points increases, after several algorithm iterations, the error of EKF estimation becomes larger and larger, and finally the EKF algorithm is almost divergent.

[0161] 3. Increase process noise value

[0162] From 0.001 to 0.1, from Fig.18 From the position and velocity estimation deviation, the deviation value reaches more than 20 after 30 time points, and the Kalman filter has almost no effect. This is because the process noise is too large, and the increase in random disturbance leads to "no rules to follow", which ultimately makes the filter unable to reduce noise. However, if the process noise is reduced to the original 0.000l, such as Fig.19 As shown, it can be clearly seen that Kalman filtering can greatly reduce noise and greatly reduce the estimation deviation.

[0163] 4. Increase measurement noise

[0164] from Fig. 20 Fig.21 It can be seen that increasing the measurement noise will cause the tracking trajectory of the Kalman filter to fluctuate, but overall it still fluctuates around the true trajectory of the target, and can also achieve a certain tracking effect.

[0165] Example 4

[0166] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0167] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0168] The processing unit performs the various methods and processes described above, such as methods S1 to S6. For example, in some embodiments, methods S1 to S6 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S6 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S6 in any other appropriate manner (e.g., by means of firmware).

[0169] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0170] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0171] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0172] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A ship gear recognition method based on target tracking and re-recognition, characterized in that: The method comprises the following steps: Obtaining ship information, including basic ship information, ship dispatch diagram, ship pictures and real-time ship video; Target recognition is performed based on YOLO5s with the extended Kalman filter algorithm. The ship's trajectory is tracked from the moment it enters the lock, and the basic information of the ship and the ship's picture are matched. The ship detection frame and basic information of the trajectory tracking result are superimposed on the real-time video of the ship. If occlusion occurs during the ship trajectory tracking process, resulting in the loss of the ship trajectory, the Deep OC-SORT algorithm is used to predict the position of the next frame or the next few frames of the target lost in the real-time video, and the lost target is searched near the predicted position; when one or more targets are found near the predicted position, the IoU calculation is performed between the content in the ship detection frame obtained by trajectory tracking and the Redis re-identification technology. When the IoU value is greater than the preset threshold, the found target and the target in the corresponding ship detection frame are regarded as the same target; During the ship trajectory tracking process, if it is identified that the ship has not moved for a preset time, the ship gear shifting and docking is judged to be completed, the ship's docking position is obtained and the actual ship gear shifting map is formed; Determine whether the ship is scheduled according to the ship scheduling diagram, actual gear diagram and basic ship information. If the ship is not scheduled according to the scheduling, an alarm will be issued.

2. A ship gear recognition method based on target tracking and re-identification according to claim 1, characterized in that: The target recognition based on YOLO5s with extended Kalman filter algorithm specifically includes the following steps: Initialize the state of the extended Kalman filter tracker: use the frame where YOL0v5s target detection fails as the third frame of the extended Kalman filter tracker, use the detection values ​​of the first two frames as the initial state of the tracker, and initialize the state vector and error covariance matrix of the second frame; State prediction: predict the current state vector and state error covariance matrix based on the previous state; Calculate the extended Kalman gain: define the observation vector, calculate the extended Kalman gain through the daily matrix and the prior results obtained in the prediction stage; Update: Update the state vector and error covariance matrix according to the extended Kalman gain to achieve continuous tracking of the target; Candidate region creation and matching: Based on the position of the prior system state bounding box of the extended Kalman filter, the region is expanded to obtain the candidate region, and the difference value hash algorithm is used to match the candidate region, and the candidate region with the smallest difference is selected as the observation value.

3. A ship gear recognition method based on target tracking and re-recognition according to claim 2, characterized in that: The state vector x and error covariance matrix P of the second frame are shown as follows: in and Represents the coordinates of the upper left corner and lower right corner of the bounding box on the x-axis and y-axis respectively. The subscript m represents the observed value, and the subscript 2 of m represents the second frame. and To calculate the result, ΔT is the time taken from the first frame to the second frame, cent(x) and cent(y) represent the centroid coordinates of the bounding box on the x-axis and y-axis, respectively.

4. A ship gear recognition method based on target tracking and re-recognition according to claim 2, characterized in that: The extended Kalman filter updates the state vector and the error covariance matrix according to the extended Kalman gain: Among them, K k is the extended Kalman gain, is the prior state vector, is the prior error covariance matrix, H is the Jacobian matrix, and Z k is the observation vector, Represents the nonlinear transformation of the state vector to the measurement vector:

5. A ship gear recognition method based on target tracking and re-recognition according to claim 2, characterized in that: The candidate region is created by moving the position of the prior system state bounding box of the extended Kalman filter in eight directions, the directions being 45° to the upper left, 45° to the upper right, 45° to the lower right, 45° to the lower left, and directly above, directly below, directly to the right, and directly to the left. The degree of movement is represented by the intersection over union (IoU). The bounding box of the prior system state is represented by E, and the bounding box after the movement is represented by F. The intersection over union (IoU) is defined as: The degree of movement is determined by presetting the intersection-and-union ratio value, and the expansion areas in eight directions are obtained. Combined with the prior system state bounding box position, nine candidate areas are obtained.

6. A ship gear recognition method based on target tracking and re-recognition according to claim 2, characterized in that: The method of using the difference value hash algorithm to match candidate regions is as follows: Reduce the expanded candidate area image to n×(n-1), where n is a preset value; Convert the reduced image into a grayscale image, assign 1 to each point in each row where the previous pixel value is greater than the next pixel value, and 0 otherwise, to generate a hash value; Calculate the Hamming distance between the candidate region and the image in the target bounding box of the previous frame. The smaller the distance, the smaller the difference in the images. Select the candidate region with the smallest difference as the observation value.

7. A ship gear recognition method based on target tracking and re-recognition according to claim 1, characterized in that: The Deep OC-SORT algorithm performs the following steps: Get the scaling rotation matrix M t =s t R t and the translation matrix T t ,in s t represents the scaling matrix, R t represents the rotation matrix; Introduce the scaling rotation matrix and translation matrix into three components of the OC-SORT algorithm: OOS, OCM and OCR: For the OOS component, the Kalman filter is updated along a linear interpolation path starting from the last known measurement, which is given by [x c ,y c ,a,s], the first two items are the center of the bounding box, and the center of the bounding box is c<-M t c+T t Transformed to obtain, where a is the area and s is the aspect ratio; For the OCM component, let p1, p2 be the upper left and lower right corners of the bounding box. OCM uses the last Δt=3 bounding box to calculate the bounding box angular velocity. At each time step t, the transformation is applied to the bounding box. p i ←M t p i +T t ; For the OCR component, for the last seen bounding box position in OCR, at each time step t, apply p i ←M t p i +T t to adjust its position; Under the OC-SORT algorithm, the Kalman state is The Kalman state is adjusted as follows by camera motion compensation:

8. A ship gear recognition method based on target tracking and re-recognition according to claim 7, characterized in that: Using low detector confidence to identify image degradation due to occlusion or blur, let e t is the tracklet appearance embedding at time t, the standard EMA is: e t =a t e t-1 +(1-a t )e new e new is the appearance of the matching detection added to the model, α t Defined as: Among them, s det is the detector confidence, σ is the detection confidence threshold used to filter noisy detections, and α f is a fixed value; Adaptive weighting: Use the standard cosine similarity between track and box embeddings to get an M×N appearance cost matrix A c , where M and N are the number of trajectories and detections, respectively, and A c [m,n] represents the item at the intersection of the mth row and the nth column, which is combined with the IoU cost matrix to obtain C = I c +a w A c , and the weighting factor w b (m, n) is added to the global a w In the above example, let T m Indicates tracking, d n Indicates detection, when T m When only one box has a high similarity score, in A c [m,:] increases the appearance weight if the detection d n has only one distinct association with a track, then the same operation is applied to A c Columns, using z diff To measure the discriminability of the detection box-track pair, z diff Defined as the difference between the highest and second highest values ​​in a row or column: Among them, ∈ is a hyperparameter; According to z diff The weighting factors are derived, resulting in the final cost matrix C: C[m,n]=IoU[m,n]+[a w +w b (m,n)]A c [m,n]. The missing target search is achieved by minimizing the cost matrix C.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images