A method for alignment of AIS and port video surveillance

Abstract

The present invention discloses an alignment method for AIS and port video surveillance, comprising the following steps: based on the principle of perspective transformation, manually calibrating four sets of monitoring screen coordinates and corresponding points of latitude and longitude coordinates in AIS records, establishing a low-precision mapping relationship between the monitoring screen coordinates and the latitude and longitude coordinates; using a deep learning target detection model to detect ships in the monitoring video and generate a ship detection frame; mapping the ship latitude and longitude coordinates in the AIS record to the monitoring video screen using a low-precision mapping relationship to generate a ship priori frame; using the KM algorithm to match the ship detection frame with the ship priori frame to obtain a set of matching pairs; using a gradient descent optimization method, based on the set of matching pairs, further optimizing the low-precision mapping relationship to establish a high-precision mapping relationship between the monitoring screen coordinates and the latitude and longitude coordinates. The present invention can establish an accurate mapping relationship between AIS and port video surveillance even when the monitoring camera parameters are unknown.

Description

Technical Field

[0001] The present invention relates to the field of port vessel monitoring and data alignment, and in particular to an alignment method for AIS and port video monitoring. Background Art

[0002] In recent years, with the continuous improvement of information infrastructure, the means of ocean perception have been continuously enriched and strengthened. For the practical need of monitoring vessels in ports, in addition to relying on port surveillance video, the AIS system, composed of shore-based (base station) facilities and shipborne equipment, can also more conveniently provide information such as vessel identification code, name, type, location, course, and speed. Combining these two types of information further facilitates the management of waterway shipping vessels.

[0003] Aligning AIS with port video surveillance means that, given a specific moment (timestamp), the AIS record and surveillance video footage at that moment (timestamp) are simultaneously acquired, and the correspondence between the latitude and longitude coordinates in the AIS record and the coordinates of the surveillance video footage is established.

[0004] In existing AIS-based vessel monitoring systems, aligning AIS longitude and latitude coordinates with the monitoring screen coordinates relies on physical parameters such as the panoramic camera's focal length and the pan / tilt angle. Obtaining these parameters increases the difficulty of aligning AIS longitude and latitude coordinates with the monitoring screen coordinates.

[0005] In view of the shortcomings of the above methods, it is urgent to propose a simpler and more effective alignment method. Summary of the Invention

[0006] The present invention aims to solve the above-mentioned problems in existing ship monitoring systems combined with AIS. It provides an alignment method for AIS and port video surveillance. Based on the principle of perspective transformation, this method only requires manual selection of a small number of corresponding points to establish a mapping relationship between the longitude and latitude coordinates in the AIS and the coordinates on the monitoring screen. The method also uses a gradient descent method to reduce the error between the ship detection frame and the prior frame, further optimize the mapping relationship, and achieve precise alignment of the longitude and latitude coordinates in the AIS and the coordinates on the monitoring screen.

[0007] The purpose of the present invention can be achieved by taking the following technical solutions:

[0008] A method for aligning AIS and port video surveillance, specifically Figure 1 As shown, the technical solution adopted by the present invention includes the following specific steps:

[0009] S1. Based on the perspective transformation principle, manually calibrate the corresponding points of four sets of monitoring screen coordinates and longitude and latitude coordinates in AIS records, and establish a low-precision mapping relationship between monitoring screen coordinates and longitude and latitude coordinates;

[0010] S2. Use a deep learning-based object detection model to detect ships in the surveillance video and generate ship detection frames.

[0011] S3. Mapping the longitude and latitude coordinates of the vessel in the AIS record to the surveillance video screen according to the low-precision mapping relationship between the surveillance camera and the control screen coordinates and the longitude and latitude coordinates to generate a priori frame of the vessel;

[0012] S4. Use the KM algorithm to match the ship detection frame with the ship prior frame to obtain a set of matching pairs;

[0013] S5. Using a gradient descent optimization method, based on the matching pair set, optimize the low-precision mapping relationship between the control screen coordinates and the longitude and latitude coordinates, and establish a high-precision mapping relationship between the monitoring screen coordinates and the longitude and latitude coordinates.

[0014] Furthermore, the process of step S1 is as follows:

[0015] S101. In the Amap API tool, a rectangular area including the visible range of the surveillance image is captured, and a frame of the surveillance video is obtained.

[0016] S102: Manually select four pairs of significant corresponding points between the rectangular area captured by the map API tool and the surveillance video. The corresponding points are selected by selecting the corner points of the island in the surveillance video and then selecting the corresponding points in the map API tool.

[0017] S103. Based on the perspective transformation principle, the four corresponding point pairs are substituted into the perspective transformation formula. The perspective transformation formula is as follows:

[0018] The coordinate transformation relationship of perspective transformation is as follows:

[0019]

[0020] Among them, x, y represent the X-axis coordinate and Y-axis coordinate in the two-dimensional plane before perspective transformation, and 1 in the augmented dimension indicates that the coordinate is in the two-dimensional plane; X, Y, Z represent the X-axis, Y-axis, and Z-axis coordinates in the three-dimensional space generated during the perspective transformation, and X′, Y′ represent the coordinates of the three-dimensional space re-projected to the two-dimensional plane, and a Z′ value of 1 indicates that it is projected into the two-dimensional plane; the matrix {a ij} represents the perspective transformation matrix, where 1<=i<=3, 1<=j<=3, a ij Represents the element in the i-th row and j-th column of the perspective transformation matrix.

[0021] According to the above formula, solve the perspective transformation matrix {a ij}, denoted as M, the perspective transformation matrix represents the low-precision mapping relationship between the monitoring screen coordinates and the latitude and longitude coordinates.

[0022] Furthermore, step S1 is based on the principle of perspective transformation. Perspective transformation is the process of projecting an image onto a new viewing plane. This process involves converting a two-dimensional coordinate system into a three-dimensional coordinate system, and then projecting the three-dimensional coordinate system onto the new two-dimensional coordinate system. The longitude and latitude coordinates recorded by the AIS and the coordinates on the surveillance screen can be considered as the top-down coordinate system and the horizontal coordinate system, respectively, and therefore can be associated through perspective transformation.

[0023] Furthermore, the process of step S2 is as follows:

[0024] S201. For the collected surveillance video, collect one frame of image every second, convert the time information in the image into a timestamp, and use the timestamp to name the image to identify the time information of collecting the image.

[0025] S202. Use the object detection model YOLOv3 pre-trained on a public dataset to perform ship detection on the captured image frames, and record the set of ship detection frames output by the object detection model YOLOv3, denoted as D.

[0026] S203 , counting the average width and height pixel values ​​of the ships in the ship detection frame set D, and recording them as (width, height).

[0027] Furthermore, in step S202, the target detection model YOLOv3, which is pre-trained on a public dataset, is used. The author of this model is Joseph Redmon, and the paper "YOLOv3: An Incremental Improvement" proposed in 2018 was made public by the author in the form of a scientific report. Although it has not been formally published, it is still widely cited. The YOLOv3 model is a continuation of the previous YOLO series of target detection models. It integrates the advantages of the mainstream network at the time and has become a widely used target detection model. The model extracts image features through the convolutional neural network of the backbone part, and then outputs the detected objects and categories of the image features through the detection head part. It can detect objects in the input image. The categories that can be detected include ships, so it can be used to automatically detect ships in the captured image frames and generate ship detection frames.

[0028] Furthermore, the process of step S3 is as follows:

[0029] S301. Group the collected AIS data according to timestamps, and use a map API tool to convert the longitude and latitude in the AIS records into pixel points of a captured area. The set of pixel points is denoted as S.

[0030] S302. Use a low-precision mapping relationship matrix M to map the acquired latitude and longitude pixel point set S to the surveillance video image as the center coordinates of the prior frame, and use the statistically obtained average width and height pixel values ​​(width, height) of the ship to generate the ship prior frame set P. This mapping is a one-to-one mapping, denoted as f: S·M±(width / 2,height / 2)→P.

[0031] Furthermore, the low-precision mapping matrix M used in step S3 is essentially a perspective transformation matrix. Perspective transformation is the process of projecting an image onto a new viewing plane. This process involves converting a two-dimensional coordinate system into a three-dimensional coordinate system, and then projecting the three-dimensional coordinate system onto a new two-dimensional coordinate system. The AIS latitude and longitude coordinates and the monitoring screen coordinates can be considered as the top-down and horizontal coordinate systems, respectively. Therefore, the low-precision mapping matrix M can be used to transform the longitude and latitude pixel coordinates into monitoring screen coordinates, which are then combined with the calculated (width, height) to generate the vessel prior bounding box.

[0032] Furthermore, the process of step S4 is as follows:

[0033] S401. For each timestamp t, construct a bipartite graph G t =(D t ,P t ,E t ), where D t represents the set of ship detection frames in the surveillance camera image at timestamp t, P t represents the set of ship prior frames after the AIS record transformation at timestamp t, E t Indicates D t All nodes in P t The set of edges between all nodes in ; the edge set E t According to the following rules: if IOU(d,p)>0,d∈D t ,y∈P t , then in the bipartite graph G t Connect an edge (d,p) between the two corresponding nodes d and p, and set the weight w of the edge dp =IOU(d,p), where IOU(d,p) represents the intersection-and-union ratio d∈D between the detection box d and the prior box p t , p∈P t .

[0034] S402, for each bipartite graph G t , use KM algorithm to calculate the best matching of bipartite graph, and record the matching result as set Match t ={(d i ,p i ,si )}, set Match t The elements in (d i ,p i ,s i ) represents the i-th matched ship detection box d at timestamp t i With the vessel prior box p i , and the ship prior box p i The latitude and longitude pixel coordinates s before mapping f i .

[0035] The KM algorithm calculates the best matching process of a bipartite graph as follows: 1) The constructed bipartite graph is represented by an adjacency matrix as the input of the KM algorithm; 2) The KM algorithm process is executed; 3) The KM algorithm process ends and the best matching pair is output.

[0036] Furthermore, in step S4, the KM algorithm is used to calculate the optimal match between the ship detection box and the ship prior box. The KM algorithm was first proposed by Kuhn in 1955 and published in "The Hungarian Method for the assignment problem," Naval Research Logistics Quarterly, Issue 2, pp. 83–97. Munkres reviewed the algorithm in 1957 and published it in "Algorithms for the Assignment and Transportation Problems," Journal of the Society for Industrial and Applied Mathematics, Volume 5, Issue 1, pp. 32–38. This is a classic algorithm for solving the optimal matching of bipartite graphs. By assigning a vertex to each vertex, the algorithm transforms the problem of finding a maximum-weighted matching into the problem of continuously finding augmenting paths to maximize the number of matchings in the bipartite graph. The constructed bipartite graph is input into the algorithm, which outputs the optimal matching of the bipartite graph. The optimal matching ensures that the weight of the weighted bipartite graph is maximized, thus ensuring the maximum possible match between the ship detection box and the ship prior box.

[0037] Furthermore, the process of step S5 is as follows:

[0038] S501, using the low-precision mapping relationship matrix M as a learnable parameter, using the matching results of the ship inspection box and the ship prior box at all timestamps Match t As input data, construct the following objective function: L(M) = ∑ t ∑ i distance(s ti *M,center(dti ))

[0039] Where distance represents L2 distance, s ti Indicates the i-th matching pair of latitude and longitude pixel coordinates at timestamp t, d ti It represents the i-th matched ship inspection box at timestamp t. The center function is used to calculate the center coordinates of the rectangular box. The calculation formula of the center function is:

[0040] center(box)=((box left +box right ) / 2,(box up +box down ) / 2)

[0041] Among them, box represents a rectangular box, box left Indicates the x-axis coordinate of the upper left corner of the rectangular box, box right Indicates the x-axis coordinate of the lower right corner of the rectangular box, box up Indicates the y-axis coordinate of the upper left corner of the rectangular box, box down Indicates the y-axis coordinate of the lower right corner;

[0042] S502: Optimize the perspective transformation matrix M using gradient descent to obtain a high-precision mapping relationship between the surveillance camera and the latitude and longitude coordinates. The optimization formula used is:

[0043]

[0044] Among them, a ij represents the element in the i-th row and j-th column of the perspective transformation matrix M, α represents the learning rate constant, Represents the objective function for parameter a ij Find the partial derivative, which represents the gradient of the objective function, a ij ′ represents a ij The value after gradient optimization.

[0045] Furthermore, in step S5, gradient descent is used to optimize the objective function, thereby obtaining a high-precision mapping relationship between the surveillance cameras and the latitude and longitude coordinates. Since the objective function is the center distance between the matching box pairs, minimizing this function can effectively increase the overlap between the matched ship detection boxes and the ship prior boxes, making the perspective transformation matrix more accurate. Furthermore, this optimization problem is unconstrained, so gradient descent can be used to optimize the perspective transformation matrix M.

[0046] The working principle of this invention: Perspective transformation is the process of projecting an image onto a new viewing plane. This process involves converting a two-dimensional coordinate system into a three-dimensional coordinate system, and then projecting the three-dimensional coordinate system onto the new two-dimensional coordinate system. AIS latitude and longitude coordinates and surveillance screen coordinates can be considered as the top-down and horizontal coordinate systems, respectively. Therefore, perspective transformation can be used to associate and align them. Based on the principle of perspective transformation, the present invention first converts the longitude and latitude in AIS records into two-dimensional image coordinates through a map API. Then, by manually selecting corresponding point pairs in the monitoring image, the perspective transformation matrix is ​​solved to serve as a low-precision mapping relationship between the AIS and video surveillance. Furthermore, a target detection model pre-trained on a public dataset is used to extract ship detection frames from the video surveillance image. A priori ship frames are generated from the AIS records using the map API and the low-precision mapping relationship. Furthermore, a KM algorithm is used to obtain matching pairs of ship detection frames and a priori ship frames. The mapping relationship matrix is ​​used as a learnable parameter, and the center distance of the matching frame pairs is used as the optimization objective function. Then, through gradient descent optimization, a high-precision mapping relationship between the AIS longitude and latitude coordinates and the video surveillance image coordinates is obtained, achieving precise alignment between the two.

[0047] The present invention has the following advantages and effects compared to the prior art:

[0048] 1. This paper proposes a novel alignment method for AIS and surveillance images. Based on the principle of perspective transformation, the two perspectives of AIS and video surveillance are linked. When the specific parameters of the camera are unknown, the longitude and latitude in the AIS and the pixel coordinates of the surveillance image can be aligned. Only a small amount of manual selection is required to obtain a low-precision mapping relationship.

[0049] 2. The present invention further uses the mapping relationship matrix as a learnable parameter, uses the KM algorithm to match the ship detection frame and the ship prior frame, and uses the center distance of the matching frame pair as the optimization objective function, and then adopts the gradient descent method for optimization, so as to obtain a more accurate mapping relationship between the AIS latitude and longitude coordinates and the video surveillance screen coordinates. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0051] Figure 1 It is a flow chart of the AIS and port video surveillance alignment method disclosed in the present invention;

[0052] Figure 2 Schematic diagram of step S4 in the AIS and port video surveillance alignment method disclosed in the present invention;

[0053] Figure 3 This is a comparison diagram of the optimization effects of step S5 in the AIS and port video surveillance alignment method disclosed in the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0055] Example

[0056] This embodiment is implemented by Figure 1 The implementation steps shown are implemented, taking the alignment of the port surveillance video upstream of Huangpu Bridge and the port area AIS as an example, which includes the following steps:

[0057] S1. Based on the perspective transformation principle, manually calibrate the corresponding points of four sets of monitoring screen coordinates and longitude and latitude coordinates in AIS records, and establish a low-precision mapping relationship between monitoring screen coordinates and longitude and latitude coordinates;

[0058] The specific implementation of step S1 is as follows: use the AutoNavi map API tool to intercept a rectangular area containing the visible range of the port surveillance video, and realize the conversion function from longitude and latitude to the pixel points of the intercepted rectangular area; take a frame of the surveillance video, manually select 4 groups of island corner corresponding point pairs in the area intercepted by the map API tool and in the surveillance video, which are ([1574,674], [1532,536]), ([1628,726], [1344,623]), ([1504,840], [166,489]), ([1436,810], [413,444]); according to the formula transformation formula, solve the perspective transformation matrix, which represents the low-precision mapping relationship M between the coordinates of the surveillance screen and the longitude and latitude coordinates. The perspective transformation formula is as follows:

[0059]

[0060] The coordinate transformation relationship of perspective transformation is as follows:

[0061]

[0062]

[0063] Among them, x, y represent the X-axis coordinate and Y-axis coordinate in the two-dimensional plane before perspective transformation, and 1 in the augmented dimension indicates that the coordinate is in the two-dimensional plane; X, Y, Z represent the X-axis, Y-axis, and Z-axis coordinates in the three-dimensional space generated during the perspective transformation, and X′, Y′ represent the coordinates of the three-dimensional space re-projected to the two-dimensional plane, and a Z′ value of 1 indicates that it is projected into the two-dimensional plane; the matrix {a ij} represents the perspective transformation matrix, where 1<=i<=3, 1<=j<=3, a ij Represents the element in the i-th row and j-th column of the perspective transformation matrix.

[0064] Substitute the four corresponding point pairs into the above formula to solve the perspective transformation matrix M:

[0065]

[0066] The perspective transformation matrix represents the low-precision mapping relationship between the monitoring screen coordinates and the latitude and longitude coordinates.

[0067] S2. Use a deep learning-based object detection model to detect ships in the surveillance video and generate ship detection frames.

[0068] The specific implementation of step S2 is as follows: using the OpenCV library, collect one frame of image every second from the collected surveillance video, and convert the time information in the image into a timestamp, and use the timestamp to name the image to identify the time information of the frame image; then use the object detection model YOLOv3 pre-trained on the COCO dataset to perform ship target detection on the video frame of each timestamp, and retain the set of ship detection frames corresponding to each timestamp t, which is recorded as D t In addition, the average width and height of the ship detection frame (width, height) = (128, 60) are counted for the generation of the ship prior frame in the subsequent stage.

[0069] S3. Map the longitude and latitude coordinates of the vessel in the AIS record to the surveillance video image based on the low-precision mapping relationship between the surveillance camera and the longitude and latitude coordinates to generate a priori frame of the vessel;

[0070] The specific implementation of step S3 is: group the collected AIS data according to the timestamp, and convert the longitude and latitude in the AIS record into the pixel coordinates of the intercepted area through the map API tool, and record the pixel coordinate set at the timestamp t as S t ={(s x ,s y )}, where s x Indicates the horizontal coordinate, s y Represents the vertical coordinate; then use the low-precision mapping relationship M to convert the latitude and longitude pixel coordinates (s x ,sy ), transform it into the surveillance video image, and generate the center coordinates of the prior frame (p centerx ,p centery ), where p centerx Represents the horizontal coordinate of the center of the prior box, p centery Represents the vertical coordinate of the center of the prior frame; then, based on the average width and height of the ship (width, height) = (128, 60), a set of ship detection prior frames is generated, denoted as P t ={(p leftx ,p lefty ,p rightx ,p righty )}, where p leftx Indicates the horizontal coordinate of the upper left corner of the ship prior box, p lefty Indicates the vertical coordinate of the upper left corner of the ship prior box, p rightx Represents the horizontal coordinate of the lower right corner of the ship prior box, p righty Indicates the vertical coordinate of the lower right corner of the ship prior box.

[0071] The process of generating the ship prior frame is as follows:

[0072]

[0073] S4. Use the KM algorithm to match the ship detection frame with the ship prior frame at each timestamp to obtain a set of matching pairs;

[0074] The specific implementation of step S4 is as follows:

[0075] S401. For each timestamp t, construct a bipartite graph G t =(D t ,P t ,E t ), where D t represents the set of ship detection frames in the surveillance camera image at timestamp t, P t represents the set of ship prior frames after the AIS record transformation at timestamp t, E t Indicates D t All nodes in P t The set of edges between all nodes in ; the edge set E t According to the following rules: if IOU(d,p)>0,d∈D t ,y∈P t , then in the bipartite graph G t Connect an edge (d,p) between the two corresponding nodes d and p, and set the weight w of the edge dp =IOU(d,p), where IOU(d,p) represents the intersection-and-union ratio d∈D between the detection box d and the prior box p t , p∈Pt .

[0076] S402, for each bipartite graph G t , use KM algorithm to calculate the best match, and record the matching result as set Match t ={(d i ,p i ,s i )}, set Match t The elements in (d i ,p i ,s i ) represents the i-th matched ship detection box d at timestamp t i With the vessel prior box p i . And the ship prior box p i The latitude and longitude pixel coordinates s before mapping f i .

[0077] S5. Using a gradient descent optimization method, based on the matching pair set, further optimize the low-precision mapping relationship to establish a high-precision mapping relationship between the monitoring screen coordinates and the latitude and longitude coordinates;

[0078] The specific implementation of step S5 is as follows:

[0079] S501, take the low-precision mapping relationship matrix M as a learnable parameter, and use the Match under all timestamps t As input data, construct the optimization objective function as shown below:

[0080]

[0081] Where distance represents L2 distance, s ti Indicates the i-th matching pair of latitude and longitude pixel coordinates at timestamp t, d ti It represents the i-th matched ship inspection box at timestamp t. The center function is used to calculate the center coordinates of the rectangular box. The calculation formula is:

[0082] center(box)=((box left +box right ) / 2,(box up +box down ) / 2)

[0083] Among them, box represents a rectangular box, box left Indicates the x-axis coordinate of the upper left corner of the rectangular box, box right Indicates the x-axis coordinate of the lower right corner of the rectangular box, box up Indicates the y-axis coordinate of the upper left corner of the rectangular box, box downIndicates the y-axis coordinate of the lower right corner.

[0084] S502, using gradient descent to minimize the objective function L(M), optimize the mapping relationship matrix M, and the parameter update formula is:

[0085] Among them, a ij represents the element in the i-th row and j-th column of the perspective transformation matrix M, α represents the learning rate constant, Represents the objective function for parameter a ij Find the partial derivative, which represents the gradient of the objective function.

[0086] The learning rate constant α is set to 0.00000015. After 20 rounds of gradient descent optimization, the high-precision mapping relationship M' between the surveillance camera and the latitude and longitude coordinates is obtained:

[0087]

[0088] Figure 3 The comparison effect before and after the optimization of the mapping relationship matrix is ​​shown. It can be seen that the optimized mapping relationship matrix can make the IOU value of the ship detection box and the ship prior box larger, indicating that it has higher accuracy.

[0089] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for aligning AIS and port video surveillance, characterized in that: The alignment method comprises the following steps: S1. Based on the perspective transformation principle, manually calibrate the corresponding points of four sets of monitoring screen coordinates and longitude and latitude coordinates in AIS records, and establish a low-precision mapping relationship between monitoring screen coordinates and longitude and latitude coordinates; S2. Use a deep learning-based object detection model to detect ships in the surveillance video and generate ship detection frames. S3. Mapping the longitude and latitude coordinates of the vessel in the AIS record to the surveillance video screen based on a low-precision mapping relationship between the surveillance camera and the surveillance screen coordinates and the longitude and latitude coordinates to generate a priori frame of the vessel; S4. Use the KM algorithm to match the ship detection frame with the ship prior frame to obtain a set of matching pairs. The process is as follows: S401. For each timestamp t, construct a bipartite graph G t =(D t ,P t ,E t ), where D t represents the set of ship detection frames in the surveillance camera image at timestamp t, P t represents the set of ship prior frames after the AIS record transformation at timestamp t, E t Indicates D t All nodes in P t The set of edges between all nodes in ; the edge set E t According to the following rules: if IOU(d,p)>0,d∈D t ,y∈P t , then in the bipartite graph G t Connect an edge (d,p) between the two corresponding nodes d and p, and set the weight w of the edge dp =IOU(d,p), where IOU(d,p) represents the intersection-and-union ratio d∈D between the detection box d and the prior box p t , p∈P t ; S402, for each bipartite graph G t , use KM algorithm to calculate the best matching of bipartite graph, and record the matching result as set Match t ={(d i ,p i ,s i )}, set Match t The elements in (d i ,p i ,s i ) represents the i-th matched ship detection box d at timestamp t i With the vessel prior box p i , and the ship prior box p i The latitude and longitude pixel coordinates s before mapping f i ; S5. Using a gradient descent optimization method, based on the matching pair set, optimize the low-precision mapping relationship between the monitoring screen coordinates and the latitude and longitude coordinates, and establish a high-precision mapping relationship between the monitoring screen coordinates and the latitude and longitude coordinates.

2. The method for aligning AIS and port video surveillance according to claim 1, characterized in that: The process of step S1 is as follows: S101. In the map API tool, capture a rectangular area that includes the visible range of the surveillance camera and obtain a frame of the surveillance video; S102, manually selecting four pairs of significant corresponding points in the area captured by the map API tool and in the surveillance video, wherein the corresponding points are selected by selecting island corner points in the surveillance video and corresponding points in the map API tool; S103. Based on the perspective transformation principle, the four selected corresponding point pairs are substituted into the perspective transformation formula. The perspective transformation formula is as follows: The coordinate transformation relationship of perspective transformation is as follows: Among them, x, y represent the X-axis coordinate and Y-axis coordinate in the two-dimensional plane before perspective transformation, and 1 in the augmented dimension indicates that the coordinate is in the two-dimensional plane; X, Y, Z represent the X-axis, Y-axis, and Z-axis coordinates in the three-dimensional space generated during the perspective transformation, and X′, Y′ represent the coordinates of the three-dimensional space re-projected to the two-dimensional plane, and Z′ value 1 indicates that it is projected into the two-dimensional plane; the matrix {a ij } represents the perspective transformation matrix, where 1<=i<=3, 1<=j<=3, a ij Represents the element in row i and column j of the perspective transformation matrix; According to the above formula, solve the perspective transformation matrix {a ij }, denoted as M, the perspective transformation matrix represents the low-precision mapping relationship between the monitoring screen coordinates and the latitude and longitude coordinates.

3. The method for aligning AIS and port video surveillance according to claim 1, characterized in that: The process of step S2 is as follows: S201. For the collected surveillance video, collect one frame of image every second, convert the time information in the image into a timestamp, and use the timestamp to name the image; S202. Use the object detection model YOLOv3 pre-trained on a public dataset to perform ship detection on the captured image frames, and record the set of ship detection frames output by the object detection model YOLOv3, denoted as D. S203: Count the average width and height pixel values ​​of the ships in the ship detection frame set D and record them as (width, height) for subsequent generation of ship prior frames.

4. The method for aligning AIS and port video surveillance according to claim 1, characterized in that: The process of step S3 is as follows: S301. Group the collected AIS records by timestamp and convert the longitude and latitude coordinates in the AIS records into pixels of the intercepted area using a map API tool. The set of pixels is denoted as S. S302. Use a low-precision mapping relationship matrix M to map the acquired latitude and longitude pixel point set S to the surveillance video image as the center coordinates of the prior frame, and use the statistically obtained average width and height pixel values ​​(width, height) of the ship to generate the ship prior frame set P. This mapping is a one-to-one mapping, denoted as f: S·M±(width / 2,height / 2)→P, where the symbol "→" represents the mapping relationship.

5. The method for aligning AIS and port video surveillance according to claim 1, characterized in that: The process of step S5 is as follows: S501, using the low-precision mapping relationship matrix M as a learnable parameter, using the matching results of the ship inspection box and the ship prior box at all timestamps Match t As input data, construct the following objective function: Where distance represents L2 distance, s ti Indicates the i-th matching pair of latitude and longitude pixel coordinates at timestamp t, d ti It represents the i-th matched ship inspection box at timestamp t. The center function is used to calculate the center coordinates of the rectangular box. The calculation formula of the center function is: center(box)=((box left +box right ) / 2,(box up +box down ) / 2) Among them, box represents a rectangular box, box left Indicates the x-axis coordinate of the upper left corner of the rectangular box, box right Indicates the x-axis coordinate of the lower right corner of the rectangular box, box up Indicates the y-axis coordinate of the upper left corner of the rectangular box, box down Indicates the y-axis coordinate of the lower right corner; S502. Optimize the perspective transformation matrix M using a gradient descent method to obtain a high-precision mapping relationship between the surveillance camera and the latitude and longitude coordinates. The formula for gradient descent optimization is: Among them, a ij represents the element in the i-th row and j-th column of the perspective transformation matrix M, α represents the learning rate constant, Represents the objective function for parameter a ij Find the partial derivative, which represents the gradient of the objective function.

Images