Ship Track Prediction Method Based on Water Flow Detection

Abstract

The present invention relates to a ship trajectory prediction method based on water flow detection, belonging to the technical field of image processing. The method includes: collecting multiple frames of images and detecting ships using an object detection model; when a detected ship appears, separating the detected ship, extracting its corner features, and tracking the corner features using the optical flow method; on the basis of successful ship detection, using a Gaussian mixture background modeling and frame difference method to extract water flow moving targets based on the collected multiple frames of image data; calculating the water flow motion vector based on the multi-frame water flow moving target detection method, performing a weighted vector sum with the ship's driving vector, and calculating the position of the ship in the next frame under the condition of determining the image frame rate, and fitting the ship trajectory route according to the velocity vectors of all water flows on the ship trajectory route. The present invention adopts an image processing method, takes into account the influence of water flow speed during the ship's driving process in the real scene, realizes the prediction of the ship trajectory, and thus improves the accuracy of ship trajectory prediction.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, relates to multi-target tracking and water flow recognition technology, and specifically relates to a ship track prediction method based on water flow detection. Background Art

[0002] Ship track prediction has always been a hot application of image processing in the field of shipping, and it is used in various industries, such as military, bridge collision avoidance, unmanned ships and other industries. At the same time, the practices are different in each industry. In the military and unmanned ship industries, image processing is the mainstream; while in the construction and water conservancy industries, passive collision avoidance is mainly used, using peripherals for path collection and distance detection to achieve ship track prediction. Regardless of the industry, there is no fixed and universal solution for ship track prediction, only the most suitable solution. Therefore, the present invention mainly proposes a single solution, which is not limited to a certain industry or application scenario.

[0003] At present, image processing technology is relatively mature, such as target detection, target recognition, target tracking and other technologies, which have almost reached the same accuracy as the human eye, or even higher. In the process of ship track prediction, the mainstream approach of image processing is to identify the ship and fit the path of the ship by combining the time-series video stream information to achieve the purpose of prediction. This approach is relatively simple, but there are many factors that affect the ship's progress in the real world, not only the ship itself, but also external factors, such as water flow direction, water flow speed, wind speed, wind direction, etc., which will affect the speed and direction of the ship. Only by considering the influence of these factors in the actual environment as much as possible can we have a solution that is more in line with the actual situation. Although the actual situation is complex and there are many influencing factors, the specific implementation cost and the actual scene environment must also be considered. In the current common ship track prediction system, there are few solutions that combine water flow speed for comprehensive consideration.

[0004] In water flow, the calculation of water velocity is often a major problem that troubles people. In most environments, the image features of water are not obvious enough, and it is difficult to continuously extract effective features from the image. In addition, during the navigation of a ship, the water flow changes in many ways and the situation is complicated.

[0005] Therefore, the present invention proposes a new water flow detection method to predict the ship's track. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a ship track prediction method based on water flow detection, which adopts image processing, combines the influence of water flow speed on the ship during its travel in real scenes, and combines the speed of water waves in the water flow with the ship's travel speed to realize the prediction of the ship's track, thereby improving the accuracy of ship track prediction.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A ship trajectory prediction method based on water flow detection, specifically comprising the following steps:

[0009] S1: Use a camera to collect multiple frames of image data, and use an object detection model to detect ships;

[0010] S2: When a detected ship appears, separate the detected ship, extract the corner features of the ship, replace the tracking of the moving ship target with the tracking of the corner features, use the optical flow method to track the feature points. When multiple moving ships appear, record the positions of the ships detected in each frame, perform corner feature detection within a certain range near each ship, and perform tracking according to the positions of the feature points, that is, judge the tracking situation of the ship based on the distribution of the corner features of the ship in the front and rear two frames of images;

[0011] S3: On the basis of successful ship detection, based on the collected multiple frames of image data, use a mixture of Gaussian background modeling and frame difference method to extract water flow moving targets;

[0012] S4: Based on the multi-frame water flow moving target detection method, calculate the water flow motion vector, perform a weighted vector sum with the ship's traveling vector. Under the condition of determining the image frame rate, calculate the next frame position of the ship, and fit the ship's trajectory route according to the velocity vectors of all water flows on the ship's trajectory route.

[0013] Further, in step S1, for ship detection, it specifically includes the following steps:

[0014] S101: Before detection, use the training model of the object detection model and the collected ship data set to train a model capable of identifying ships;

[0015] S102: Use image annotation software to annotate the collected ship images. After annotation, use the training set for model training. After the training model is completed, input the test set for testing. When it is found that it cannot be recognized, input it into the model for retraining until the model is relatively perfect, until the model recognition accuracy rate reaches 98%;

[0016] S103: Use the trained model to perform ship detection on each frame of image collected by the camera.

[0017] Further, in step S2, for extracting the corner features of the ship, it specifically includes the following steps:

[0018] S201: Based on multiple ships detected by the object detection model, calculate their corner features, and set appropriate X max 、Y max values, respectively representing the maximum range of movement in the x-axis and y-axis directions;

[0019] S202: Traverse the corner features of each ship, obtain the central positions X and Y of the ship object, and perform feature matching on the pixel points of the image within the ranges (X - X max / 2, X + X max / 2) and (Y - Y max / 2, Y + Y max / 2);

[0020] S203: Perform regional division based on the distribution of feature matching points, fit the distribution of points according to the Gaussian distribution, and ensure that at least 50% of the feature points of the target's latest position fall within the region. Compare with the detection range of the target detection model: when the range is smaller than the detection range of the target detection model, use the detection range of the target detection model; otherwise, use the previous detection range as the judgment basis for corner tracking in the next frame.

[0021] Furthermore, in step S3, a mixture of Gaussian background modeling and frame difference method is used to extract the water flow moving target, which specifically includes the following steps:

[0022] S301: Input the front and rear two frames of images, perform gray processing, smooth Gaussian filtering, and binary thresholding on them in sequence. After processing, subtract the two frames of images, and through dilation processing and contour detection, obtain the water flow change image in the two frames of images;

[0023] S302: Obtain the contour change range S i from each closed-loop water flow contour image formed through contour detection and dilation processing. Traverse from top to bottom within this range to obtain the pixel vertex coordinates, and fit the bottom and top contour waveforms: first, smooth the waveforms through the simple moving average method; then, use the cubic Bezier to fit the waveforms, connect the corner points of the top and bottom contour waveforms, which represents the water wave flow direction within this water flow region, and the length of the connected straight line segment is D i . Given the known image acquisition frame rate R, calculate the water flow velocity vector D i / R.

[0024] Furthermore, in step S302, the waveforms are smoothed through the simple moving average method, and its calculation formula is h t =(y t-1 +y t-2 +y t-3 +...+y t-n ) / n, where h t represents the coordinate of the smoothed result, n represents the number of moving averages, and y t-n represents the first n moving average results.

[0025] Furthermore, step S4 specifically includes the following steps:

[0026] S401: Calculate the water flow motion vector and the ship's travel vector on the ship's passing route, and then obtain the predicted value of the ship's speed vector for the next frame through the vector calculation formula;

[0027] S402: Assume that on the ship's track route, when the water flow velocity vector remains unchanged at each position, when the ship travels to the next frame position, perform vector calculation in the manner of step S401 until the ship travels out of the image range, so as to fit the ship's track route.

[0028] Further, in step S401, the water flow motion vector V1 on the ship's passing route is equal to the sum of the water flow vectors converging from multiple parties, and the expression is: V1 = (D1 + D2 +... + D i ) / R;

[0029] The ship's travel vector V2 is equal to the ratio of the difference in the ship's position coordinates to the image acquisition frame rate R, and the expression is: V2 = (X n - X n-1 , Y n - Y n-1 ) / R, where (X n , Y n ) is the ship's position coordinates of the nth frame;

[0030] The vector calculation formula is: V1 × W1 + V2 × W2, where W1 is the water flow vector weight and W2 is the ship's travel vector weight.

[0031] Further, in step S402, the expression for fitting the ship's track route is: where m represents that the ship needs to travel out of the detection range m times.

[0032] Further, the detection ship can be replaced by any object that can float in the water.

[0033] The beneficial effects of the present invention are as follows: The present invention adopts an image processing method, takes into account the influence of the water flow speed during the ship's travel in the real scenario, combines the speed of the water waves in the water with the ship's travel speed to realize the prediction of the ship's track, thereby improving the accuracy of ship track prediction.

[0034] Other advantages, objectives and features of the present invention will be described to some extent in the subsequent description, and to some extent, will be obvious to those skilled in the art based on the study of the following text, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. Brief Description of the Drawings

[0035] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in detail and preferably with reference to the accompanying drawings, where:

[0036] Figure 1 It is a flowchart of the ship track prediction method based on water flow detection according to the present invention;

[0037] Figure 2 It is a schematic diagram of the detected ship vector calculation provided by the present invention;

[0038] Figure 3 It is a flowchart of detecting water flow provided by the present invention;

[0039] Figure 4 It is a water flow motion processing flow under the multi-ship image recognition mode provided by the present invention;

[0040] Figure 5 It is a water flow velocity vector display diagram provided by the present invention;

[0041] Figure 6 It is a detected water flow vector display diagram under the multi-ship detection mode provided by the present invention. Specific Embodiments

[0042] The following uses specific specific examples to illustrate the embodiments of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the drawings provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0043] Among them, the drawings are only for illustrative purposes, showing only schematic diagrams, not physical diagrams, and should not be construed as a limitation to the present invention; for better illustrating the embodiments of the present invention, some components in the drawings will be omitted, enlarged, or reduced, and do not represent the dimensions of actual products; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0044] In the drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "rear", etc. indicating the orientation or positional relationship, they are based on the orientation or positional relationship shown in the drawings. This is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the positional relationship in the drawings are only for illustrative purposes and cannot be construed as a limitation of the present invention. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0045] Please refer to Figures 1 to 6 , Figure 1 A ship track prediction method based on water flow detection provided by the present invention, comprising the following steps:

[0046] S1: Based on multiple frames of image data collected by a camera, a target detection model is used to detect ships.

[0047] S2: When the detection target appears, the detected ship is separated, the corner point features of the ship are extracted, and the tracking of the moving ship target is replaced by the tracking of the corner point features. The optical flow method is used to track the feature points. When multiple moving targets appear, the positions of the ships detected in each frame are recorded, and corner point feature detection is performed within a certain range near each ship. The tracking of the target is judged according to the distribution of the ship corner point features in the front and rear two frames of images.

[0048] S3: On the basis of successful ship detection, multiple frames of image data collected by a camera are used to extract water flow moving targets by using a mixture of Gaussian background modeling and frame difference method.

[0049] S4: Based on the detection of multiple frames of water flow moving targets, the water flow motion vector is calculated, and a weighted vector sum is made with the ship motion vector. Under the condition of determining the image frame rate, the next frame position of the ship is calculated, and the ship track route is fitted according to the velocity vectors of all the water flows on the ship track route.

[0050] For step S1, based on multiple frames of image data collected by the camera, a target detection model is used to detect the hull. Specifically, a dataset of ships on the network is downloaded and collected, and the model is trained through the target detection model. In this embodiment, this target detection model is constructed using an improved Fast R-CNN network structure: the main improvement lies in optimizing and improving the problem of highly overlapping candidate regions based on the original Fast R-CNN model network structure, introducing RPN, whose main function is to extract candidate boxes, calculating the intersection over union (IoU) for each candidate box, determining that there is a target if the IoU is greater than 0.8, determining it as the background if it is less than 0.3, and setting it to 0 if it is between 0.3 and 0.8, and it does not participate in training. Compared with Faster R-CNN, which uses the traditional Selective Search method and takes a long time to generate detection boxes, this network structure is simpler, mainly optimizing the problem of redundant calculations in the original Fast R-CNN. In addition, an image annotation software is used to annotate the dataset. In this embodiment, the annotation format is VOC, and a total of 40,000 pieces of data are annotated for training and 10,000 pieces of data are used for testing. After annotation, the training set is used to train the model. After the training of the model is completed, the test dataset is input for testing. When it is found that it cannot be recognized, it is input into the model for retraining until the model is relatively perfect. In this embodiment, the recognition accuracy of the model reaches 98%. The trained model is used to detect the hull for each frame of image read by the camera.

[0051] For step S2, the multi-target detection method extracts corner features according to the following steps:

[0052] Based on the multiple targets detected by the target detection model, calculate their corner features. In this embodiment, the Harris corner detection algorithm is used for the calculation of corner features, and appropriate X max , Y max values are set. In this embodiment, they are respectively 30 and 40 pixel values, representing the maximum range of movement in the x-axis and y-axis directions; traverse the corner features of each target to obtain the central positions X and Y of the target object, and within the intervals (X - X max / 2, X + X max / 2), (Y - Y max / 2, Y + Y maxWithin the range of (1 / 2), perform feature matching on the pixel points of the image; divide the area according to the distribution of feature matching points, fit the distribution of the points according to the Gaussian distribution, and the latest position of the target should at least satisfy that 50% of the feature points fall within the area range, and compare it with the detection range of the target detection model to make a judgment: when the range is smaller than the detection range of the target detection model, then use the detection range of the target detection model, otherwise use the previous detection range as the judgment basis for the corner point tracking in the next frame. After realizing the tracking of the ship target, the calculation of the ship speed vector can be realized, as Figure 2 shown. After calculating its vector, the ship feature point value is used as the id of the ship through hash calculation, and the driving trajectory and speed vector information of the ship in multiple consecutive frames of images are recorded. In this embodiment, the id value of hash is only used as the ship identifier, and when the ship sails out of the detection range, it is cleared and deleted. When the ship travels more than 1 / 3 of the image size, the detection of the water flow movement near the ship is started next to prevent the situation of insufficient ship data and inaccurate prediction.

[0053] In this embodiment, the track prediction of multiple ships is also carried out. The essence of its method is roughly the same as that of the single ship track prediction: in the previous steps, multiple ships are detected, the detection ranges of these ships are marked, and the water flow outside the marked range is processed as described above. The image change process is as Figure 4 shown. It shows that the method of the present invention is applicable to the extraction of water flow movement characteristics under the driving of multiple ships.

[0054] For step S3, the following steps are used to extract the water flow movement target by using the mixture Gaussian background modeling and the frame difference method:

[0055] In this embodiment, input the front and rear two frames of images, perform gray processing, smooth Gaussian filtering with a 3×3 convolution kernel, binary thresholding processing with a 5×5 convolution kernel in sequence. After processing, the two frames of images are subtracted. After hiding the unobvious noise information through 3-core Gaussian blur, perform dilation processing with a 3×3 convolution kernel and contour detection with a 3×3 convolution kernel to obtain the water flow change images in the two frames of images. The specific process of this embodiment refers to Figure 3 ; obtain the contour change range S i through each of the closed-loop water flow contour images formed after contour detection and dilation processing. Traverse from top to bottom within this range to obtain the pixel vertex coordinates, and fit the bottom and top contour waveforms. In this embodiment, the waveform is smoothed first by the simple moving average method, and its calculation formula is h t =(y t-1 +y t-2 +y t-3 +...+y t-n ) / n, h t represents the smoothed result coordinate, n represents the number of moving averages, and yt-n represent the first n moving average results; then use cubic Bezier fitting for the waveform, connect the corner points of the top and bottom waveforms, which represents the water wave flow direction within this water flow area, and the length of the line segment connecting the straight lines is D i , in the case of a known image acquisition frame rate R, calculate each of the water flow vectors D i / R, and the vector display diagram as shown in Figure 5 can be obtained. The vector display diagram of the detected water flow vectors in the multi-ship detection mode is as shown in Figure 6 .

[0056] For step S4, the calculation and prediction are carried out according to the following steps:

[0057] Through the above steps, the water flow vector on the ship's route can be obtained as the sum of the water flow vectors converging from multiple parties V1 = (D1 + D2 +... + D i ) / R, and the ship's travel vector is the ratio of the difference in the ship's position coordinates to the image acquisition frame rate V2 = (X n - X n-1 , Y n - Y n-1 ) / R; assuming the water flow vector weight is W1 and the ship's travel vector weight is W2, through the vector calculation formula V1×W1 + V2×W2, the steps are as shown in Figure 5 , and the predicted value of the ship's speed vector for the next frame is obtained. On the assumed ship's track route, when the water flow speed vectors at each position remain unchanged, when the ship travels to the next frame position, the above vector calculation formula is also used for vector calculation until the ship travels out of the image range, so as to fit the ship's track route, which is expressed by the formula: where m represents that the ship needs to travel out of the detection range m times. In this embodiment, W1 takes the value of 0.08 and W2 takes the value of 0.65.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the purpose and scope of the present technical solution, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A ship track prediction method based on water flow detection, characterized in that, The method specifically includes the following steps: S1: Use a camera to collect multiple frames of image data, and use an object detection model to detect ships; S2: When a detected ship appears, separate the detected ship, extract the corner features of the ship, change the tracking of the moving ship target to the tracking of the corner features, use the optical flow method to track the feature points. When multiple moving ships appear, record the positions of the ships detected in each frame, perform corner feature detection within a certain range near each ship, and judge the tracking situation of the ship according to the distribution of the corner features of the ship in the front and rear two frames of images; S3: On the basis of successful ship detection, based on the collected multiple frames of image data, use a mixture of Gaussian background modeling and frame difference method to extract water flow movement targets, which specifically includes the following steps: S301: Input the front and rear two frames of images, perform gray processing, smooth Gaussian filtering, and binary thresholding in sequence. After processing, subtract the two frames of images, and through dilation processing and contour detection, obtain the water flow change images in the two frames of images; S302: Obtain the contour change range S from each closed-loop water flow contour image formed after contour detection and dilation processing i , traverse from top to bottom within this range to obtain the pixel vertex coordinates, and fit the bottom and top contour waveforms: First, perform waveform smoothing through the simple moving average method, and its calculation formula is h t = (y t-1 + y t-2 + y t-3 +... + y t-n ) / n, where h t represents the smoothed result coordinate, n represents the number of moving averages, and y t-n represents the first n moving average results; then use cubic Bezier to fit the waveform, connect the corner points of the top and bottom contour waveforms, which represents the water wave flow direction within this water flow area, and the line segment length of the connected straight line is D i . Given the known image acquisition frame rate R, calculate each water flow velocity vector D i / R; S4: Based on the multi-frame water flow movement target detection method, calculate the water flow movement vector, perform a weighted vector sum with the ship's travel vector. Under the condition of determining the image frame rate, calculate the next frame position of the ship, and fit the ship's track route according to the velocity vectors of all water flows on the ship's track route, which specifically includes the following steps: S401: Calculate the water flow movement vector and the ship's travel vector on the ship's passing route, and then obtain the predicted value of the ship's next frame velocity vector through the vector calculation formula; S402: Assume that on the ship's track route, when the water flow velocity vectors at each position remain unchanged, when the ship travels to the next frame position, perform vector calculation in the manner of step S401 until the ship travels out of the image range, so as to fit the ship's track route.

2. The ship track prediction method based on water flow detection according to claim 1, wherein In step S1, when detecting ships, it specifically includes the following steps: S101: Before detection, use the training model of the object detection model and the collected ship data set to train a model that can identify ships; S102: Use image annotation software to annotate the collected ship images. After annotation, use the training set to train the model. After the training model is completed, input the test set for testing. When it is found that the model cannot recognize, input the model for retraining until the model is relatively perfect; S103: Use the trained model to detect ships in each frame of image collected by the camera.

3. The ship track prediction method based on water flow detection according to claim 1, wherein In step S2, when extracting the corner features of the ship, it specifically includes the following steps: S201: Calculate the corner features of multiple ships detected by the target detection model, and set appropriate X max , Y max values, which respectively represent the maximum range of movement in the x-axis and y-axis directions; S202: Traverse the corner features of each ship to obtain the central positions X and Y of the ship object, and perform feature matching on the pixel points of the image within the ranges (X - X max / 2, X + X max / 2) and (Y - Y max / 2, Y + Y max / 2); S203: Divide the region according to the distribution of the feature matching points, fit the distribution of the points according to the Gaussian distribution, and the latest position of the target should at least satisfy that 50% of the feature points fall within the region range, and make a comparison and judgment with the detection range of the object detection model: when the range is smaller than the detection range of the object detection model, then use the detection range of the object detection model, otherwise use the previous detection range as the judgment basis for the next frame of corner tracking.

4. The ship track prediction method based on water flow detection according to claim 1, characterized in that, In step S401, the water flow motion vector V1 on the route passed by the ship is equal to the sum of the water flow velocity vectors converging from multiple parties, and the expression is: V1 = (D1 + D2 +... + D i ) / R; The ship's travel vector V2 is equal to the ratio of the difference in the ship's position coordinates to the image acquisition frame rate R, and the expression is: V2 = (X n - X n-1 , Y n - Y n-1 ) / R, where (X n , Y n ) is the ship's position coordinates of the nth frame; The vector calculation formula is: V1×W1 + V2×W2, where W1 is the water flow vector weight and W2 is the ship's travel vector weight.

5. The ship track prediction method based on water flow detection according to claim 4, wherein In step S402, the expression for fitting the ship's track route is: where m represents that the ship needs to travel out of the detection range m times.

6. The ship track prediction method based on water flow detection according to any one of claims 1 to 5, characterized in that The said detection ship can be replaced by any object that can float in water flow.

Images