A ship attitude detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the application provides a ship posture detection method, acquires a monitoring image sequence of a target ship; carries out target detection on each frame of image in the monitoring image sequence, obtains a ship body detection result and a wake detection result of the target ship in each frame of image; and determines the posture of the target ship according to the ship body detection result and the wake detection result in each frame of image. The posture of the target ship is determined by carrying out target detection on the monitoring image sequence of the target ship, using the ship body detection result of the target ship and the wake detection result of the target ship, the wake condition generated by the interaction between the ship body of the target ship and the water surface during the sailing process is considered, and then the posture detection of the target ship can be assisted by the wake condition of the target ship, so that the accuracy of the ship posture detection is improved.

Description

Technical Field

[0001] This invention relates to the field of ship attitude detection technology, and in particular to a ship attitude detection method, device, electronic equipment and storage medium. Background Technology

[0002] When a ship enters a port, the captain or pilot usually adjusts and controls the ship's attitude to dock in a prescribed or safe manner. This requires the captain or pilot to accurately grasp the ship's attitude in order to make timely adjustments. Therefore, ships are equipped with attitude sensor systems to sense the ship's own attitude. However, since the attitude sensor system is installed in the ship's coordinate space, the sensing of the ship's attitude is greatly affected by the spatial range of the ship, and the attitude sensor system itself has a certain systematic error, resulting in a low accuracy rate in detecting the ship's attitude. Summary of the Invention

[0003] This invention provides a ship attitude detection method to address the problem of low accuracy in existing ship attitude detection methods. By performing target detection on a sequence of monitored images of the target ship, the ship's attitude is determined using the detection results of the ship's hull and wake. The method considers the wake generated by the interaction between the ship's hull and the water surface during navigation, thereby using the wake information to assist in the ship's attitude detection and improve its accuracy.

[0004] In a first aspect, embodiments of the present invention provide a method for detecting ship attitude, the method comprising:

[0005] Acquire a sequence of monitoring images of the target vessel;

[0006] Target detection is performed on each frame of the monitoring image sequence to obtain the hull detection result and wake detection result of the target vessel in each frame of the image;

[0007] The attitude of the target vessel is determined based on the hull detection results and the wake detection results in each frame of the image.

[0008] Optionally, acquiring the monitoring image sequence of the target vessel includes:

[0009] When an approaching vessel is detected within the first preset area, the port entry information and navigation parameters of the approaching vessel are obtained.

[0010] Based on the port entry information of the arriving vessel and the navigation parameters, a second preset area and a camera are determined, wherein the area of ​​the second preset area is smaller than that of the first preset area.

[0011] When the incoming vessel is detected to have entered the second preset area, the incoming vessel is identified as the target vessel, and the camera is controlled to track and capture images of the target vessel to obtain a sequence of monitoring images of the target vessel.

[0012] Optionally, determining the second preset area and the imaging equipment based on the port entry information of the arriving vessel and the navigation parameters includes:

[0013] Based on the arrival information of the incoming vessel, determine the reserved berth and planned route of the incoming vessel;

[0014] Based on the navigation parameters of the incoming vessel and the planned route, predict the predicted route of the incoming vessel in the first preset area, and calculate the deviation rate between the predicted route and the planned route.

[0015] When the offset rate is greater than a preset value, it is determined whether the predicted route is a safe route based on the current navigation conditions;

[0016] If the predicted route is determined to be a safe route, then a second preset area and a shooting device are determined based on the predetermined berth and the predicted route.

[0017] Optionally, the step of performing target detection on each frame of the monitored image sequence to obtain the target vessel's hull detection result and wake detection result in each frame includes:

[0018] Perform ship hull detection on each frame of the monitored image sequence to obtain at least one ship hull detection result in each frame.

[0019] Perform wake flow detection on each frame of the monitored image sequence to obtain at least one wake flow detection result in each frame.

[0020] Match at least one of the hull detection results with at least one of the wake detection results in each frame of the image to determine the hull detection result and wake detection result of the target vessel in each frame of the image.

[0021] Optionally, the hull detection result includes the hull area, and the wake detection result includes the wake straightness. Determining the attitude of the target vessel based on the hull detection result and the wake detection result in each frame of the image includes:

[0022] Based on the hull area of ​​the target vessel in two adjacent frames, determine the rate of change of the hull area of ​​the target vessel in two adjacent frames, and determine the rate of change of the area of ​​the target vessel in the monitoring image sequence based on the rate of change of the hull area of ​​the target vessel in two adjacent frames.

[0023] Based on the wake straightness of the target vessel in two adjacent frames, determine the rate of change of the wake straightness of the target vessel in two adjacent frames, and determine the sequence of the rate of change of the wake straightness of the target vessel in the monitoring image sequence based on the rate of change of the wake straightness of the target vessel in two adjacent frames.

[0024] The attitude of the target vessel is determined based on the area change rate sequence and the straightness change rate sequence.

[0025] Optionally, determining the attitude of the target vessel based on the area change rate sequence and the straightness change rate sequence includes:

[0026] The area change rate sequence and the flatness change rate sequence are fused to obtain a fused sequence;

[0027] The area change rate sequence and the flatness change rate sequence are input into a trained time series prediction model for prediction processing to obtain the attitude of the target ship.

[0028] Optionally, before inputting the area change rate sequence and the flatness change rate sequence into the trained time-series prediction model for prediction processing to obtain the attitude of the target ship, the method further includes:

[0029] Obtain a training dataset and a time-series prediction model to be trained. The training dataset includes a sample fusion sequence and corresponding pose labels. The sample fusion sequence and the fusion sequence are obtained using the same method.

[0030] The time series prediction model to be trained is subjected to supervised training using the training dataset, and a trained time series prediction model is obtained after training is completed.

[0031] Secondly, embodiments of the present invention also provide a ship attitude detection device, the ship attitude detection device comprising:

[0032] The first acquisition module is used to acquire the monitoring image sequence of the target vessel;

[0033] The detection module is used to perform target detection on each frame of the monitored image sequence to obtain the hull detection result and wake detection result of the target vessel in each frame of the image.

[0034] The determination module is used to determine the attitude of the target vessel based on the hull detection results and the wake detection results in each frame of the image.

[0035] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the ship attitude detection method provided in embodiments of the present invention.

[0036] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the ship attitude detection method provided in the embodiments of the present invention.

[0037] In this embodiment of the invention, a monitoring image sequence of the target vessel is acquired; target detection is performed on each frame of the monitoring image sequence to obtain the hull detection result and wake detection result of the target vessel in each frame; the attitude of the target vessel is determined based on the hull detection result and wake detection result in each frame. By performing target detection on the monitoring image sequence of the target vessel and using the hull detection result and wake detection result of the target vessel to determine the attitude of the target vessel, the wake situation generated by the interaction between the hull and the water surface during the navigation of the target vessel is taken into account. Therefore, the wake situation of the target vessel can be used to assist in the attitude detection of the target vessel, thereby improving the accuracy of the vessel attitude detection. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of a ship attitude detection method provided in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the structure of a ship attitude detection device provided in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, Figure 1 This is a flowchart of a ship attitude detection method provided in an embodiment of the present invention. The ship attitude detection method includes the following steps:

[0044] 101. Obtain the monitoring image sequence of the target vessel.

[0045] In this embodiment of the invention, the target vessel is a vessel that needs to enter the port for berthing. Specifically, the target vessel is a vessel that needs to enter a specific area for berthing, and the specific area can be a specific range outside the port.

[0046] An electronic fence can be set up at the boundary of a specific area to detect whether a vessel is entering that specific area. If a vessel is detected entering the specific area, its port entry information can be obtained by establishing communication. The port entry information may include information such as port entry permission, reserved berth, and planned route. If the port entry permission is verified, the vessel can be identified as the target vessel, and the corresponding camera equipment can be controlled to track and photograph the target vessel to obtain a monitoring image sequence of the target vessel.

[0047] It should be noted that the above monitoring image sequence includes multiple frames of images. The above monitoring image sequence can be the original monitoring video or the monitoring video after all frames have been extracted. Each frame of the above monitoring image sequence includes the target vessel and the wake of the target vessel.

[0048] 102. Perform target detection on each frame of the monitored image sequence to obtain the target vessel's hull detection results and wake detection results in each frame.

[0049] In this embodiment of the invention, the above-mentioned monitoring image sequence includes multiple frame images, each frame image including the target ship and the wake of the target ship. The hull and wake of the target ship can be detected by a trained target detection model, thereby obtaining the hull detection result and the wake detection result of the target ship.

[0050] The object detection described above can be achieved through an object detection model, which can be an object detection model based on a deep convolutional neural network, such as an object detection model based on convolutional neural networks like R-CNN, Faster R-CNN, or YOLO.

[0051] The above hull detection results can be represented by a hull detection frame (x1, y1, w1, h1, s, v1), where (x1, y1) represents the coordinates of the center point of the hull detection frame, w1 represents the width of the hull detection frame, h1 represents the height of the hull detection frame, s represents the hull area, and v1 represents the target vessel's hull vector velocity. The above wake detection results can be represented by a wake detection frame (x2, y2, w2, h2, c, v2), where (x2, y2) represents the coordinates of the center point of the wake detection frame, w2 represents the width of the wake detection frame, h2 represents the height of the wake detection frame, c represents the wake straightness, and v2 represents the target vessel's wake vector velocity.

[0052] It should be noted that the aforementioned hull area refers to the area of ​​the hull within the frame image. The aforementioned wake straightness refers to the degree of straightness of the wake. Wake straightness represents the proportion of the straight section of the wake to the total length of the wake. A wake straightness of 1 indicates that the ship is traveling straight, while a wake straightness less than 1 indicates that the ship is not traveling straight.

[0053] 103. Determine the attitude of the target vessel based on the hull detection results and wake detection results in each frame of the image.

[0054] In this embodiment of the invention, after obtaining the hull detection results and wake detection results in each frame of the image, the attitude of the target vessel can be determined by the hull vector velocity and wake vector velocity in the hull detection results. Since the rotation of the hull affects the formation and shape of the wake, this effect has a certain delay. For example, when the hull is not rotating, the hull vector velocity and the wake vector velocity are consistent. When the hull begins to rotate, the hull vector velocity changes, while the wake vector velocity has not yet changed, resulting in an inconsistency between the hull vector velocity and the wake vector velocity.

[0055] Specifically, the vector angle between the ship's hull vector velocity and the wake vector velocity in the same frame can be calculated. This vector angle indicates whether the hull and wake vector velocities are consistent. If the vector angle is 0 or less than a preset angle, it can be determined that the target ship's hull and wake vector velocities are consistent. If the vector angle is not 0 or greater than the preset angle, it can be determined that the target ship's hull and wake vector velocities are inconsistent. When the target ship's hull and wake vector velocities are consistent, the target ship's attitude can be determined based on the hull vector velocity. When the target ship's hull and wake vector velocities are inconsistent, the target ship's attitude can be determined based on the hull vector velocity plus the corresponding vector angle. The corresponding attitude can be determined based on the direction in the vector velocity. This direction can be in the camera coordinate system, which can be converted to the real coordinate system, and the real coordinate system direction can be used as the target ship's attitude.

[0056] The attitude of a target vessel can also be determined based on spatiotemporal information. Specifically, the attitude can be determined by the changes in the vessel's hull and wake between different frames. For example, the attitude can be determined based on the rate of change of the target vessel's hull vector velocity and the rate of change of the target vessel's wake vector velocity, or it can be determined based on the rate of change of the target vessel's hull area and the rate of change of the target vessel's wake straightness. Determining the attitude of a target vessel through spatiotemporal information allows for consideration of attitude changes in both time and space, thus making the attitude detection results more accurate.

[0057] In one possible embodiment, the above-described ship attitude detection method is applied to a server. After obtaining the attitude of the target ship, the attitude of the target ship can be sent to a user terminal for display. The user terminal can be a fixed terminal on the target ship, a mobile terminal of the captain or pilot, or a fixed terminal of the port management department.

[0058] In this embodiment of the invention, a monitoring image sequence of the target vessel is acquired; target detection is performed on each frame of the monitoring image sequence to obtain the hull detection result and wake detection result of the target vessel in each frame; the attitude of the target vessel is determined based on the hull detection result and wake detection result in each frame. By performing target detection on the monitoring image sequence of the target vessel and using the hull detection result and wake detection result of the target vessel to determine the attitude of the target vessel, the wake situation generated by the interaction between the hull and the water surface during the navigation of the target vessel is taken into account. Therefore, the wake situation of the target vessel can be used to assist in the attitude detection of the target vessel, thereby improving the accuracy of the vessel attitude detection.

[0059] It is understood that in the specific embodiments of this application, data such as monitoring image sequences and monitoring videos of ships or ports are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0060] Optionally, in the step of acquiring the monitoring image sequence of the target vessel, when an approaching vessel is detected in the first preset area, the port entry information and navigation parameters of the approaching vessel are acquired; based on the port entry information and the navigation parameters, a second preset area and a shooting device are determined, wherein the area of ​​the second preset area is smaller than that of the first preset area; when an approaching vessel is detected to enter the second preset area, the approaching vessel is determined to be the target vessel, and the shooting device is controlled to track and shoot the target vessel to obtain the monitoring image sequence of the target vessel.

[0061] In this embodiment of the invention, the first preset area is a fixed, specific area used to detect whether there are incoming vessels. This specific area can be a specific range outside the port. An electronic fence can be set at the boundary of the specific area. The electronic fence is used to detect whether a vessel enters the specific area, and the vessel entering the specific area is identified as an incoming vessel. If an incoming vessel is detected entering the specific area, its port entry information and navigation parameters can be obtained by establishing communication with the incoming vessel. The port entry information may include port entry permissions, reserved berths, planned routes, etc. The navigation parameters may include control parameters pre-set by the incoming vessel according to the planned route and the incoming vessel's own status parameters.

[0062] After obtaining the arrival information and navigation parameters of the incoming vessel, the destination of the vessel can be determined based on the predetermined berth in the arrival information. Using the navigation parameters, the predicted route of the incoming vessel between the destination and a first preset area is predicted. A second preset area is then determined based on the predicted route, which can be an extension of the predicted route. After predicting the route, corresponding camera equipment can be determined. This camera can be a fixed camera in the port area, capable of capturing images of the second preset area between the berth and the first preset area. Specifically, multiple camera devices are installed in the port area. After obtaining the predicted route, the optimal shooting position for the preset route can be determined based on the predicted route and the positions of the multiple camera devices. The camera corresponding to the optimal shooting position is then designated as the camera device for tracking and capturing images of the incoming vessel within the second preset area.

[0063] In one possible embodiment, a first mapping table can be pre-constructed, which includes the mapping relationship between port entry information and navigation parameters and a second preset area and a shooting device. Given the port entry information and navigation parameters, the corresponding second preset area and shooting device can be found through the mapping relationship recorded in the first mapping table.

[0064] In another possible embodiment, a second mapping table can be pre-constructed. This second mapping table includes a mapping relationship between the predicted route of the arriving vessel and the second preset area, as well as the imaging equipment, which can be predicted using the navigation parameters of the arriving vessel between the destination and the first preset area. After obtaining the port entry information and the navigation parameters of the arriving vessel, the destination of the arriving vessel can be determined based on the predetermined berth in the port entry information. The predicted route of the arriving vessel can be predicted between the destination and the first preset area using the navigation parameters of the arriving vessel. The second preset area and the imaging equipment corresponding to the predicted route can be found through the mapping relationship recorded in the second mapping table.

[0065] When an incoming vessel enters the second preset area from the first preset area, the incoming vessel can be identified as the target vessel, and the camera can be controlled to track and photograph the target vessel, thereby obtaining a sequence of monitoring images of the target vessel in the second preset area.

[0066] Optionally, in the step of determining the second preset area and the camera equipment based on the port entry information and navigation parameters of the incoming vessel, the predetermined berth and planned route of the incoming vessel can be determined based on the port entry information of the incoming vessel; the predicted route of the incoming vessel in the first preset area can be predicted based on the navigation parameters and planned route of the incoming vessel, and the deviation rate between the predicted route and the planned route can be calculated; if the deviation rate is greater than a preset value, it can be determined whether the predicted route is a safe route based on the current navigation conditions; if the predicted route is determined to be a safe route, the second preset area and the camera equipment can be determined based on the predetermined berth and the predicted route.

[0067] In this embodiment of the invention, after a vessel enters the first preset area, it can be identified as an approaching vessel. A communication link is established with the approaching vessel to obtain its port entry information and navigation parameters. The port entry information may include port entry permissions, reserved berths, planned routes, etc. The navigation parameters may include control parameters pre-set by the approaching vessel according to the planned route and the vessel's own status parameters.

[0068] After obtaining the arrival information of the incoming vessel and the navigation parameters, the destination of the incoming vessel can be determined based on the predetermined berth in the arrival information, and the predicted route of the incoming vessel can be predicted using the aforementioned navigation parameters. Specifically, the navigation parameters and current environmental information can be combined with the current port environment information, input into route simulation software for simulation processing, and the predicted route can be obtained. After obtaining the predicted route, the positional distance between the predicted route and the planned route at each time point is calculated, and then the average positional distance at all times is calculated as the deviation rate between the predicted route and the planned route.

[0069] After obtaining the deviation rate between the predicted and planned routes, it can be determined whether the deviation rate is greater than a preset value. If the deviation rate is greater than the preset value, it indicates a significant deviation between the predicted and planned routes, and the approaching vessel will deviate considerably from the planned route according to the navigation parameters. If the deviation rate is less than or equal to the preset value, it indicates a smaller deviation between the predicted and planned routes, and the approaching vessel will deviate less from the planned route. When the deviation rate between the predicted and planned routes is greater than the preset value, it can be further determined whether the predicted route is a safe route based on the current navigation conditions of the port. These current navigation conditions may include water depth, wind force, and the presence of other vessels along the predicted route. If the predicted route is determined to be a safe route, a second preset area can be determined based on the predicted route and the designated berth, and the camera equipment can be selected based on the predicted route. If the predicted route is determined to be an unsafe route, the approaching vessel is notified so that it can correct its navigation parameters.

[0070] By determining the second preset area through the predicted route, the width of the predicted route between the first preset area and the predetermined berth can be expanded to obtain the second preset area.

[0071] After predicting the flight path, the corresponding camera can be determined based on the predicted path. This camera can be a fixed camera in the port area, capable of capturing images of a second preset area between the berth and the first preset area. Specifically, multiple cameras are installed in the port area. After obtaining the predicted path, the optimal shooting position for the preset path can be determined based on the predicted path and the positions of the multiple cameras. The camera corresponding to the optimal shooting position is then designated as the camera to track and capture incoming vessels within the second preset area. The optimal shooting position can be a location where the angle with the predicted path is within the range of (θ1, θ2). There can be one or more cameras; when there are multiple cameras, one camera corresponds to one shooting time period to ensure that the target vessel's hull and wake can be tracked and captured within the range of (θ1, θ2) by multiple cameras.

[0072] When an incoming vessel enters the second preset area from the first preset area, the incoming vessel can be identified as the target vessel, and the camera can be controlled to track and photograph the target vessel, thereby obtaining a sequence of monitoring images of the target vessel in the second preset area.

[0073] Optionally, in the step of performing target detection on each frame of the monitoring image sequence to obtain the hull detection result and wake detection result of the target vessel in each frame of the image, hull detection can be performed on each frame of the monitoring image sequence to obtain at least one hull detection result in each frame of the image; wake detection can be performed on each frame of the monitoring image sequence to obtain at least one wake detection result in each frame of the image; and at least one hull detection result in each frame of the image can be matched with at least one wake detection result to determine the hull detection result and wake detection result of the target vessel in each frame of the image.

[0074] In this embodiment of the invention, in the monitored image sequence, there may be multiple ships in one frame of the image. Two target detection models can be used to detect the hull and wake of the ships in each frame of the image respectively. For example, the hull detection model can be used to detect the hull of the target ship to obtain the hull detection results of all ships (x1, y1, w1, h1, s, v1). i The wake detection model is used to detect the wake of ships in each frame of the image, and the wake detection results of all ships (x2, y2, w2, h2, c, v2) are obtained. j The aforementioned hull detection model and wake detection model are both detection models built based on deep convolutional neural networks. They can be target detection models based on convolutional neural networks such as R-CNN, Faster R-CNN, and YOLO. The hull detection results (x1, y1, w1, h1, s, v1) are obtained. i And wake detection results (x2, y2, w2, h2, c, v2) j Then, the ship hull detection results (x1, y1, w1, h1, s, v1) can be obtained. i And wake detection results (x2, y2, w2, h2, c, v2) j Matching is performed to obtain the hull detection results and wake detection results of the target ship in the frame image.

[0075] Alternatively, a target detection model can be used to detect the hull and wake of the target vessel. This target detection model can be based on a deep convolutional neural network. Furthermore, the target detection model includes a common network, a hull detection branch network, a wake detection branch network, and an output network. The common network extracts common features of the hull and wake; the hull detection branch network extracts and detects hull features from the common features; the wake detection branch network extracts and detects wake features from the common features; and the output network outputs the hull detection results (x1, y1, w1, h1, s, v1) and wake detection results (x2, y2, w2, h2, c, v2) for each vessel. The target vessel's hull and wake detection results are then determined from the hull and wake detection results of each vessel. The aforementioned object detection model is obtained through supervised training on a pre-prepared training set, which includes sample images of different types of ships. Each sample image contains the ship and its corresponding wake. Each ship has a corresponding hull label and wake label. The sample images are input into the object detection model to be trained for processing, resulting in hull detection results and wake detection results. The error loss between the hull detection result and the corresponding hull label is calculated to obtain the hull loss function. Similarly, the error loss between the wake detection result and the wake label is calculated to obtain the wake loss function. The hull loss function and the wake loss function are added together to obtain the total loss function. The optimization objective is to minimize the total loss function. The parameters of the object detection model are adjusted through backpropagation. This training process is iterated until the object detection model converges at the minimum of the total loss function or the number of iterations reaches a preset value. Training then stops, resulting in a well-trained object detection model. It should be noted that the training methods for the aforementioned hull detection model and wake detection model are similar to those for the aforementioned object detection model. The difference lies in that the sample images in the training set used for the hull detection model include the hull, with each hull corresponding to a hull label, and the optimization objective is to minimize the hull loss function. In contrast, the sample images in the training set used for the wake detection model include the wake of the corresponding vessel, with each wake corresponding to a wake label, and the optimization objective is to minimize the wake loss function.

[0076] Optionally, the hull detection results include the hull area, and the wake detection results include the wake straightness. In the step of determining the attitude of the target vessel based on the hull detection results and wake detection results in each frame of the image, the area change rate of the target vessel's hull area in two adjacent frames can be determined based on the hull area corresponding to the target vessel in two adjacent frames, and the area change rate sequence of the target vessel in the monitoring image sequence can be determined based on the area change rate of the target vessel's hull area in two adjacent frames; the straightness change rate of the target vessel's wake straightness in two adjacent frames can be determined based on the wake straightness corresponding to the target vessel in two adjacent frames, and the straightness change rate sequence of the target vessel in the monitoring image sequence can be determined based on the straightness change rate of the target vessel's wake straightness in two adjacent frames; the attitude of the target vessel can be determined based on the area change rate sequence and the straightness change rate sequence.

[0077] In this embodiment of the invention, the above-mentioned hull detection result can be (x1, y1, w1, h1, s), where (x1, y1) represents the coordinates of the center point of the hull detection frame, w1 represents the width of the hull detection frame, h1 represents the height of the hull detection frame, and s represents the hull area; the above-mentioned wake detection result can be (x2, y2, w2, h2, c), where (x2, y2) represents the coordinates of the center point of the wake detection frame, w2 represents the width of the wake detection frame, h2 represents the height of the wake detection frame, and c represents the wake straightness.

[0078] Generally, two adjacent frames are the (n-1)th frame and the nth frame, and the rate of change of the ship's hull area Δs between the two adjacent frames is... n It can be the area s of the ship's hull in the (n-1)th frame image. n-1 The hull area s in the nth frame image n The ratio of the straightness of the wake straightness in two adjacent frames. n It can be the wake flatness c in the (c-1)th frame image. n-1 The wake straightness c in the nth frame image n The ratio of .

[0079] The rate of change of the hull area Δs in all adjacent frames of the monitored image sequence was obtained. n and straightness change rate Δc n Then, the area change rate Δs can be... n Arrange the frames in chronological order to obtain the area change rate sequence (Δs1, Δs2, ..., Δs) of the target ship in the monitored image sequence. n The rate of change of flatness Δc can be used to... n Arrange the frames sequentially to obtain the flatness change rate sequence (Δc1, Δc2, ..., Δc) of the target vessel in the monitored image sequence. n).

[0080] After obtaining the area change rate sequence (△s1, △s2, ..., △s...), n ) and the sequence of changes in straightness (Δc1, Δc2, ..., Δc) n After that, the first attitude of the target ship can be predicted based on the area change rate sequence, and the second attitude can be predicted based on the straightness change rate sequence. When the first attitude and the second attitude are consistent, the first attitude is determined to be the attitude of the target ship. When the first attitude and the second attitude are inconsistent, since the wake usually tends to converge with the ship's, the second attitude can be determined to be the attitude of the target ship.

[0081] Different RNN recurrent network models can be used to predict the area change rate sequence and the flatness change rate sequence, respectively. Different RNN recurrent network models are obtained through supervised training on different training sets.

[0082] Optionally, in the step of determining the attitude of the target ship based on the area change rate sequence and the straightness change rate sequence, the area change rate sequence and the straightness change rate sequence can be fused to obtain a fused sequence; the area change rate sequence and the straightness change rate sequence can be input into a trained time series prediction model for prediction processing to obtain the attitude of the target ship.

[0083] In this embodiment of the invention, the area change rate sequence (△s1, △s2, ..., △s) can be used. n ) and the sequence of flatness change rate (△c1, △c2, ..., △c n The fusion process is performed to obtain the fusion sequence (△s1, △s2, ..., △s). n ; △c1, △c2, ..., △c n The fused sequence is input into a trained temporal prediction model for prediction processing, and the predicted attitude is output as the attitude of the target ship. The aforementioned temporal prediction model can be an RNN recurrent network model, an LSTM long short-term memory network model, or other temporal models.

[0084] By using a time-series prediction model to extract time-series information from the fused sequence, the attitude of the target vessel can be determined more accurately.

[0085] Optionally, before inputting the area change rate sequence and the straightness change rate sequence into the trained time series prediction model for prediction processing to obtain the attitude of the target ship, a training dataset and a time series prediction model to be trained can be obtained. The training dataset includes a sample fusion sequence and the corresponding attitude label. The sample fusion sequence and the fusion sequence are obtained using the same method. The time series prediction model to be trained is then trained in a supervised manner using the training dataset, and the trained time series prediction model is obtained after training is completed.

[0086] In this embodiment of the invention, the time-series prediction model to be trained can be a recurrent neural network (RNN) model, a long short-term memory (LSTM) model, or other time-series models. The training dataset includes sample fusion sequences, which are obtained in the same way as those in the previous embodiment. Each sample fusion sequence corresponds to a pose label. During training, the sample fusion sequences are input into the time-series prediction model to be trained to obtain prediction results. The error loss between the prediction results and the pose labels of the sample fusion sequences is calculated. Minimizing the error loss is the optimization objective. The parameters of the time-series prediction model to be trained are adjusted using backpropagation, and this parameter adjustment process is iterated until the time-series prediction model converges at the point of minimum error loss, or the number of iterations reaches a preset number. Training is then stopped, and a trained time-series prediction model is obtained.

[0087] In one possible embodiment, the sample fusion sequence can be preprocessed before training. The preprocessing may include randomly masking the elements in the sample fusion sequence, extracting or inserting frames into the sample fusion sequence. Frame extraction shortens the length of the sample fusion sequence, while frame insertion increases the length of the sample fusion sequence, thereby increasing the amount of sample data in the training dataset and improving the training effect.

[0088] It should be noted that the ship attitude detection method provided in this embodiment of the invention can be applied to devices such as smartphones, computers, and servers that can perform ship attitude detection.

[0089] like Figure 2 As shown, an embodiment of the present invention provides a ship attitude detection device, which includes:

[0090] The first acquisition module 201 is used to acquire a sequence of monitoring images of the target vessel;

[0091] The detection module 202 is used to perform target detection on each frame of the monitored image sequence to obtain the hull detection result and wake detection result of the target vessel in each frame of the image.

[0092] The determination module 203 is used to determine the attitude of the target vessel based on the hull detection results and the wake detection results in each frame of the image.

[0093] Optionally, the first acquisition module 201 includes:

[0094] The acquisition submodule is used to acquire the port entry information and navigation parameters of the incoming vessel when an incoming vessel is detected in the first preset area.

[0095] The first processing submodule is used to determine a second preset area and a shooting device based on the port entry information of the incoming vessel and the navigation parameters. The area of ​​the second preset area is smaller than that of the first preset area.

[0096] The second processing submodule is used to determine that the incoming vessel is the target vessel when it is detected that the incoming vessel has entered the second preset area, and to control the shooting device to track and shoot the target vessel to obtain a monitoring image sequence of the target vessel.

[0097] Optionally, the first processing submodule includes:

[0098] The determining unit is used to determine the predetermined berth and planned route of the incoming vessel based on the port entry information of the incoming vessel.

[0099] The first processing unit is configured to predict the predicted route of the incoming vessel in the first preset area based on the navigation parameters of the incoming vessel and the planned route, and to calculate the offset rate between the predicted route and the planned route.

[0100] The second processing unit is used to determine whether the predicted route is a safe route based on the current navigation conditions when the offset rate is greater than a preset value.

[0101] The third processing unit is used to determine a second preset area and a shooting device based on the predetermined berth and the predicted route if the predicted route is determined to be a safe route.

[0102] Optionally, the detection module 202 includes:

[0103] The first detection submodule is used to perform ship hull detection on each frame of the monitored image sequence to obtain at least one ship hull detection result in each frame.

[0104] The second detection submodule is used to perform wake detection on each frame of the monitored image sequence to obtain at least one wake detection result in each frame.

[0105] The third processing submodule is used to match at least one of the hull detection results with at least one of the wake detection results in each frame of the image to determine the hull detection result and wake detection result of the target vessel in each frame of the image.

[0106] Optionally, the determining module 203 includes:

[0107] The fourth processing submodule is used to determine the area change rate of the target ship's hull area in two adjacent frames based on the corresponding hull area of ​​the target ship in two adjacent frames, and to determine the area change rate sequence of the target ship in the monitoring image sequence based on the area change rate of the target ship's hull area in two adjacent frames.

[0108] The fifth processing submodule is used to determine the straightness change rate of the wake straightness of the target vessel in two adjacent frames of images based on the wake straightness of the target vessel in two adjacent frames of images, and to determine the straightness change rate sequence of the target vessel in the monitoring image sequence based on the straightness change rate of the wake straightness of the target vessel in two adjacent frames of images.

[0109] The determination submodule is used to determine the attitude of the target ship based on the area change rate sequence and the straightness change rate sequence.

[0110] Optionally, the determining submodule includes:

[0111] The fourth processing unit is used to fuse the area change rate sequence and the flatness change rate sequence to obtain a fused sequence;

[0112] The fifth processing unit is used to input the area change rate sequence and the straightness change rate sequence into the trained time series prediction model for prediction processing to obtain the attitude of the target ship.

[0113] Optionally, the device further includes:

[0114] The second acquisition module is used to acquire the training dataset and the time-series prediction model to be trained. The training dataset includes a sample fusion sequence and the corresponding pose label. The sample fusion sequence and the fusion sequence are obtained by the same method.

[0115] The training module is used to perform supervised training on the time series prediction model to be trained using the training dataset, and the trained time series prediction model is obtained after training is completed.

[0116] It should be noted that the ship attitude detection device provided in this embodiment of the invention can be applied to devices such as smartphones, computers, and servers that can perform ship attitude detection.

[0117] The ship attitude detection device provided in this embodiment of the invention can realize all the processes implemented by the ship attitude detection method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0118] See Figure 3 , Figure 3This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3 As shown, it includes: a memory 302, a processor 301, and a computer program for a ship attitude detection method stored in the memory 302 and executable on the processor 301, wherein:

[0119] The processor 301 is used to call the computer program stored in the memory 302 and perform the following steps:

[0120] Acquire a sequence of monitoring images of the target vessel;

[0121] Target detection is performed on each frame of the monitoring image sequence to obtain the hull detection result and wake detection result of the target vessel in each frame of the image;

[0122] The attitude of the target vessel is determined based on the hull detection results and the wake detection results in each frame of the image.

[0123] Optionally, the acquisition of the monitoring image sequence of the target vessel executed by processor 301 includes:

[0124] When an approaching vessel is detected within the first preset area, the port entry information and navigation parameters of the approaching vessel are obtained.

[0125] Based on the port entry information of the arriving vessel and the navigation parameters, a second preset area and a camera are determined, wherein the area of ​​the second preset area is smaller than that of the first preset area.

[0126] When the incoming vessel is detected to have entered the second preset area, the incoming vessel is identified as the target vessel, and the camera is controlled to track and capture images of the target vessel to obtain a sequence of monitoring images of the target vessel.

[0127] Optionally, the step of processor 301 determining the second preset area and the imaging device based on the port entry information of the arriving vessel and the navigation parameters includes:

[0128] Based on the arrival information of the incoming vessel, determine the reserved berth and planned route of the incoming vessel;

[0129] Based on the navigation parameters of the incoming vessel and the planned route, predict the predicted route of the incoming vessel in the first preset area, and calculate the deviation rate between the predicted route and the planned route.

[0130] When the offset rate is greater than a preset value, it is determined whether the predicted route is a safe route based on the current navigation conditions;

[0131] If the predicted route is determined to be a safe route, then a second preset area and a shooting device are determined based on the predetermined berth and the predicted route.

[0132] Optionally, the processor 301 performs target detection on each frame of the monitored image sequence to obtain the target vessel's hull detection result and wake detection result in each frame, including:

[0133] Perform ship hull detection on each frame of the monitored image sequence to obtain at least one ship hull detection result in each frame.

[0134] Perform wake flow detection on each frame of the monitored image sequence to obtain at least one wake flow detection result in each frame.

[0135] Match at least one of the hull detection results with at least one of the wake detection results in each frame of the image to determine the hull detection result and wake detection result of the target vessel in each frame of the image.

[0136] Optionally, the hull detection result includes the hull area, the wake detection result includes the wake straightness, and the step of determining the attitude of the target vessel based on the hull detection result and the wake detection result in each frame of the image executed by the processor 301 includes:

[0137] Based on the hull area of ​​the target vessel in two adjacent frames, determine the rate of change of the hull area of ​​the target vessel in two adjacent frames, and determine the rate of change of the area of ​​the target vessel in the monitoring image sequence based on the rate of change of the hull area of ​​the target vessel in two adjacent frames.

[0138] Based on the wake straightness of the target vessel in two adjacent frames, determine the rate of change of the wake straightness of the target vessel in two adjacent frames, and determine the sequence of the rate of change of the wake straightness of the target vessel in the monitoring image sequence based on the rate of change of the wake straightness of the target vessel in two adjacent frames.

[0139] The attitude of the target vessel is determined based on the area change rate sequence and the straightness change rate sequence.

[0140] Optionally, the process executed by processor 301 to determine the attitude of the target vessel based on the area change rate sequence and the flatness change rate sequence includes:

[0141] The area change rate sequence and the flatness change rate sequence are fused to obtain a fused sequence;

[0142] The area change rate sequence and the straightness change rate sequence are input into a trained time series prediction model for prediction processing to obtain the attitude of the target ship.

[0143] Optionally, before inputting the area change rate sequence and the flatness change rate sequence into the trained time-series prediction model for prediction processing to obtain the attitude of the target ship, the method executed by the processor 301 further includes:

[0144] Obtain a training dataset and a time-series prediction model to be trained. The training dataset includes a sample fusion sequence and corresponding pose labels. The sample fusion sequence and the fusion sequence are obtained using the same method.

[0145] The time series prediction model to be trained is subjected to supervised training using the training dataset, and a trained time series prediction model is obtained after training is completed.

[0146] It should be noted that the electronic device provided in the embodiments of the present invention can be applied to devices such as smartphones, computers, and servers that can perform ship attitude detection methods.

[0147] The electronic device provided in this embodiment of the invention can implement all the processes of the ship attitude detection method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0148] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the ship attitude detection method or the application-side ship attitude detection method provided in this invention, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0149] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0150] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for detecting ship attitude, characterized in that, The method includes the following steps: Acquire a sequence of monitoring images of the target vessel; Target detection is performed on each frame of the monitored image sequence to obtain the hull detection result and wake detection result of the target vessel in each frame. Specifically, hull detection is performed on each frame of the monitored image sequence to obtain at least one hull detection result in each frame; wake detection is performed on each frame of the monitored image sequence to obtain at least one wake detection result in each frame; at least one hull detection result in each frame is matched with at least one wake detection result to determine the hull detection result and wake detection result of the target vessel in each frame. The hull detection result includes the hull area, and the wake detection result includes the wake straightness. Based on the hull detection results and wake detection results in each frame of the image, the attitude of the target vessel is determined. Specifically, based on the hull area of ​​the target vessel in two adjacent frames, the area change rate of the hull area in those two frames is determined, and the area change rate sequence of the target vessel in the monitoring image sequence is determined based on the area change rate of the hull area in those two adjacent frames. Based on the wake straightness of the target vessel in two adjacent frames, the straightness change rate of the wake straightness in those two adjacent frames is determined, and the straightness change rate sequence of the target vessel in the monitoring image sequence is determined based on the straightness change rate of the wake straightness in those two adjacent frames. Based on the area change rate sequence and the straightness change rate sequence, the attitude of the target vessel is determined.

2. The ship attitude detection method as described in claim 1, characterized in that, The acquisition of the monitoring image sequence of the target vessel includes: When an approaching vessel is detected within the first preset area, the port entry information and navigation parameters of the approaching vessel are obtained. Based on the port entry information of the arriving vessel and the navigation parameters, a second preset area and a camera are determined, wherein the area of ​​the second preset area is smaller than that of the first preset area. When the incoming vessel is detected to have entered the second preset area, the incoming vessel is identified as the target vessel, and the camera is controlled to track and capture images of the target vessel to obtain a sequence of monitoring images of the target vessel.

3. The ship attitude detection method as described in claim 2, characterized in that, The step of determining the second preset area and the imaging equipment based on the port entry information of the arriving vessel and the navigation parameters includes: Based on the arrival information of the incoming vessel, determine the reserved berth and planned route of the incoming vessel; Based on the navigation parameters of the incoming vessel and the planned route, predict the predicted route of the incoming vessel in the first preset area, and calculate the deviation rate between the predicted route and the planned route. When the offset rate is greater than a preset value, it is determined whether the predicted route is a safe route based on the current navigation conditions; If the predicted route is determined to be a safe route, then a second preset area and a shooting device are determined based on the predetermined berth and the predicted route.

4. The ship attitude detection method as described in claim 1, characterized in that, Determining the attitude of the target vessel based on the area change rate sequence and the straightness change rate sequence includes: The area change rate sequence and the flatness change rate sequence are fused to obtain a fused sequence; The area change rate sequence and the flatness change rate sequence are input into a trained time series prediction model for prediction processing to obtain the attitude of the target ship.

5. The ship attitude detection method as described in claim 4, characterized in that, Before inputting the area change rate sequence and the flatness change rate sequence into the trained time-series prediction model for prediction processing to obtain the attitude of the target ship, the method further includes: Obtain a training dataset and a time-series prediction model to be trained. The training dataset includes a sample fusion sequence and corresponding pose labels. The sample fusion sequence and the fusion sequence are obtained using the same method. The time series prediction model to be trained is subjected to supervised training using the training dataset, and a trained time series prediction model is obtained after training is completed.

6. A ship attitude detection device, characterized in that, The ship attitude detection device includes: The first acquisition module is used to acquire the monitoring image sequence of the target vessel; The detection module is used to perform target detection on each frame of the monitored image sequence to obtain the hull detection result and wake detection result of the target vessel in each frame. Specifically, it performs hull detection on each frame of the monitored image sequence to obtain at least one hull detection result in each frame; it performs wake detection on each frame of the monitored image sequence to obtain at least one wake detection result in each frame; and it matches at least one hull detection result with at least one wake detection result in each frame to determine the hull detection result and wake detection result of the target vessel in each frame. The hull detection result includes the hull area, and the wake detection result includes the wake straightness. The determination module is used to determine the attitude of the target vessel based on the hull detection results and the wake detection results in each frame of the image. Specifically, based on the hull area of ​​the target vessel in two adjacent frames, the module determines the area change rate of the hull area in the two adjacent frames, and determines the area change rate sequence of the target vessel in the monitoring image sequence based on the area change rate of the hull area in the two adjacent frames; based on the wake straightness of the target vessel in two adjacent frames, the module determines the straightness change rate of the wake straightness in the two adjacent frames, and determines the straightness change rate sequence of the target vessel in the monitoring image sequence based on the straightness change rate of the wake straightness in the two adjacent frames; and determines the attitude of the target vessel based on the area change rate sequence and the straightness change rate sequence.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the ship attitude detection method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the ship attitude detection method as described in any one of claims 1 to 5.

Images