An autonomous target detection and tracking method for marine life-saving devices

Abstract

The present invention discloses an autonomous target detection and tracking method for a marine life-saving device. An image acquisition device, a wireless image transmission module, and a two-degree-of-freedom pan-tilt are installed on the life-saving device. The image acquisition device is installed on the two-degree-of-freedom pan-tilt. The image acquisition device obtains image information, and the mother ship performs image processing and generates control data, which is transmitted back to the life-saving device through wireless transmission to complete the motion control of the two-degree-of-freedom pan-tilt and the life-saving device. Alternatively, an edge computing module is installed on the life-saving device, and image processing is performed directly on the life-saving device. After the target position is obtained, the control data is calculated on the life-saving device to realize the motion control of the two-degree-of-freedom pan-tilt and the life-saving device. The present invention installs an image acquisition device on an existing marine rescue device, and extracts the position of the drowning person in the image through an image target detection algorithm, which is used as feedback to control the motion of the life-saving device and subsequent rescue actions.

Description

Technical Field

[0001] The invention relates to an autonomous target detection and tracking method for a marine lifesaving device. Background Art

[0002] Domestic and foreign rescue equipment and methods for rescuing people who fall into the sea mainly use some relatively traditional rescue methods, such as launching cables, throwing lifebuoys, inflatable rubber rafts or helicopter rescue. However, these are not suitable for severe sea conditions or when the person who falls into the sea is injured or unconscious. Therefore, self-powered marine search and rescue devices have become more and more common in recent years. These devices can autonomously approach people who fall into the water in harsh environments and can rescue unconscious people who fall into the water. However, a new problem has arisen: how can the search and rescue device determine the exact location of the person at sea before rescuing the person who falls into the water?

[0003] When rescuing people who have fallen into the water at sea, the mother ship or lifeboat is an under-driven large inertia time-delay system, which cannot achieve accurate dynamic positioning. At the same time, to prevent the lifeboat from causing secondary injuries when it approaches the people who have fallen into the water, it is usually necessary to deploy marine life-saving devices to complete the salvage and rescue. Traditional marine life-saving devices include life buoys, life nets, inflatable rubber boats, etc. These life-saving devices need to be deployed by search and rescue personnel near the people who have fallen into the water, and the people who have fallen into the water need to complete the remaining rescue steps by themselves. In recent years, some new powered marine rescue devices have been added. They can autonomously approach the people who have fallen into the water under the remote control of the staff, and can complete the rescue when the people who have fallen into the water are unconscious or injured.

[0004] Although some of the current marine rescue devices have autonomous power systems, they are still manually controlled by the rescue personnel on the mother ship. During the remote control process, the first-person perspective and the device are often inconsistent, so they cannot be controlled freely. At the same time, the device cannot be too far away from the rescue personnel during the remote control process, otherwise the remote control will go wrong due to the rescue personnel's field of vision, and the rescue of the drowned person will be missed, causing delays in rescue. It may also directly hit the drowned person, causing secondary injuries to the person. At the same time, because a marine rescue device requires one or more rescue personnel to operate, the rescue devices deployed are limited, and the drowned person cannot be rescued quickly and in time, delaying the rescue time of the drowned person and causing life safety.

[0005] The task of object detection is to find objects of interest in images or videos, and detect their positions and sizes at the same time. The development of object detection can be divided into two periods: the traditional object detection algorithm period (1998-2014) and the deep learning-based object detection algorithm period (2014-present). The deep learning-based object detection algorithm has developed into two technical routes: single-stage method and two-stage method.

[0006] Image enhancement is to purposefully emphasize the overall or local characteristics of an image, such as improving the color, brightness, and contrast of an image, making an originally unclear image clear or emphasizing certain features of interest, expanding the differences between features of different objects in the image, suppressing features of no interest, and improving the visual effect of the image. In recent years, convolutional neural networks have made great breakthroughs in many low-level computer vision tasks, including image super-resolution, deblurring, defogging, denoising, and image enhancement. Summary of the invention

[0007] In order to solve the problems existing in the above-mentioned prior art, an autonomous target detection method applied to marine rescue equipment is proposed. An image acquisition device is installed on the existing marine rescue equipment, and the position of the person falling into the water in the image is extracted through the image target detection algorithm, which is used as feedback to control the movement of the device and subsequent rescue actions.

[0008] The present invention can be implemented through the following technical solutions:

[0009] An autonomous target detection and tracking method for a marine lifesaving device, wherein an image acquisition device, a wireless image transmission module, and a two-degree-of-freedom pan-tilt are installed on the lifesaving device, wherein the image acquisition device is installed on the two-degree-of-freedom pan-tilt, and the image acquisition device acquires image information, performs image processing on a mother ship, and generates control data, which is transmitted back to the lifesaving device via wireless transmission to complete motion control of the two-degree-of-freedom pan-tilt and the lifesaving device, or an edge computing module is installed, image processing is performed directly on the lifesaving device, and control data is calculated on the lifesaving device after the target position is acquired to realize motion control of the two-degree-of-freedom pan-tilt and the lifesaving device.

[0010] Furthermore, the control method is an automatic control mode, specifically: first, the mother ship throws the life-saving device on the sea surface, and the image acquisition device starts to acquire images. The acquired images are transmitted back to the mother ship through the wireless image transmission module for the mother ship to monitor and record. At the same time, the acquired images will use the edge computing module or the mother ship for image enhancement, and then the enhanced images will be sent to the target detection model to identify whether there are people falling into the water in the image and identify the position and approximate distance of the people falling into the water in the image. If there are no people falling into the water in the image, the cruise mode is entered. If a person falling into the water is detected, the pan-tilt angle will be dynamically adjusted according to the position information of the person falling into the water, so that the image acquired by the image acquisition device can be locked on the person falling into the water. Then, the direction in which the life-saving device should move is calculated according to the pan-tilt angle and control data is given to control the life-saving device to gradually approach the person falling into the water. After determining that the rescue can be carried out at the designated location, the results will be reported to the mother ship, and the personnel on the mother ship will complete the rescue according to the image information transmitted back by the image transmission.

[0011] Furthermore, the image enhancement step includes performing image deraining, specifically, using a low-pass filter to decompose the derained image into a base layer and a detail layer, as shown in formula (4):

[0012] J=J base +J detail (4)

[0013] By using a low-pass filter to process the image, we can obtain a base layer containing the basic information of the image. Then, the original image and the base layer are subtracted to get the detail layer, which mainly includes some rain lines and image detail information. The original image is directly sent to the convolutional neural network model, and the detail layer is used as input. The model output is combined with the base layer to obtain a rain-free image.

[0014] Furthermore, the method of calculating the direction in which the lifesaving device should move according to the pan / tilt angle and providing control data includes a path planning method for dynamically adjusting the direction of movement;

[0015] Assuming the safe rescue distance of the life-saving device is r, the life-saving device should always be outside the distance r of the person who falls into the water. Therefore, a circle is made with the person who falls into the water as the center and r as the radius. A tangent is made from the center point O′ of the life-saving device. The direction of the tangent is the direction of the expected movement speed V1 of the life-saving device.

[0016] From simple trigonometric calculations, it can be concluded that the angle θ3 between the desired motion direction and the gimbal orientation is:

[0017]

[0018] Where, L O′H The distance from the lifesaving device to the person who falls into the water.

[0019] Then the forward direction of the lifesaving device V θ :

[0020] V θ =θ2-θ1-θ3 (6)

[0021] Where θ1 is the actual movement direction of the lifesaving device and the angle with the X axis. θ2 is the angle between the direction of the gimbal and the X axis. Both θ1 and θ2 can be obtained by the attitude sensors installed on the lifesaving device and the gimbal.

[0022] Furthermore, the control method is a manual control mode, specifically: first, the mother ship throws the life-saving device onto the sea surface, and at the same time, the image acquisition device collects image information, and returns the image information to the mother ship through the wireless image transmission module. The search and rescue personnel can see the real-time picture displayed on the mother ship. The search and rescue personnel lock the position of the person who fell into the water based on the information transmitted back by the image, and manually control the life-saving device to move gradually closer to the person who fell into the water to complete the rescue.

[0023] The present invention has the following beneficial effects:

[0024] (1) By installing an image acquisition device and a pan / tilt head on the life-saving device, the shortcomings of traditional life-saving devices, such as limited search area and missing people who fall into the water due to limited field of view, are solved.

[0025] (2) By performing enhancement operations such as deraining and defogging on the collected images, the image quality is improved and the shortcomings of failing to successfully identify the target due to problems such as image deformation and blur are solved.

[0026] (3) By detecting targets and identifying whether there are people falling into the water in the image, multiple life-saving devices can be detected at the same time, solving the problem of low search efficiency due to insufficient personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A difference diagram for edge computing module use in the present invention;

[0028] Figure 2 This is a flowchart for implementing automatic control in the present invention;

[0029] Figure 3 This is a flowchart for implementing manual control in the present invention;

[0030] Figure 4 is the motion planning diagram in the present invention;

[0031] Figure 5-12 This is a diagram of the detection results of a person falling into water in the present invention. DETAILED DESCRIPTION

[0032] The following describes the implementation of the present invention through specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification.

[0033] The autonomous target detection and tracking method of a marine life-saving device of the present invention is mainly used in a life-saving device with a self-powered system. There is no need to modify the structure of the existing life-saving device. It is only necessary to install the additional device required by the method on the device. The additional device mainly includes: an image acquisition device, a wireless image transmission module, and a two-degree-of-freedom pan-tilt. The image acquisition device is installed on the two-degree-of-freedom pan-tilt. The optional equipment is an edge computing module, such as Figure 1 As shown. If the edge computing module is installed, image processing is performed directly on the life-saving device. After obtaining the target position, the control data is calculated on the life-saving device to directly control the pan-tilt head and the life-saving device. If the edge computing module is not installed, image processing will be performed on the mother ship, and the control data generated by the mother ship will be transmitted back to the life-saving device through wireless transmission to complete the control of the pan-tilt head and the life-saving device. After the installation is completed, the present invention can realize both automatic control mode and manual control mode.

[0034] In this embodiment, the automatic control mode, image acquisition, image processing and subsequent control decisions are all completed on the mother ship or the acquisition device, that is, the calculation is completed at the terminal. The specific implementation process is that the mother ship first throws the life-saving device on the sea surface, and the image acquisition device starts to acquire images. The acquired images are transmitted back to the mother ship through the wireless image transmission module for the mother ship to monitor and record. At the same time, the acquired images will be enhanced using the edge computing module or the mother ship. This step is because the volume of the life-saving device is generally small. The splashes and splashing water droplets caused by sea breeze and severe sea waves may affect the image quality and affect the subsequent target detection. Therefore, operations such as rain removal and rain fog will be performed in the image enhancement step. The enhanced image is then sent to the target detection model to identify whether there are people falling into the water in the image and identify the position and approximate distance of the people falling into the water in the image. If there are no people falling into the water in the image, the cruise mode is entered. If a person falling into the water is detected, the pan-tilt angle will be dynamically adjusted according to the position information of the person falling into the water, so that the image acquired by the image acquisition device can always be locked on the person falling into the water. Then, the direction in which the life-saving device should move is calculated based on the PTZ angle and the control data is given to control the life-saving device to gradually approach the person who falls into the water. After it is determined that the rescue can be carried out at the designated location, the results are reported to the mother ship. The personnel on the mother ship will complete the rescue based on the image information sent back by the image transmission. Figure 2 shown.

[0035] The specific implementation process of the manual control mode in this embodiment is as follows: first, the mother ship throws the life-saving device on the sea surface, and at the same time, the image acquisition device collects image information, and returns the image information to the mother ship through the wireless image transmission module, so that the search and rescue personnel can see the real-time picture displayed on the mother ship. Since the image is transmitted back by the image transmission module, the exploration distance of the search and rescue device is expanded from the ordinary naked eye range to the range within which the image module can transmit the image, and because the image acquisition device is placed on the pan-tilt, when the sea waves are severe and it is inconvenient to turn, the image acquisition device can collect 360-degree image information around the life-saving device by adjusting the pan-tilt angle, which can ensure that the person who falls into the water will not be missed and improve the search and rescue efficiency of the life-saving device. The search and rescue personnel lock the position of the person who falls into the water according to the information transmitted back by the image, and manually remotely control the life-saving device to gradually move closer to the person who falls into the water to complete the rescue.

[0036] In the image enhancement step of this embodiment, operations such as rain removal and rain fog removal are performed. Before target detection, image enhancement is an extremely important step. In the environment of rescuing people who fall into the water, there is often rain and splashing water, which will hinder the subsequent detection of people who fall into the water. Therefore, it is essential to remove rain from the image in advance.

[0037] Usually, a rainy image can be represented as a linear combination of a clean rainless image and a rain streak layer. In existing image deraining methods, due to the overlap of rain streaks and background information in the feature space, the background information may be mistakenly considered as rain streak information when removing rain from an image, resulting in incomplete rain streak removal or excessive rain removal, resulting in residual rain streaks and loss of background information. This paper uses a low-pass filter to decompose the derained image into a base layer and a detail layer, as shown in formula (7):

[0038] J=J base +J detail (7)

[0039] By using a low-pass filter to process the image, we can get the base layer containing the basic information of the image, and then subtract the original image from the base layer to get the detail layer. The detail layer mainly includes some rain lines and image details. base And go to the base after the rain J base Relationship, O base ≈J base The base layers before and after rain removal are basically the same. Therefore, this paper directly feeds the original original image into the convolutional neural network model, changes the detail layer as input, and combines the input with the base layer to obtain the rain removal image.

[0040] After the image is decomposed into base layer and detail layer, the loss function is:

[0041]

[0042] Since the rain line information of the image is a low-dimensional feature, a too deep network is not required. The final network structure is a two-layer convolutional neural network plus an activation function, with a deconvolution layer at the end.

[0043] If the deraining detail layer and the base layer output by the convolutional network are directly combined, the resulting image will be slightly blurred. Therefore, it can be seen from the figure that the base layer and the deraining detail layer are enhanced before being combined. The base layer will be enhanced using a nonlinear function, and the deraining detail layer will be directly linearly enlarged, as shown in formula (9):

[0044] O enhanced =(I base ) enhanced +2 fW (I detail ) (9)

[0045] When the life-saving device is going to rescue the person who falls into the water, because the life-saving device at sea is an under-driven system, it is impossible to adjust the speed and forward direction of the device in time. If the forward speed of the life-saving device is towards the person who falls into the water, it is easy to cause a collision with the person who falls into the water. Therefore, a rescue path planning is needed to safely reach the safe rescue distance of the person who falls into the water.

[0046] A path planning method for dynamically adjusting the forward direction is proposed in this embodiment, such as Figure 4 As shown;

[0047] Assuming the safe rescue distance of the life-saving device is r, the life-saving device should always be at a distance r from the person who falls into the water. Draw a circle with the person who falls into the water as the center and r as the radius. Draw a tangent from the center point O' of the life-saving device, and the direction of the tangent is the direction of the expected movement speed V1 of the life-saving device.

[0048] From simple trigonometric calculations, it can be concluded that the angle θ3 between the desired motion direction and the gimbal orientation is:

[0049]

[0050] Where, L O′H It is the distance from the life-saving device to the person who falls into the water.

[0051] Then the forward direction of the lifesaving device V θ :

[0052] V θ =θ2-θ1-θ3 (11)

[0053] Where θ1 is the actual movement direction of the lifesaving device and the angle with the X axis. θ2 is the angle between the direction of the gimbal and the X axis. Both θ1 and θ2 can be obtained by the attitude sensors installed on the lifesaving device and the gimbal.

[0054] As the relative positions of the lifesaving device and the person falling into the water change, the forward direction of the lifesaving device is also dynamically adjusted.

[0055] The movement speed of the life-saving device should be dynamically adjusted according to the distance from the person who falls into the water. When approaching the person who falls into the water, the speed needs to be gradually reduced. However, the text does not use the distance between the life-saving device and the person who falls into the water as an indicator. Instead, the tangent length of the circular area between the life-saving device and the person who falls into the water is used as the judgment indicator.

[0056] This example uses an object detection algorithm to detect people falling into the water. This example takes the YOLOv5-based person detection algorithm as an example.

[0057] In traditional algorithms, non-maximum suppression is usually used to screen the many prediction boxes output by the model. This example introduces prior information. Since people who fall into the water usually occupy a smaller area in the image, the candidate boxes with larger areas are removed before non-maximum suppression is performed to screen the candidate boxes. At the same time, in traditional algorithms, three-scale feature information is used to predict objects with large differences in area in the picture. This example uses prior information to modify the prediction to use two scales to improve the running speed of the algorithm. At the same time, GIOU is used to introduce the minimum enclosing rectangle of the prediction box and the true box, which improves the problem that the distance between the true box and the prediction box cannot be predicted when they do not intersect.

[0058] This experiment is based on the pytorch1.10.0 framework and runs on the ubuntu21.04 system. The model training is accelerated by GPU and is calculated on the mobile version of NVIDIA3060 graphics card, CUDA11.2 environment, and the video memory is 6g. In order to improve the real-time detection speed, this experiment uses the YOLOv5s model for detection.

[0059] Since there is no public dataset of people falling into the water at sea, we use the public object detection dataset COCO for training and test it by downloading public images of people falling into the water from the Internet. The object detection results are as follows: Figure 5-12 As shown:

[0060] In the detection result diagram of the person falling into the water, Figure 5 , 7, 9, 11 are the original images to be detected, Figure 6 , 8, 10, 12 are the output images after detection. Figure 5 and 6 It can be seen that when the lower body of the person who fell into the water is below the water surface and the upper body appears in the image, or only the head is above the water surface, the location of the person who fell into the water can be detected well. Figure 7 and 8 For long-distance detection, when the number of pixels occupied by the person falling into the water in the image is relatively small, the model can still accurately detect and distinguish between the two people falling into the water. Fig. 9 Figures 10 and 10 show that when the person who falls into the water is pitching on the water surface in an abnormal posture, the model can still detect his or her position. Fig.11 and 12 This shows that when the person who falls into the water is blocked by obstacles and only a partial area is exposed, the algorithm can still detect the person who falls into the water.

[0061] The above results prove that the detection algorithm can well detect the location of the drowning person when a maritime accident occurs, regardless of whether the drowning person consciously makes a call for help or is blocked by the wreckage of the ship, providing support for subsequent tracking and rescue work.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An autonomous target detection and tracking method for a marine lifesaving device, characterized in that: An image acquisition device, a wireless image transmission module, and a two-degree-of-freedom pan-tilt are installed on the lifesaving device. The image acquisition device is installed on the two-degree-of-freedom pan-tilt. The image acquisition device acquires image information, and the mother ship performs image processing and generates control data, which is transmitted back to the lifesaving device through wireless transmission to complete the motion control of the two-degree-of-freedom pan-tilt and the lifesaving device. Alternatively, by installing an edge computing module, image processing is performed directly on the lifesaving device, and after obtaining the target position, control data is calculated on the lifesaving device to realize the motion control of the two-degree-of-freedom pan-tilt and the lifesaving device. The control method is an automatic control mode, specifically: first, the mother ship throws the lifesaving device on the sea surface, and the image acquisition device starts to acquire images. The acquired images are transmitted back to the mother ship through the wireless image transmission module for monitoring and recording by the mother ship. At the same time, the acquired images will be enhanced using the edge computing module or the mother ship, and then the enhanced images will be sent to the target detection model to identify whether there are people falling into the water in the image and identify the position and approximate distance of the people falling into the water in the image. If there are no people falling into the water in the image, the cruise mode is entered. If a person falling into the water is detected, the pan-tilt angle will be dynamically adjusted according to the position information of the person falling into the water, so that the image acquired by the image acquisition device can be locked on the person falling into the water. Then, the direction in which the lifesaving device should move is calculated according to the pan-tilt angle and control data is given to control the lifesaving device to gradually approach the person falling into the water. After determining that the rescue can be carried out at the designated location, the results will be reported to the mother ship, and the personnel on the mother ship will complete the rescue according to the image transmitted back by the image transmission; The path planning method of calculating the direction of the lifesaving device to be advanced according to the pan / tilt angle and providing control data includes dynamically adjusting the direction of advancement; Assuming the safe rescue distance of the lifesaving device is r, the lifesaving device should always be at a distance r from the person who falls into the water. Draw a circle with the person who falls into the water as the center and r as the radius. Draw a tangent from the center point O′ of the lifesaving device. The direction of the tangent is the direction of the expected movement speed V1 of the lifesaving device. From simple trigonometric calculations, it can be concluded that the angle θ3 between the desired motion direction and the gimbal orientation is: Where, L O′H The distance from the lifesaving device to the person who falls into the water. Then the forward direction of the lifesaving device V θ : V θ =θ2-θ1-θ3 (3) Where θ1 is the actual movement direction of the lifesaving device and the angle with the X-axis, θ2 is the angle between the direction of the gimbal and the X-axis, and both θ1 and θ2 can be obtained by the attitude sensors installed on the lifesaving device and the gimbal.

2. The autonomous target detection and tracking method for marine lifesaving equipment according to claim 1, characterized in that: The image enhancement step includes performing image deraining, specifically, using a low-pass filter to decompose the derained image into a base layer and a detail layer, as shown in formula (1): J=J base +J detail (1) By processing the image with a low-pass filter, we can obtain a base layer containing the basic information of the image. Then, the original image and the base layer are subtracted to obtain the detail layer, which mainly includes some rain lines and image detail information. The original image is directly sent to the convolutional neural network model, and the detail layer is used as input instead. The model output is combined with the base layer to obtain a rain-free image.

3. The autonomous target detection and tracking method for marine lifesaving equipment according to claim 1, characterized in that: The control method is a manual control mode, specifically: first, the mother ship throws the life-saving device onto the sea surface, and at the same time, the image acquisition device acquires image information, and returns the image information to the mother ship through the wireless image transmission module. The search and rescue personnel can see the real-time picture displayed on the mother ship. The search and rescue personnel lock the position of the person who falls into the water based on the information transmitted back by the image, and manually control the life-saving device to move gradually closer to the person who falls into the water to complete the rescue.

Images