Real-time Extraction System and Method for Coastal Defense Monitoring Targets Based on Edge Computing

Abstract

The present invention discloses a real-time extraction system and method for coastal defense monitoring targets based on edge computing, belonging to the field of coastal defense monitoring. It includes: a coastal defense monitoring device for generating video images in real time, an edge computing unit connected to the coastal defense monitoring device for real-time analysis and storage at the edge side, and a backend server communicating with the edge computing unit for receiving edge-side data and transmitting superior instructions; the edge computing unit at least includes a target detection and tracking module for performing target detection and multi-target tracking based on video images; the target detection and tracking module performs target detection in the following manner: using a pre-trained target detection model to output prediction boxes marked with ships; performing post-processing on the prediction boxes to screen and obtain target prediction boxes. This solution can achieve high-precision extraction and short-term storage of targets at the edge side, reduce the latency and risk of system operation, improve the unattended ability of the system, and avoid the occurrence of control vacuum periods.

Description

Technical Field

[0001] The present invention relates to the technical field of coastal defense monitoring, and particularly to a real-time extraction system and method for coastal defense monitoring targets based on edge computing. Background Art

[0002] Traditional coastal defense monitoring systems usually rely on site industrial control computers or central servers for image data processing. This centralized processing method is restricted by network bandwidth and transmission link length, and it is prone to data transmission delays. Second, once the network connection between the front-end coastal defense monitoring and the back-end is accidentally disconnected, then during the network outage period, the front-end coastal defense monitoring loses its control ability, resulting in the emergence of a control vacuum period. In addition, when traditional object detection models perform post-processing, they only use non-maximum suppression (NMS) to remove redundant detected bounding boxes, but there will be phenomena such as repeatedly detecting multiple different-sized and overlapping object bounding boxes for the same ship target, or repeatedly marking the bow part. In the actual application process, these above-mentioned phenomena will all lead to incorrect ship target positioning and tracking, thereby affecting the accuracy of ship target detection.

[0003] Therefore, there is an urgent need for a coastal defense monitoring system that can achieve high-precision extraction and short-term storage of targets at the edge side to reduce the delay and risk of system operation and improve the unattended ability of the system. Summary of the Invention

[0004] In order to solve the problem that the traditional coastal defense monitoring system cannot timely transmit the monitoring video to the back-end for high-precision target detection and tracking during the network outage period, resulting in the emergence of a control vacuum period, the embodiments of the present invention provide a real-time extraction system and method for coastal defense monitoring targets based on edge computing.

[0005] On the one hand, a real-time extraction system for coastal defense monitoring targets based on edge computing is provided. The system includes:

[0006] A coastal defense monitoring device for generating video images in real time, an edge computing unit connected to the coastal defense monitoring device for real-time analysis and storage at the edge side, and a back-end server communicating with the edge computing unit for receiving edge-side data and transmitting superior instructions;

[0007] The edge computing unit at least includes a target detection and tracking module, and the target detection and tracking module is used for target detection and multi-target tracking based on the video image;

[0008] The target detection and tracking module performs target detection in the following manner:

[0009] Using a pre-trained target detection model to output prediction bounding boxes marked with ships;

[0010] Perform post - processing on the predicted bounding boxes to filter out the target predicted bounding boxes.

[0011] On the other hand, a method for real - time extraction of coastal defense monitoring targets based on any system embodiment of the specification is provided, including:

[0012] Set an edge computing unit connected to it on the coastal defense monitoring device side, and use the edge computing unit to obtain the video images captured in real - time by the coastal defense monitoring device; wherein, a target detection and tracking module is set in the edge computing unit, and a pre - trained target detection model is included in the target detection and tracking module;

[0013] Use the target detection model to perform target detection on the video images. After outputting the predicted bounding boxes marked with ships, perform post - processing on the predicted bounding boxes to filter out the target predicted bounding boxes;

[0014] Use the target detection and tracking module to perform multi - target tracking;

[0015] The edge computing unit sends the stored end - side data to the backend server.

[0016] The technical solutions provided by the present invention can at least bring the following beneficial effects:

[0017] By combining the edge computing unit with the coastal defense monitoring device, the traditional coastal defense equipment is endowed with independent control capabilities, realizing target extraction and short - term storage at the end - side, reducing system latency and risks, and strengthening the unattended operation ability of the coastal defense monitoring system; in addition, by performing post - processing on the predicted bounding boxes output by the target detection model, the accuracy and robustness of ship target detection are improved. Therefore, this solution can achieve high - precision target extraction and short - term storage at the end - side to reduce system operation latency and risks, improve the unattended operation ability of the system, and avoid the occurrence of control vacuum periods. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 It is a schematic diagram of a real - time extraction system for coastal defense monitoring targets based on edge computing provided by an embodiment of the present invention;

[0020] Figure 2 It is a flowchart of a method for real - time extraction of coastal defense monitoring targets based on edge computing provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0022] The following describes the specific implementation manners of the above concept.

[0023] Please refer to Figure 1 , a real-time extraction system for coastal defense monitoring targets based on edge computing provided by an embodiment of the present invention, the system includes: a coastal defense monitoring device for generating video images in real time, an edge computing unit connected to the coastal defense monitoring device for real-time analysis and storage at the edge side, and a back-end server communicating with the edge computing unit for receiving data at the edge side and transmitting superior instructions;

[0024] The edge computing unit at least includes a target detection and tracking module, and the target detection and tracking module is used for target detection and multi-target tracking based on video images;

[0025] The target detection and tracking module performs target detection in the following manner:

[0026] Using a pre-trained target detection model to output a prediction box marked with a ship;

[0027] Performing post-processing on the prediction box to screen and obtain a target prediction box.

[0028] In the embodiments of the present invention, by combining the edge computing unit with the coastal defense monitoring device, the traditional coastal defense equipment is given independent control capabilities, realizing target extraction and short-term storage at the edge side, reducing system latency and risks, and strengthening the unattended capabilities of the coastal defense monitoring system; in addition, by performing post-processing on the prediction boxes output by the target detection model, the accuracy and robustness of ship target detection are improved. Therefore, this solution can achieve high-precision extraction and short-term storage of targets at the edge side to reduce system operation latency and risks, improve the unattended capabilities of the system, and avoid the appearance of control vacuum periods.

[0029] In some embodiments, the edge computing unit further includes: a video acquisition module and an image processing module;

[0030] The video acquisition module accesses the coastal defense monitoring device through a hardware interface and is used for acquiring real-time video images;

[0031] The image processing module is used for preprocessing the video images and then sending them to the edge computing unit.

[0032] In this embodiment, in order to ensure the delay, the video acquisition module accesses the original video image of the coastal defense monitoring device through the SDI interface.

[0033] The image processing module is mainly used to convert the original image data of the video frame for subsequent use by the target detection and tracking module.

[0034] Specific preprocessing steps include:

[0035] Use a video processing engine with edge computing to convert the original YUV format image to RGB format;

[0036] Scale the size of the image according to the required input size of the target detection model. For example, when the original image size is 1920*1080, use bilinear interpolation to scale the original image to the input size of the target detection model, which is 640*640;

[0037] Normalize the values of each channel of the image, that is, divide by 256 numerically.

[0038] When scaling the image, scale the image proportionally under the condition of keeping the aspect ratio of the original image unchanged, so that the resolution of the longest side is the same as the input size of the target detection model, and the missing part of the short side is filled with 0;

[0039] When the coastal defense monitoring acquires video images under rainy and foggy weather conditions, first use the dark channel prior algorithm to de-fog the image, and then perform image scaling and normalization.

[0040] In some embodiments, when the target detection and tracking module executes to output a prediction box marked with a ship using a pre-trained target detection model, it includes:

[0041] Input the video image into a pre-trained YOLOv7 target detection model, and obtain feature maps at three different scales of the video image through the extraction of the backbone network;

[0042] Fuse the feature maps at three different scales through the FPN network to obtain three different sizes of prediction results;

[0043] Send the prediction results into the prediction head structure, and through reparameterized convolution and non-maximum suppression algorithms, output a prediction box marked with a ship.

[0044] In this embodiment, the edge computing unit on the terminal side uses the YOLOv7 object detection model for object detection to improve the detection accuracy. However, if only the non-maximum suppression algorithm (NMS) is used to remove the redundant detected bounding boxes, there will be phenomena such as repeatedly detecting multiple bounding boxes of different sizes and overlapping each other for the same ship object, or repeatedly marking the bow part. In the actual application process, these above-mentioned phenomena will all lead to incorrect ship object positioning and tracking, thereby affecting the ship object detection accuracy.

[0045] Therefore, in some embodiments, when the object detection and tracking module performs post-processing on the prediction bounding boxes and filters to obtain the target prediction bounding boxes, it includes:

[0046] Filter the prediction bounding boxes according to the size and overlapping situation of the prediction bounding boxes to obtain intermediate prediction bounding boxes;

[0047] Filter the intermediate prediction bounding boxes according to the relative position relationship of the bow in the ship to obtain the target prediction bounding boxes.

[0048] In this embodiment, based on the size and overlapping situation of the prediction bounding boxes, the existing prediction bounding boxes are filtered respectively to remove the redundant prediction bounding boxes and the prediction bounding boxes with a lot of overlapping similarity. Then, based on the relative position relationship between the ship and its bow, the possible bow prediction bounding boxes are further filtered to ensure that the prediction bounding boxes are not located at the four corners of the ship, thus effectively solving the common problems of object overlap and repeated detection in ship object detection and improving the accuracy of the detection results and the robustness in complex overlapping scenarios.

[0049] In some embodiments, filtering the prediction bounding boxes according to the size and overlapping situation of the prediction bounding boxes to obtain intermediate prediction bounding boxes includes:

[0050] Pair the prediction bounding boxes in pairs to obtain matching pairs, and calculate the maximum intersection area of the matching pairs;

[0051] Determine the matching pairs corresponding to the maximum intersection area greater than the first preset threshold as overlapping matching pairs;

[0052] For each overlapping matching pair, perform calculating the similarity score value of the overlapping matching pair according to the size and overlapping situation of the two prediction bounding boxes of the overlapping matching pair;

[0053] When the similarity score value is less than the preset score threshold, regard the prediction bounding boxes in the overlapping matching pair as intermediate prediction bounding boxes, and regard the overlapping matching pair as a filtered matching pair;

[0054] When the similarity score value is greater than or equal to the preset score threshold, regard the prediction bounding box with a smaller maximum intersection area in the overlapping matching pair as the intermediate prediction bounding box.

[0055] It should be noted that for the prediction bounding boxes includeda and the prediction box b of the matching pair, the prediction box a intersection area IOA a = ([[]] S a ∩ S b ) / S a ; the prediction box b intersection area IOA b = ([[]] S a ∩ S b ) / S b ; where S a , S b are respectively the areas of the prediction box a and the prediction box b , S a ∩ S b is the area of the intersection part of the prediction box a and the prediction box b . The maximum intersection area of the matching pair = max( IOA a , IOA b ); for example, if IOA a > IOA b , then the maximum intersection area of the matching pair is IOA a .

[0056] Specifically, for the prediction boxes output by the target prediction model, the maximum intersection area ( IOA ) between the matching pairs is calculated by traversing in a pairwise pairing manner. If the maximum IOA in a pair of matching pairs is greater than the first preset threshold (i.e., the preset maximum intersection area threshold), then this pair of matching pairs is regarded as a suspicious overlapping matching pair, and then a list of suspicious overlapping matching pairs is constructed. Then, the ship prediction boxes with high overlapping similarity are further screened by the preset scoring threshold to obtain the remaining screened matching pairs and intermediate prediction boxes. For example, in the actual execution process, setting the first preset threshold to 0.85 can effectively screen out the prediction boxes with a high overlapping area.

[0057] It should be noted that for the matching pairs corresponding to the maximum intersection area not greater than the first preset threshold, they are directly retained, that is, determined as intermediate prediction boxes.

[0058] Specifically, if the similarity score value of an overlapping match pair is less than the preset score threshold, then the two first prediction boxes within the overlapping match pair are retained; otherwise, the prediction box with a larger IOA in the overlapping match pair is deleted, and the overlapping match pair is removed from the list of suspicious overlapping match pairs. The retained overlapping match pairs are the filtered match pairs, and the retained prediction boxes are the intermediate prediction boxes. For example, in the actual execution process, setting the preset score threshold to 0.7 can more accurately filter out the duplicate prediction boxes of the same ship target.

[0059] In some embodiments, the similarity score value is calculated in the following manner:

[0060] Determine the aspect ratio based on the sizes of the two prediction boxes of the overlapping match pair;

[0061] Calculate the ratio of the minimum aspect ratio to the maximum aspect ratio of the two prediction boxes of the overlapping match pair, and take the product of this ratio and the maximum intersection area of the two prediction boxes of the overlapping match pair as the similarity score value of the overlapping match pair.

[0062] Specifically, for each overlapping match pair in the list of suspicious overlapping match pairs, use the following formula to calculate the similarity score value of the overlapping match pair:

[0063]

[0064] Wherein, sim ab is the similarity score of the overlapping match pair including the prediction box a and the prediction box b ; IOA a , IOA b are respectively the maximum intersection areas of the prediction box a , the prediction box b in the overlapping match pair; γ a , γ b are respectively the aspect ratios of the prediction box a and the prediction box b ; is γ a and γ b the minimum aspect ratio among; is γ a and γ b the maximum aspect ratio among.

[0065] Since the object detection model for ships usually includes several detection heads for detecting features at different scales to detect objects of multiple scales. Therefore, for the same ship, due to the different observation scales of different detection heads, different detection results of different sizes are output for the ship. However, there is only one ship object, so it is necessary to filter out the redundant detection frames. In the present invention, by considering the aspect ratio to determine the shape similarity of two prediction frames, it not only avoids the interference of the prediction frame scale change on the similarity evaluation, but also can more accurately eliminate the overlapping frames with similar aspect ratios and retain the overlapping frames with dissimilar shapes; at the same time, by combining the maximum intersection area, the adaptive recognition of ship objects is improved.

[0066] In some embodiments, according to the relative position relationship of the bow in the ship, the intermediate prediction frames are screened to obtain the target prediction frames, including:

[0067] Define the intermediate prediction frame with a larger size in the screening matching pair as the first prediction frame, and the intermediate prediction frame with a smaller size as the second prediction frame;

[0068] For each screening matching pair, the following operations are performed:

[0069] Taking each vertex of the first prediction frame as an anchor point, construct virtual frames with the same size as the second prediction frame inside the first prediction frame; where the number of virtual frames is the same as the number of anchor points;

[0070] Calculate the intersection over union (IoU) between the second prediction frame and the virtual frames;

[0071] When the IoU is greater than or equal to the second preset threshold, delete the second prediction frame to obtain the target prediction frame.

[0072] In this embodiment, each screening matching pair includes a first prediction frame A with a larger size and a second prediction frame B with a smaller size (i.e., the size of A is greater than that of B). For each screening matching pair, let the four vertices of A be ( x A1 , y A1 ), ( x A2 , y A2 ), ( x A3 , y A3 ), ( x A4 , y A4 ), the width of A is w B , and the height of A is h B. Taking the four vertices of A as the anchor points respectively, construct a virtual box with the same width and height as B inside A. Taking the point ( x A1 , y A1 ) as an example, the coordinates of the four vertices of the constructed virtual box are respectively ( x A1 , y A1 ), ( x A1 , y A1 + h B ), ( x A1 + w B , y A1 + h B ), ( x A1 + w B , y A1 ); Taking the point ( x A2 , y A2 ) as an example, the coordinates of the four vertices of the constructed virtual box are respectively ( x A2 , y A2 - h B ), ( x A2 , y A2 ), ( x A2 + w B , y A2 ), ( x A2 + w B , y A2 - h B ); Taking the point ( x A3 , y A3 ) as an example, the coordinates of the four vertices of the constructed virtual box are respectively ( x A3 - w B ,y A3 - h B ), ([[]] x A3 - w B , y A3 ), ([[]] x A3 , y A3 ), ([[]] x A3 , y A3 - h B ); Taking the point ([[]] x A4 , y A4 ) as an example, the coordinates of the four vertices of the constructed virtual box are respectively ([[]] x A4 - w B , y A4 ), ([[]] x A4 - w B , y A4 + h B ), ([[]] x A4 , y A4 + h B ), ([[]] x A4 , y A4 ). After the four virtual boxes are constructed, use these virtual boxes to calculate the intersection over union (IoU) with B. If the calculated IoU values are all less than the second preset threshold (i.e., the preset bow IoU threshold), then retain B; otherwise, delete B. In this way, the finally retained intermediate prediction boxes are the final target prediction boxes.

[0073] In the present invention, if the bow part of a certain ship is repeatedly detected, then the prediction box of this bow will surely be located at the corner position of the prediction box of this ship. Based on this relative position relationship between the bow and the ship target, for the intermediate prediction boxes retained after the screening, continue to filter the prediction boxes with a larger Intersection over Area (IOA) in the screened matching pairs, and remove the smaller prediction boxes that may be the bow. Finally, obtain the target prediction box for the ship target, thus solving the problems of overlapping target boxes and repeated detection of the bow position in the existing detection methods, and effectively improving the accuracy and robustness of ship target detection.

[0074] The method of the present invention improves the post-processing part of the traditional target detection algorithm. Starting from the problem of overlapping prediction boxes encountered in the actual detection task, further filter the prediction boxes using the aspect ratio relationship and IOA, and through multiple screenings and introducing relevant knowledge of the inherent characteristics of ship targets, greatly alleviate the problems of overlapping target boxes and repeated detection of the bow position in the detection results, making the ship target detection results more accurate and providing more reliable technical support for the intelligent coastal defense system.

[0075] In some embodiments, the target detection and tracking module performs multi-target tracking in the following manner:

[0076] For each target detection result, the following operations are performed:

[0077] Based on the ship type marked in the current target prediction box, divide the current target. When there is no Kalman tracker under the current ship type, it is determined that the current target appears for the first time, initialize a corresponding potential Kalman tracker for it, and retain the first appearance time, ship type, and target prediction box of this ship target; where the ship target and the Kalman tracker are in a one-to-one correspondence relationship;

[0078] When there is a Kalman tracker under the current ship type, perform the Intersection over Union (IoU) operation between the current target prediction box and the target box predicted by each Kalman tracker under the current ship type respectively, and construct a cost matrix using the IoU;

[0079] Use the Hungarian algorithm to find the optimal match of the cost matrix. If the match is successful, update the target box predicted by this Kalman tracker based on the current target prediction box;

[0080] If the match is not successful, it is determined that the current target appears for the first time, initialize a corresponding potential Kalman tracker for it, and retain the first appearance time, ship type, and target prediction box of this ship target;

[0081] When the potential Kalman tracker successfully matches three times in a row, it is considered that the ship target really exists, and a number is assigned to the Kalman tracker based on the order of the same type of targets appearing within the day.

[0082] For each numbered Kalman tracker, the target box predicted and updated in real time by the Kalman filter is used as the position of the target for tracking.

[0083] If the potential Kalman tracker fails to match successfully five times in a row, the Kalman tracker is deleted, and it is considered that the ship target has disappeared.

[0084] In this embodiment, by using the target detection results output by the target detection model to update and correct the prediction results of the Kalman tracker, it can be understood that the detection frequency of the target detection model is lower than the prediction frequency of the Kalman tracker. By judging whether the potential Kalman tracker successfully matches three times in a row, it is judged whether the ship target in the Kalman tracker really exists, and a number is assigned to the Kalman tracker containing the real ship target accordingly. For each numbered Kalman tracker, the target box predicted and updated in real time by the Kalman filter is used as the position of the target for tracking. When the tracked target box crosses the line or enters the alarm area, an alarm signal is generated and a screenshot is saved.

[0085] In some embodiments, the edge computing unit further includes: a target storage and reporting module;

[0086] The target storage and reporting module is used to store the target detection results and target tracking results based on a pre-set storage policy, and report the results based on the communication quality between the edge computing unit and the backend server.

[0087] In this embodiment, the target storage and reporting module mainly includes two processes, the target storage process and the target reporting process. Among them, the target storage process includes:

[0088] Once a numbered Kalman tracker for a target is generated, the original frame image corresponding to the reporting time is saved locally, and the specific information of the target, namely the appearance date, category, serial number within the category, initial reporting time, device status, target status, target box position and the storage path of the original image, is stored in the edge-side database.

[0089] For each numbered target, the following operations are performed:

[0090] The specific information of the target is stored once every 5s, and the corresponding original frame image is saved.

[0091] If the target triggers a line-crossing or intrusion alarm, the target position, alarm status and alarm setting information are stored once, and the image of its alarm-triggering behavior is saved.

[0092] If the target stops triggering the alarm, store the target position once and update the alarm status;

[0093] When it disappears, store the target information including the disappearance time of the target and the last appearance position once.

[0094] In addition, to ensure the normal operation of the edge computing unit, periodically clean the target information and images whose recording time exceeds the specified time limit. For each piece of target information existing in the edge-side database, a reporting flag is used to indicate the transmission status of the information.

[0095] Specifically, the target reporting process includes:

[0096] When the communication with the backend network is smooth, once there is a piece of unreported target information in the edge-side database, use the Http protocol to send the information and the corresponding image to the backend receiving server, and mark the information as reported;

[0097] When the communication with the backend network is not smooth, first store the target information locally, and then report the target information and the corresponding image in the order of reporting time after the network becomes smooth.

[0098] It should be noted that in order to reduce the pressure on the transmission link, the original image corresponding to the target information can be encoded into the Base64 format first and then sent to the backend receiving server.

[0099] In the embodiment of the present invention, the backend server includes a data receiving module and an instruction transmission module;

[0100] The data receiving module receives the target data and images transmitted back from the front end through the Http protocol, stores the target data in the backend database, decodes the image data and stores it in the backend image storage device, and at the same time cleans all data whose existence time exceeds the retention period.

[0101] The instruction transmission module uses the Http protocol to send the target categories to be controlled, the types of alarm areas and alarm lines, the endpoint coordinates, and the duration to the edge computing unit, so as to realize the remote control of the edge device.

[0102] Please refer to Figure 2 , the embodiment of the present invention provides a real-time extraction method for coastal defense monitoring targets based on any system embodiment of the specification. The method includes:

[0103] 200. Set an edge computing unit connected to it at the coastal defense monitoring device side, and use the edge computing unit to obtain the video images captured in real time by the coastal defense monitoring device; wherein, a target detection and tracking module is set in the edge computing unit, and a pre-trained target detection model is included in the target detection and tracking module;

[0104] 202. After performing object detection on the video image using the object detection model and outputting the prediction bounding boxes marked with ships, post - process the prediction bounding boxes to filter out the target prediction bounding boxes.

[0105] 204. Use the object detection and tracking module for multi - object tracking.

[0106] 206. The edge computing unit sends the stored edge - side data to the backend server.

[0107] Regarding the content of the above - mentioned method, since it is based on the same concept as the system embodiment of the present invention, for the specific content, reference can be made to the description in the system embodiment of the present invention, and details will not be repeated here.

[0108] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non - exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device.

[0109] Those of ordinary skill in the art can understand that all or part of the steps to implement the above - mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer - readable storage medium. When the program is executed, it performs the steps including the above - mentioned method embodiments; and the aforementioned storage medium includes various media such as ROM, RAM, magnetic disk or optical disc that can store program codes.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time extraction system for coastal defense monitoring targets based on edge computing, characterized in that, include: A coastal defense monitoring device for generating video images in real time, an edge computing unit connected to the coastal defense monitoring device for performing real-time analysis and storage on the end side, and a backend server communicating with the edge computing unit for receiving end-side data and transmitting superior instructions; The edge computing unit includes at least a target detection and tracking module, and the target detection and tracking module is used to perform target detection and multi-target tracking based on the video image; The target detection and tracking module performs target detection in the following manner: Use the pre-trained object detection model to output the predicted box labeled with the ship; Post-processing the prediction frame to obtain a target prediction frame by screening; When the target detection and tracking module performs post-processing on the prediction frame to screen and obtain the target prediction frame, it includes: According to the size and overlap of the prediction boxes, the prediction boxes are screened to obtain an intermediate prediction box; According to the relative position relationship of the bow of the ship, the intermediate prediction frame is screened to obtain a target prediction frame; The step of screening the prediction boxes according to the size and overlap of the prediction boxes to obtain the intermediate prediction boxes includes: Pair the prediction frames in pairs to obtain matching pairs, and calculate the maximum intersection area of the matching pairs; Determine the matching pairs corresponding to the maximum intersection areas greater than a first preset threshold as overlapping matching pairs; For each of the overlapping matching pairs, calculating a similarity score value of the overlapping matching pair according to the sizes and overlapping conditions of the two prediction boxes of the overlapping matching pair; When the similarity score value is less than a preset score threshold, the prediction boxes in the overlapping matching pair are all used as intermediate prediction boxes, and the overlapping matching pair is used as a screening matching pair; When the similarity score value is greater than or equal to the preset score threshold, the prediction frame with the smaller maximum intersection area in the overlapping matching pair is used as the intermediate prediction frame; The similarity score is calculated as follows: Determine the aspect ratio according to the sizes of the two prediction boxes of the overlapping matching pair; Calculating the ratio of the minimum aspect ratio to the maximum aspect ratio of the two prediction boxes of the overlapping matching pair, and taking the product of the ratio and the maximum intersection area of the two prediction boxes of the overlapping matching pair as the similarity score value of the overlapping matching pair; The similarity score of the overlapping matching pair is calculated using the following formula: Among them, simab is the similarity score of the overlapping matching pair including prediction box a and prediction box b; IOAa and IOAb are the maximum intersection areas of prediction box a and prediction box b in the overlapping matching pair, respectively; γa and γb are the aspect ratios of prediction box a and prediction box b, respectively; is the minimum aspect ratio of γa and γb; is the maximum aspect ratio of γa and γb; The method of screening the intermediate prediction frame according to the relative position relationship of the bow of the ship to obtain the target prediction frame includes: Define the intermediate prediction box with a larger size in the screening matching pair as a first prediction box, and the intermediate prediction box with a smaller size as a second prediction box; For each of the screening matching pairs, execute: Taking each vertex of the first prediction box as an anchor point, constructing a virtual box with the same size as the second prediction box inside the first prediction box; wherein the number of the virtual boxes is the same as the number of the anchor points; Calculating an intersection-over-union ratio between the second prediction box and the virtual box; When the intersection-over-union ratio is greater than or equal to a second preset threshold, the second prediction box is deleted to obtain a target prediction box.

2. The system according to claim 1, wherein The edge computing unit also includes: a video acquisition module and an image processing module; The video acquisition module accesses the coastal defense monitoring device through a hardware interface and is used to obtain real-time video images; The image processing module is used to preprocess the video images and then send them to the target detection and tracking module.

3. The system according to claim 1, characterized in that, When the target detection and tracking module executes the output of the predicted bounding boxes marked with ships using a pre-trained target detection model, it includes: Inputting the video image into a pre-trained YOLOv7 target detection model, and through the extraction of the backbone network, obtaining feature maps of the video image at three different scales; Performing feature fusion on the feature maps at three different scales through the FPN network to obtain three different sizes of prediction results; Sending the prediction results into the prediction head structure, and through reparameterized convolution and non-maximum suppression algorithms, outputting the predicted bounding boxes marked with ships.

4. The system according to claim 1, wherein The target detection and tracking module performs multi-target tracking in the following manner: For each target detection result, the following operations are performed: Based on the ship types marked in the current target prediction bounding box, the current target is divided. When there is no Kalman tracker for the current ship type, it is determined that the current target appears for the first time, and a corresponding potential Kalman tracker is initialized for it, and the first appearance time, ship type, and target prediction bounding box of the ship target are retained; where, the ship target and the Kalman tracker are in a one-to-one correspondence relationship; When there is a Kalman tracker for the current ship type, the current target prediction bounding box is respectively subjected to the area intersection-over-union operation with each target bounding box predicted by the Kalman tracker for the current ship type, and a cost matrix is constructed using the area intersection-over-union; Using the Hungarian algorithm to find the optimal match of the cost matrix. If the match is successful, the target bounding box predicted by the Kalman tracker itself is updated based on the current target prediction bounding box; If the match is not successful, it is determined that the current target appears for the first time, and a corresponding potential Kalman tracker is initialized for it, and the first appearance time, ship type, and target prediction bounding box of the ship target are retained; When the potential Kalman tracker successfully matches three times in a row, it is considered that the ship target truly exists, and a number is assigned to the Kalman tracker based on the order of appearance of the same type of target within the day; For each numbered Kalman tracker, the target bounding box predicted and updated in real time by the Kalman filter is used as the position of the target for tracking; If the potential Kalman tracker fails to match successfully five times in a row, the Kalman tracker is deleted, and it is considered that the ship target has disappeared.

5. The system according to claim 1, wherein The edge computing unit further includes: a target storage and reporting module; The target storage and reporting module is used to store the target detection results and target tracking results based on a pre-set storage policy, and report the results based on the communication quality between the edge computing unit and the backend server.

6. A real-time extraction method for coastal defense monitoring targets based on the system according to any one of claims 1-5, characterized in that Including: An edge computing unit connected to the coastal defense monitoring device is set on the device side of the coastal defense monitoring device, and the edge computing unit is used to obtain the video images captured in real time by the coastal defense monitoring device; where, a target detection and tracking module is set in the edge computing unit, and a pre-trained target detection model is included in the target detection and tracking module; After performing object detection on the video image using the target detection model and outputting a prediction box marked with a ship, post-process the prediction box to screen and obtain a target prediction box; Perform multi-object tracking using the target detection and tracking module; The edge computing unit sends the stored edge-side data to the backend server.

Images