Edge node-based ship video condensation and intelligent retrieval method and system

Abstract

The application provides a ship video recording condensation and intelligent retrieval method and system based on an edge node, and belongs to the technical field of edge computing. The method comprises the following steps: S1: collecting ship video recording data; S2: generating condensed video and metadata by an improved Gaussian mixture model, an improved DeepSORT algorithm and a dynamic condensation ratio adjustment algorithm; S3: synchronizing index information of the condensed video and the metadata by each edge node through a publish-subscribe mode; S4: training a lightweight intelligent retrieval model through a knowledge distillation algorithm; S5: calling the lightweight intelligent retrieval model to match corresponding condensed video segments and returning a retrieval result; and S6: collecting feedback information of the retrieval result and performing incremental optimization on the lightweight intelligent retrieval model through an elastic weight consolidation algorithm. The application realizes efficient monitoring management through an edge localization processing+cross-node cooperation+closed-loop optimization architecture, and meets the actual needs of ship safety monitoring and operation management.

Description

Technical Field

[0001] This invention relates to the field of intelligent ship monitoring and edge computing, specifically to a method and system for ship video recording summarization and intelligent retrieval based on edge nodes. Background Technology

[0002] During ship navigation, a large amount of video data needs to be collected through multi-area monitoring equipment (such as bridge operations, deck work, cargo loading and unloading, etc.). Traditional technologies have the following limitations:

[0003] High storage pressure: The original video data volume is large (high-definition cameras can store tens of TB of data per day), while local storage resources on ships are limited and it is difficult to retain the data for a long time.

[0004] Low retrieval efficiency: It relies on manual review of complete video recordings to investigate events, which is time-consuming and labor-intensive, and lacks intelligent retrieval methods;

[0005] Centralized processing bottleneck: If data is uploaded to the cloud or central control center for unified processing, the limited network bandwidth of the ship will result in high latency and poor real-time performance.

[0006] Poor scene adaptability: Ship scenes have complex situations such as wave reflection, equipment shaking, and target occlusion, making it difficult for traditional video condensation (fixed ratio compression) and retrieval models (heavyweight models) to balance efficiency and accuracy. Summary of the Invention

[0007] This invention provides a method and system for ship video recording summarization and intelligent retrieval based on edge nodes, achieving efficient monitoring and management through an architecture of "edge localization processing + cross-node collaboration + closed-loop optimization".

[0008] Edge node deployment and data acquisition: Edge nodes are deployed in various monitoring areas of the ship to collect and record video in real time and store it locally to provide raw data for subsequent processing;

[0009] Dynamic video summarization: Based on an improved Gaussian mixture model to detect moving targets, combined with DeepSORT to achieve multi-target tracking, and then dynamically adjusts the summarization ratio according to the target density to generate a concise summarization video and associated metadata;

[0010] Cross-node data synchronization: Synchronize condensed videos and metadata of each node through a publish-subscribe model to ensure data consistency across the entire ship and support cross-regional retrieval;

[0011] Lightweight intelligent retrieval: A lightweight retrieval model is trained using knowledge distillation technology and deployed on edge nodes. Combined with hierarchical indexing, it quickly responds to users' text or image retrieval requests.

[0012] Closed-loop optimization mechanism: Based on user feedback on search results, the algorithm incremental optimization model is reinforced through elastic weights to continuously improve search accuracy.

[0013] The overall solution integrates edge computing and intelligent algorithms to achieve efficient storage and accurate retrieval of ship video recordings with limited resources, meeting the actual needs of ship safety monitoring and operation management.

[0014] One aspect of the embodiments of this specification discloses a method for ship video recording summarization and intelligent retrieval based on edge nodes, including:

[0015] S1: Deploy edge nodes in each monitoring area of ​​the ship, initialize the equipment parameters, collect ship video data, and store the raw video data in the temporary buffer of the edge nodes;

[0016] S2: Edge nodes perform moving target detection on the original video recording in the temporary buffer using an improved Gaussian mixture model to obtain a target mask; based on the target mask, the improved DeepSORT algorithm is used to track the moving target to obtain the target ID and motion trajectory; the number of targets is counted according to the target ID, and combined with the time information of the motion trajectory, a dynamic compression ratio adjustment algorithm based on target density is used to filter valid frames and reassemble them to generate a compressed video. At the same time, metadata containing the target ID, motion trajectory and timestamp is generated. The compressed video and metadata are stored in the long-term storage area, and the original video recording in the temporary buffer is deleted.

[0017] S3: Each edge node synchronizes the index information of condensed video and metadata through a publish-subscribe model. If missing data is detected locally, it requests transmission and storage from the publishing node to ensure that the long-term storage area of ​​each edge node contains the condensed video and metadata of the entire ship.

[0018] S4: The central control node collects the synchronous metadata of each edge node to build a training dataset. The motion trajectory in the metadata is used to extract target features. A lightweight intelligent retrieval model is trained through the knowledge distillation algorithm and the model is deployed to each edge node.

[0019] S5: Users input search requests through the central control system, and edge nodes call the lightweight intelligent search model. Combining motion trajectory time information in metadata with hierarchical indexing algorithms, the corresponding condensed video clips are matched in the local long-term storage area, and the search results are returned.

[0020] S6: Collect user feedback on search results, incrementally optimize the lightweight intelligent search model through the elastic weight consolidation algorithm, and synchronize the optimized model to each edge node to form a closed loop.

[0021] In this specification, in step S2, the dynamic compression ratio adjustment algorithm based on target density interacts with the improved DeepSORT algorithm. Specifically, the target ID output by the improved DeepSORT algorithm is used to count the number of targets in each frame, calculate the average target density within the time window, and determine the dynamic compression ratio based on the average target density. The higher the average target density, the smaller the compression ratio.

[0022] In this specification, in step S4, the knowledge distillation algorithm includes constructing a teacher model and a student model. The teacher model is a high-precision YOLOv8 and BERT combination model, and the student model is a lightweight MobileNetV3 and simplified BERT combination model. During training, the output of the teacher model is used as a guide to optimize the student model by fusing classification loss and distillation loss, so that the student model can be adapted to the computing power of the edge nodes.

[0023] In this specification, in step S5, the hierarchical indexing algorithm includes a first-level index and a second-level index. The first-level index divides the metadata into partitions according to the edge node ID and time interval. The second-level index indexes the target feature vector in the metadata in each partition through a KD tree. During retrieval, the candidate partition is first located through the first-level index, and then the target feature vector similar to the retrieval request is quickly matched through the second-level index.

[0024] In this specification, in step S6, the elastic weight consolidation algorithm interacts with the user feedback in step S5, specifically as follows: determining the importance of model parameters to historical tasks based on user feedback and generating a Fisher information matrix; during incremental training, limiting the variation of important parameters by adding a parameter protection term to the loss function to prevent the model from forgetting historical knowledge.

[0025] In this specification, in step S3, the interaction process of the publish-subscribe mode is as follows: each edge node publishes the index information of the local condensed video and metadata to the message queue, other nodes subscribe to the message queue and compare the local index. If it is missing, they send a request containing the file identifier to the publishing node. The publishing node transmits the data through the breakpoint resume protocol. After the receiving node verifies the integrity, it updates the local storage.

[0026] In this specification, in step S2, the improved Gaussian mixture model is optimized for ship scenarios. By dynamically adjusting the number of Gaussian components and the update rate, the background modeling adapts to scenarios of wave reflection and equipment vibration, and the target mask can accurately distinguish moving targets from the undulating background.

[0027] In this specification, in step S5, the retrieval request includes a text description or a target feature image. The edge node converts the text description into a semantic vector and the target feature image into a visual feature vector through a lightweight intelligent retrieval model. Then, it calculates the similarity with the target feature vector in the metadata and filters out condensed video clips with similarity reaching a threshold.

[0028] In this specification, the triggering condition for the closed-loop optimization in step S6 is as follows: when the number of collected user feedback reaches a preset threshold, the elastic weight consolidation algorithm is started to incrementally optimize the model. The optimized model is updated to each edge node through the synchronization mechanism in step S3 until the retrieval accuracy meets the preset requirements.

[0029] Another aspect of the embodiments of this specification discloses a ship video recording summarization and intelligent retrieval system based on edge nodes. Applying any one of the above-described methods for ship video recording summarization and intelligent retrieval based on edge nodes, the ship video recording summarization and intelligent retrieval system based on edge nodes includes:

[0030] The acquisition module is used to deploy edge nodes in various monitoring areas of the ship, acquire ship video data after initializing the equipment parameters, and store the raw video data in the temporary buffer of the edge nodes;

[0031] The edge processing module is used to detect moving targets in the original video recordings in the temporary buffer at edge nodes using an improved Gaussian mixture model to obtain target masks. Based on the target masks, the module tracks the moving targets using an improved DeepSORT algorithm to obtain target IDs and motion trajectories. The module counts the number of targets based on their IDs and combines the time information of the motion trajectories. It then uses a dynamic compression ratio adjustment algorithm based on target density to filter valid frames and reassemble them to generate a compressed video. Simultaneously, it generates metadata containing target IDs, motion trajectories, and timestamps. The compressed video and metadata are stored in the long-term storage area, and the original video recordings in the temporary buffer are deleted.

[0032] The synchronization module is used to synchronize the index information of condensed video and metadata on each edge node through a publish-subscribe model. If missing data is detected locally, it requests transmission and storage from the publishing node to ensure that the long-term storage area of ​​each edge node contains the condensed video and metadata of the entire ship.

[0033] The training and deployment module is used to collect synchronous metadata from each edge node at the central control node to build a training dataset. The motion trajectory in the metadata is used to extract target features. A lightweight intelligent retrieval model is trained through a knowledge distillation algorithm and the model is deployed to each edge node.

[0034] The feedback module allows users to input search requests through the central control system. Edge nodes call a lightweight intelligent search model, which combines motion trajectory time information in metadata with a hierarchical indexing algorithm to match the corresponding condensed video clips in the local long-term storage area and return the search results.

[0035] The optimization module is used to collect user feedback on search results, incrementally optimize the lightweight intelligent search model through the elastic weight consolidation algorithm, and synchronize the optimized model to each edge node to form a closed loop.

[0036] The embodiments described in this specification can achieve at least the following beneficial effects:

[0037] Improve video storage efficiency: By dynamically adjusting the compression ratio, redundant videos are adaptively compressed while retaining key information (such as personnel operations and equipment operation), thereby reducing storage resource consumption.

[0038] Accelerate retrieval response speed: Based on a lightweight model and hierarchical index, enable fast local retrieval at edge nodes, reduce dependence on centralized computing power, and meet real-time query requirements;

[0039] Enhanced scene adaptability: The optimized target detection and tracking algorithm can cope with the complex environment of ships (such as reflection and occlusion) and improve the stability of target recognition;

[0040] Optimize cross-node collaboration capabilities: Through an efficient data synchronization mechanism, information sharing among multiple edge nodes is achieved, supporting video retrieval and linked analysis across the entire ship;

[0041] Achieving continuous model evolution: By combining incremental optimization mechanisms with user feedback, the retrieval model can be dynamically adapted to actual application scenarios, maintaining high accuracy over the long term. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of a ship video recording summarization and intelligent retrieval method based on edge nodes, as described in some embodiments of the present invention.

[0044] Figure 2 This is a schematic diagram of the ship video recording enrichment process involved in some embodiments of the present invention.

[0045] Figure 3This is a schematic diagram of the cross-node data synchronization process involved in some embodiments of the present invention.

[0046] Figure 4 This is a schematic diagram of the intelligent retrieval process involved in some embodiments of the present invention.

[0047] Figure 5 This is a schematic diagram of the model closed-loop optimization process involved in some embodiments of the present invention. Detailed Implementation

[0048] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0049] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0050] like Figure 1 As shown, this embodiment provides a method for ship video recording summarization and intelligent retrieval based on edge nodes, including:

[0051] S1: Deploy edge nodes in each monitoring area of ​​the ship, initialize the equipment parameters, collect ship video data, and store the raw video data in the temporary buffer of the edge nodes;

[0052] S2: Edge nodes perform moving target detection on the original video recording in the temporary buffer using an improved Gaussian mixture model to obtain a target mask; based on the target mask, the improved DeepSORT algorithm is used to track the moving target to obtain the target ID and motion trajectory; the number of targets is counted according to the target ID, and combined with the time information of the motion trajectory, a dynamic compression ratio adjustment algorithm based on target density is used to filter valid frames and reassemble them to generate a compressed video. At the same time, metadata containing the target ID, motion trajectory and timestamp is generated. The compressed video and metadata are stored in the long-term storage area, and the original video recording in the temporary buffer is deleted.

[0053] S3: Each edge node synchronizes the index information of condensed video and metadata through a publish-subscribe model. If missing data is detected locally, it requests transmission and storage from the publishing node to ensure that the long-term storage area of ​​each edge node contains the condensed video and metadata of the entire ship.

[0054] S4: The central control node collects the synchronous metadata of each edge node to build a training dataset. The motion trajectory in the metadata is used to extract target features. A lightweight intelligent retrieval model is trained through the knowledge distillation algorithm and the model is deployed to each edge node.

[0055] S5: Users input search requests through the central control system, and edge nodes call the lightweight intelligent search model. Combining motion trajectory time information in metadata with hierarchical indexing algorithms, the corresponding condensed video clips are matched in the local long-term storage area, and the search results are returned.

[0056] S6: Collect user feedback on search results, incrementally optimize the lightweight intelligent search model through the elastic weight consolidation algorithm, and synchronize the optimized model to each edge node to form a closed loop.

[0057] In some embodiments, in step S2, the interaction process between the motion trajectory and the dynamic compression ratio adjustment algorithm is as follows: the duration of the target in the video segment is determined based on the time information of the motion trajectory, the target density is calculated in combination with the number of targets, and the dynamic compression ratio is adjusted according to the target density and the duration of the trajectory. The longer the duration of the trajectory, the higher the effective frame retention ratio of the target.

[0058] In some embodiments, in step S4, the motion trajectory in the metadata is used to extract target features specifically by: extracting the motion trend features (such as linear motion or reciprocating motion) of the target from the position coordinates of the motion trajectory, and combining them with the category information corresponding to the target ID to jointly constitute the sample features of the training dataset for training of the lightweight intelligent retrieval model.

[0059] In some embodiments, in step S5, the process of combining the motion trajectory time information with the hierarchical indexing algorithm is as follows: when the first-level index of the hierarchical index is divided according to time intervals, the start and end timestamps of the motion trajectory in the metadata are referenced to make the partition boundary match the time range of the target activity, thereby improving the partition positioning efficiency during retrieval.

[0060] In some embodiments, in step S2, when the improved DeepSORT algorithm generates the motion trajectory, it solves the problem of trajectory breakage after the target is occluded or disappears briefly by fusing the appearance features of the target with the continuity of the motion trajectory, ensuring that the time information and position coordinates of the motion trajectory are continuous and complete, and providing reliable data for subsequent condensed video generation and metadata construction.

[0061] In some embodiments, in step S3, the synchronized metadata includes motion trajectory information. Each edge node verifies the consistency between the timestamp and the location coordinates of the motion trajectory to ensure that the synchronized metadata across nodes matches in timeline and spatial location, thus avoiding time or location deviations during retrieval.

[0062] In some embodiments, in step S5, when the retrieval request is a target activity within a specific time period, the edge node quickly locates the condensed video segment corresponding to the target's location coordinates within that time period by matching the time information of the motion trajectory in the metadata, thereby improving the spatiotemporal accuracy of the retrieval results.

[0063] In some embodiments, during step S2, when generating the condensed video, valid frames containing the complete activity of the target are filtered based on the position coordinates of the motion trajectory, and redundant frames without a target or with incomplete target trajectories are removed to ensure that the condensed video can completely present the motion process of the target.

[0064] In some embodiments, during step S4, the feature vector of the motion trajectory and the feature vector of the target category are fused as input during the training process of the lightweight intelligent retrieval model, so that the model learns the correlation between the target category and the motion pattern, thereby improving the retrieval accuracy of "specific motion of a specific category target" (such as "crane hoisting motion").

[0065] In some embodiments, in step S6, the user feedback information includes a matching evaluation of the target motion trajectory in the search results. The elastic weight consolidation algorithm adjusts the parameter weights related to the motion trajectory features in the model according to the evaluation, so that the model can learn the motion trajectory patterns of different targets more accurately and optimize the subsequent search results.

[0066] In some embodiments, the deployment process of the lightweight intelligent retrieval model in step S4 includes: quantizing and pruning the trained student model to reduce the model parameter size and computational load, ensuring that the single-frame inference time of the edge nodes meets the real-time retrieval requirements, and that the target feature vector output by the model can be directly used for the hierarchical index construction in step S5.

[0067] The technical concept of this invention is as follows:

[0068] S1: Edge Node Initialization and Ship Video Recording

[0069] Core function: To provide high-quality raw video data for the entire system, and to ensure that the processing foundation of subsequent video summarization, retrieval and other links is consistent through standardized equipment deployment and parameter configuration, while establishing a stable data storage architecture.

[0070] Equipment deployment and initialization

[0071] Edge node devices are deployed in key monitoring areas of the vessel. Specific locations (which can be customized based on actual needs) include: the bridge (1 node, monitoring crew operations and instrument panels), the deck (2 nodes, covering the forward hoisting area and aft work area respectively), the cargo hold entrance (1 node, monitoring cargo loading and unloading processes), and the engine room (1 node, monitoring equipment operating status). Each edge node contains the following core components (specific selection can also be customized based on actual needs):

[0072] 1. High-definition camera: It adopts a starlight-level CMOS sensor, supports 1920×1080 resolution (to ensure clear target details), and has a fixed frame rate of 25 frames / second (to balance smoothness and storage usage). The lens field of view is adjusted according to the area (e.g., 90° wide-angle for the cockpit and 120° ultra-wide-angle for the cargo door entrance).

[0073] 2. Local storage module: Divided into a temporary cache area (10GB, using NVMe protocol, read / write speed ≥500MB / s, used to temporarily store unprocessed raw video) and a long-term storage area (100GB, using SATA interface, used to store condensed video and metadata).

[0074] 3. Data processing chip: NVIDIA Jetson Orin NX (8-core ARM CPU + 1024-core GPU) is selected, which supports parallel processing of video frames and running lightweight AI models;

[0075] 4. Network communication module: Supports Gigabit Ethernet and Wi-Fi 6, enabling data transmission between nodes through the ship's internal local area network (LAN), with an average latency controlled within 50ms.

[0076] The initialization process consists of three steps:

[0077] 1. Device self-test: After the edge node is powered on, it automatically detects the camera focal length (calibrated by shooting a standard checkerboard pattern, error ≤1%), frame rate (continuous detection for 30 seconds, fluctuation range ≤±1 frames / second), storage module read and write speed (time taken to write a 1GB file to the temporary cache ≤2 seconds), and network connectivity (ping gateway packet loss rate ≤1%). If any item fails to meet the standard, a local alarm is triggered (indicator light flashes) and the log is recorded.

[0078] 2. Parameter Configuration: The central control system sends unified parameters to each node, including video encoding format (H.265, compression ratio higher than H.26450%), file naming rules ("edge node ID_timestamp_segment number", such as "deck_front_20XX08191400_001"), and temporary cache area cleanup threshold (processed files are automatically deleted when the remaining space is <2GB).

[0079] 3. Time synchronization: Align with the ship's central control clock via NTP protocol to ensure that the timestamp error of each node is ≤10ms (to avoid timeline confusion during subsequent cross-node data synchronization).

[0080] Video capture and storage

[0081] The camera collects video data in real time according to the set parameters, as follows:

[0082] 1. Data collection scope: The bridge node covers the captain's console, radar display screen, and crew positions; the deck node covers cranes, container stacking areas, and personnel passageways; the cargo hold node covers hatch opening and closing, and loading and unloading machinery operation areas; the engine room node covers generator sets and pipeline pressure gauges.

[0083] 2. Data format: The original video is encapsulated as an MP4 file using H.265 encoding. A segment is generated every 5 minutes (each segment is about 500MB, which facilitates subsequent block processing). Metadata (including node ID, capture time, resolution, and frame rate) is embedded in the file header.

[0084] 3. Storage strategy: Video clips are first written to a temporary buffer. Once generated, the S2 processing flow is triggered immediately (by notifying the processing chip through a local inter-process communication mechanism). After S2 completes the condensation process and generates metadata, the original files in the temporary buffer are deleted (to free up space), and only the condensed video and metadata in the long-term storage area are retained.

[0085] Connection with subsequent steps: The original video clips in the temporary buffer are the input data of S2. Their naming rules and timestamps provide a time reference for target tracking (DeepSORT) in S2, ensuring that video clips from different nodes can be associated through timestamps.

[0086] S2: Local video recording compression processing at edge nodes

[0087] Algorithm 1: An improved Gaussian mixture model (GMM) for moving target detection

[0088] Core function: To solve the problem of unstable backgrounds caused by wave reflections and equipment vibration in ship scenes, accurately distinguishing moving targets (personnel, cargo, equipment) from stationary backgrounds (decks, bulkheads, sea surface). Ship video condensation workflow (combined with surface fitting) is as follows: Figure 2 As shown.

[0089] Model building details:

[0090] Let the video frame sequence be ,in The pixel value of the t-th frame image (pixel coordinates (x, y) is... (x,y)).

[0091] Ship surveillance videos exhibit high-frequency background fluctuations (such as pixel value jumps caused by wave reflections and image shifts due to ship swaying). Traditional Gaussian Mixture Models (GMMs) often misinterpret these fluctuations as moving targets. To address this, the improved GMM constructs K Gaussian components for each pixel (x, y) (K=5 in ship scenes, two more components than in conventional scenes to accommodate fluctuations). The parameters of each component are defined as follows:

[0092] : The mean of the k-th Gaussian component of pixel (x,y) in frame t (representing the brightness of the background pixel corresponding to this component);

[0093] In frame t, the variance of the k-th Gaussian component of pixel (x,y) (a larger value indicates more dramatic fluctuations in the background area, such as ocean waves). (usually larger than the deck area).

[0094] In frame t, the weight of the k-th Gaussian component of pixel (x,y) (the higher the weight, the more likely the component is to represent the real background), and satisfying the following conditions: =1.

[0095] Model training (background dynamically updated):

[0096] During the initialization phase (first 300 frames), the five Gaussian components of each pixel are randomly assigned an initial mean (close to the pixel value of the first frame), variance (initial value 50), and weight (averaged 0.2). In subsequent frame updates, the parameters are adjusted according to the following rules:

[0097] 1. Matching judgment: If the current pixel value satisfy If the pixel matches the k-th Gaussian component, it is considered to belong to the background.

[0098] 2. Parameter update during matching (Formulas 1-3):

[0099] (1)

[0100] (2)

[0101] (3)

[0102] in, =0.01 (Update rate of mean and variance; the value is lower in the ship scene than in the normal scene to avoid the background being skewed by short-term fluctuations). =0.005 (weight update rate). For example, pixel values ​​in the wave region fluctuate frequently; by reducing... This can prevent the background average from being raised by a sudden burst of strong light;

[0103] 3. Parameter update when there is a mismatch: Only reduce the weight of that component. This prevents non-background components from becoming dominant.

[0104] Model application (moving target detection):

[0105] During each frame processing, pixels The top B Gaussian components sorted by weight (B= =3) Match:

[0106] If a match is successful, it is determined to be a background pixel, and the target mask is used. =0;

[0107] If the match fails, it is determined to be a moving target pixel, target mask. =1 (1 represents the target).

[0108] For example, when a person walks across the deck, their pixel values ​​differ significantly from the background Gaussian components. =1; while the pixel values ​​of the wave reflection fluctuate within the range of 2.5σ, and are still judged as background ( =0), reducing false positives.

[0109] Algorithm 2: Improved DeepSORT for Multi-Target Tracking

[0110] Core function: Based on the target mask output by GMM, it enables continuous tracking of multiple targets (such as personnel working simultaneously or moving cargo) in a ship scene, and solves the problem of trajectory breakage caused by occlusion (such as cargo obscuring personnel) and temporary disappearance (such as personnel entering the cabin).

[0111] Model building details:

[0112] The tracking process is divided into two parts: "target detection box extraction" and "cross-frame target association".

[0113] 1. Object detection bounding box extraction: Object mask output from GMM In the process, the minimum bounding rectangle of each moving target is extracted through connected component analysis, denoted as . ,in:

[0114] : The center coordinates of the i-th target in the t-th frame;

[0115] The width and height of the target bounding box;

[0116] 2. Cross-frame target association: By fusing motion and appearance features, determine whether target i in frame t and target j in frame (t-1) are the same target, and assign similarity. Defined as:

[0117] (4)

[0118] in:

[0119] Motion similarity (Mahaviran distance) reflects the deviation between the Kalman filter-predicted position and the actual detected position.

[0120] (5)

[0121] For the predicted bounding box of target j in frame t-1, To predict the covariance matrix, a smaller value indicates a more accurate prediction; The minimum bounding rectangle (i.e., the detection box);

[0122] Appearance similarity (cosine distance) reflects the consistency of visual features of targets in two frames.

[0123] (6)

[0124] Let i be the feature vector of target i in frame t, extracted by CNN, containing information such as color and contour, like the color of a person's safety helmet and the texture of a cargo box; Let j be the feature vector of target j in frame t-1;

[0125] =0.3 (Motion Feature Weight): In ship scenarios, the target movement is relatively slow (such as people walking or cargo being hoisted), and the appearance features are more stable, so the motion feature weight is reduced.

[0126] Model training and application:

[0127] Training: Using labeled data (including target IDs and detection boxes) from ship monitoring scenarios, optimize the state transition matrix of the Kalman filter and the CNN feature extractor to achieve the desired results. It can distinguish between different targets (e.g., the feature vector distance between "crane" and "personnel" is >0.8);

[0128] Application: For each frame of targets, the Hungarian algorithm is used to find them. The largest matching pair is assigned a unique ID. If the target disappears within 30 frames (e.g., personnel enter the cabin), retain its ID and wait for it to reappear; if it disappears for more than 30 frames, release the ID.

[0129] For example: When hoisting goods and obstructing personnel, It may increase (predicted location deviation), but Still relatively small (the characteristics of the personnel's safety helmets have not changed), through The fusion process can still maintain ID continuity.

[0130] Algorithm 3: Dynamic Concentration Ratio Adjustment Based on Target Density

[0131] Core function: Based on the target tracking results output by DeepSORT, dynamically adjust the video compression ratio to maximize the compression of redundancy while ensuring the integrity of key information (e.g., more compression for low-density scenes and less compression for high-density scenes).

[0132] Model building details:

[0133] The concentration ratio is determined by the target density. Specific steps:

[0134] 1. Target density calculation: Define the target density of frame t. The ratio of the number of targets to the area of ​​the image:

[0135] (7)

[0136] in:

[0137] : Total number of targets in frame t (via To remove duplicates, if there are 3 people and 1 crane appearing at the same time, =4);

[0138] W×H=1920×1080 (video resolution, unit: pixels);

[0139] 2. Average density within the time window:

[0140] Calculate the time window based on the tracking results. Average target density within: ,

[0141] For example, for a 5-minute video clip (containing 7500 frames, 25 frames / second), the average density is calculated using a 10-second window (250 frames). : (8)

[0142] The window length is set to 10 seconds to balance real-time performance and stability, and to avoid instantaneous density fluctuations affecting the judgment.

[0143] Original video length (Frame count), condensed duration Valid frames are retained and redundant frames are discarded according to the timeline of the tracking trajectory.

[0144] 3. Determination of dynamic concentration ratio: based on Set ratio R:

[0145] (9)

[0146] Interaction process with Algorithm 2:

[0147] DeepSORT output It is a calculation The sole basis (ensuring each target is counted only once), and thus through and Determine R. For example:

[0148] Low-density scenes ( <0.001): For example, on an empty deck at night, there is only one target (seagull) within 10 seconds. =1, =1 / (1920×1080)≈4.8× R=20 (10 minutes of original video condensed into 30 seconds);

[0149] Medium density scenes ( <0.005): If there are two crew members operating the cockpit, =2, ≈9.6× R=10 (10 minutes condensed into 1 minute);

[0150] High-density scenes ( 0.005): For example, if 10 people are loading and unloading cargo in the cargo hold, =10, ≈4.8× R=5 (10 minutes condensed into 2 minutes).

[0151] Condensed video generation:

[0152] Filter valid frames from the original video using R:

[0153] For high-density scenes, retain 1 frame out of every R=5 frames (to retain more details);

[0154] For low-density scenes, retain 1 frame out of every R=20 frames (for redundancy compression).

[0155] The selected valid frames are reassembled chronologically to generate a condensed video, while retaining the timestamp associated with the target ID (for subsequent retrieval).

[0156] S3: Synchronization between metadata and edge nodes of condensed video

[0157] Core function: To enable data sharing among edge nodes, ensuring that any node can access the ship's condensed video and metadata, supporting model training in S4 (requiring full data) and cross-regional retrieval in S5 (such as simultaneously querying related events on decks and in cargo holds). The cross-node data synchronization process is as follows: Figure 3 As shown.

[0158] Communication architecture and synchronization protocol

[0159] Each edge node forms a distributed network through the ship's local area network, using a "publish-subscribe" message queue (based on RabbitMQ) as the data synchronization hub. The specific architecture is as follows:

[0160] Message queue server: Deployed on the central control node, it is responsible for receiving the published information from each edge node and pushing it to subscribers. It supports message persistence (to avoid loss in case of power failure) and priority queue (metadata synchronization has higher priority than video index).

[0161] Node roles: Each edge node acts as both a publisher (pushing locally generated data indexes) and a subscriber (retrieving data indexes from other nodes), and is uniquely identified by its node ID (e.g., “cockpit_1”, “deck_front_2”).

[0162] The synchronization protocol uses a custom protocol based on TCP, which includes the following fields:

[0163] Header (16 bytes): Synchronization data type (metadata / video index), data length, checksum;

[0164] Main body (variable length): File identifier (associated with the file name of the condensed video generated by S2), storage path, generation time, data hash value (SHA-256, used to verify integrity);

[0165] Tail (4 bytes): End marker.

[0166] Synchronization process

[0167] Data publishing: After S2 generates a condensed video (such as "deck_front_202508191400_001_con.mp4") and its metadata ("deck_front_202508191400_001_meta.json"), the edge nodes extract the index information (file identifier, storage path, generation time) of both, calculate the hash value, encapsulate it into a message, and publish it to the message queue. The publishing frequency is consistent with the processing frequency of S2 (once every 5 minutes).

[0168] Index Subscription and Comparison: Other nodes subscribe to the message queue in real time. Upon receiving a new index, they query the local long-term storage area using the file identifier.

[0169] If the file already exists locally (hash value matches), ignore it;

[0170] If the local file is missing or the hash value does not match (the file is corrupted), a data request (containing the file identifier and the local storage path) is sent to the publishing node.

[0171] Data transmission and verification: After receiving the request, the publishing node sends the file via a resume transmission protocol (supporting continuation of transmission after pause):

[0172] Metadata (JSON format, PB encoded + LZ77 compressed, single file <100KB) is transmitted first. After the receiving node decompresses it, it verifies the hash value. If it is correct, it is written to the "meta" subdirectory of the long-term storage area.

[0173] The condensed video (H.264 encoded, each segment is about 50MB) is transmitted in chunks (each chunk is 1MB). The receiving node verifies the hash of each chunk it receives. After all chunks are received, the files are merged, the total hash value is verified, and if there is no error, the file is written to the "video" subdirectory and the local index (which records the mapping between file identifiers and storage paths) is updated.

[0174] Synchronization status feedback: After the receiving node completes storage, it sends a "success" confirmation to the publishing node. If the transmission fails (such as network interruption), the request will be resent after 30 seconds. If the retries fail 3 times, the log will be recorded and reported to the central control system.

[0175] Connection with preceding and following steps:

[0176] Input: S2-generated condensed video and metadata (associated via file identifier);

[0177] Output: A condensed collection of videos and metadata for the entire ship, stored in the long-term storage area of ​​each node. The metadata serves as the dataset for training the S4 model, while the condensed videos and their indexes provide the data foundation for retrieval in S5. Synchronized metadata consistency (identical metadata across nodes) ensures that the model trained by S4 is universal across all nodes on the ship, and the integrity of the video index ensures that S5 can perform cross-region retrieval.

[0178] S4: Training and Deployment of Intelligent Retrieval Models (Knowledge Distillation Algorithm)

[0179] Core function: to compress high-precision but complex retrieval models into lightweight models, adapting them to low-computing-power chips at edge nodes, while maintaining high retrieval accuracy.

[0180] Model building details:

[0181] A "teacher-student" distillation architecture is adopted, where both output target feature vectors (for cross-modal retrieval). The specific structure is as follows:

[0182] Teacher Model High-precision but heavyweight, consisting of YOLOv8 (feature extraction, 8 CSPDarknet modules) and BERT (semantic encoding, 12 Transformer layers), outputting feature vectors. (Includes visual-semantic fusion information of the target);

[0183] Student Model Lightweight, consisting of MobileNetV3 (feature extraction, replacing YOLOv8's standard convolutions with depthwise separable convolutions) and a simplified BERT (6 Transformer layers), outputting a feature vector. (The parameters are only 15% of those of the teacher model, adapted to the ARM chip of the edge node).

[0184] Interaction process with S2:

[0185] S2 outputs the target feature vector (Extracted from DeepSORT's CNN) and the labels corresponding to the target IDs (such as "personnel" and "crane") are used as training data:

[0186] Positive sample pairs: ( , (Feature matching with tags, such as the feature "crane" corresponding to the tag "lifting operation");

[0187] Negative sample pairs: ( , (Feature and label do not match, such as the "personnel" feature corresponding to the "lifting operation" label).

[0188] Model training (distillation loss function):

[0189] The training objective is to enable the student model Output Approximating the teacher model Output Total loss for:

[0190] (10)

[0191] in:

[0192] Classification loss (ensuring the student model can correctly identify the target category):

[0193] (11)

[0194] C=10 represents the number of target categories in the ship scenario (c is the category index), such as "personnel", "crane", "container", etc. This is a real label; For the student model (the superscript S represents the student model, and is related to the student model) The predicted probability (corresponding to the subscript S); It is the natural logarithm function;

[0195] Distillation loss (transmitting implicit knowledge from the teacher model):

[0196] (12)

[0197] Let KL divergence be a metric. =3 is the temperature parameter, which makes the output probability distribution of the teacher model smoother, making it easier for students to learn details; KL divergence measures the difference between the two distributions, and the smaller the value, the closer the student is to the teacher; (·) represents the softmax activation function; For teacher model The output feature vector; For student models The output feature vector; The distribution separator for KL divergence;

[0198] =0.4: Classification loss weight (in the ship scenario, more emphasis is placed on distinguishing target categories, so a certain weight is retained).

[0199] Model Deployment:

[0200] After training is complete, only the student model is deployed. The processing chips at the edge nodes (such as NVIDIA Jetson Nano) have a single-frame inference time of <50ms (meeting the requirements of real-time retrieval) and maintain an accuracy of more than 92% of the teacher model (e.g., the retrieval accuracy of "crane operation" drops from 95% to 90%, but still meets practical requirements).

[0201] S5: Processing and responding to user search requests (hierarchical indexing algorithm)

[0202] Core Function: Based on the student model feature vectors of S4, it quickly locates condensed video clips matching user requests, solving the retrieval delay problem caused by limited computing power of edge nodes. The intelligent retrieval process is as follows: Figure 4 As shown.

[0203] Model building details:

[0204] A two-level index structure is used to reduce the number of feature comparisons during retrieval.

[0205] 1. First-level index (partition index): based on "edge node IDm" and "time range" "Partitioning metadata" ,For example:

[0206] Edge node ID m=1 corresponds to the cockpit, m=2 corresponds to the deck;

[0207] The time interval is divided by day, such as [20XX-08-19 00:00, 20XX-08-19 23:59];

[0208] Each partition is associated with all target feature vectors of that node within that time period. And a condensed video index;

[0209] 2. Secondary index (feature index): in each partition Internally, feature vectors are indexed using KD-trees. KD-trees recursively divide the feature space (e.g., by alternating between the first dimension, the second dimension, etc.) to organize high-dimensional features (512 dimensions) into a tree structure, so that similar features are clustered in the same subtree, reducing the number of comparisons during retrieval.

[0210] Interaction process with S4:

[0211] User search requests are processed through the following process:

[0212] 1. Request Parsing: The user inputs text (e.g., "Find personnel activities on deck at 2 PM on August 19th") or images (e.g., personnel photos), which the central control system converts into query vectors. :

[0213] Text request: Encoded into a semantic vector using a simplified BERT encoding based on S4;

[0214] Image request: Extracted as a visual feature vector using S4's MobileNetV3;

[0215] 2. First-level index filtering: Locate the partition based on the "edge node (deck, m=2)" and "time (August 19th, 2 PM)" in the request. ;

[0216] 3. Secondary index retrieval: Calculate the KD-tree of this partition. With all Similarity:

[0217] (13)

[0218] Cosine similarity, the closer the value is to 1, the better the match;

[0219] 4. Result Filtering: Retain The feature vector and its corresponding metadata (including condensed video path and timestamp) are the search results.

[0220] For example: When a user searches for "personnel activities on deck at 2 PM on August 19th":

[0221] The first-level index locates the partitions m=2 (deck) and [14:00, 15:00].

[0222] The secondary index quickly finds semantic vectors similar to "personnel activities" in the KD-tree of this partition. (e.g., personnel characteristics with a similarity of 0.85);

[0223] Return the corresponding condensed video clip (such as the personnel operation clip from 14:05 to 14:10) and timestamp.

[0224] S6: Feedback on search results and model optimization (Elastic Weight Consolidation Algorithm, EWC)

[0225] Core function: Incrementally optimize the student model in S4 based on user feedback on search results. This also prevents the model from forgetting historical knowledge (such as the learned ability to recognize "crane"). The model closed-loop optimization process is as follows: Figure 5 As shown.

[0226] Model building details:

[0227] The EWC algorithm achieves incremental learning by preserving parameters important to historical tasks. Specifically:

[0228] 1. Parameter Importance Assessment: Define Model Parameters (e.g., the convolutional kernel weights in MobileNetV3), calculate the Fisher information matrix F, where Indicates parameters Importance to historical mission:

[0229] (14)

[0230] This represents the probability distribution of the student model's prediction for label y. Larger means The more important the historical task, such as identifying "cranes", the better the convolution kernel parameters. (Higher value); take the expected value after squaring. This yields an average measure of the importance of this parameter—the higher the value, the greater the significance. The more critical the impact on model predictions.

[0231] 2. Incremental training loss: Add a parameter protection term and a new loss function. for:

[0232] (15)

[0233] in:

[0234] : The optimal parameters after the training of the historical task is completed (such as the optimal parameters for recognizing "crane" after the initial training).

[0235] =0.5: Protection item weight (balancing new task learning and retention of historical knowledge).

[0236] Interaction process with S5:

[0237] User feedback influences model optimization through the following processes:

[0238] 1. Feedback Collection: Users mark the search results returned by S5 as "relevant" or "irrelevant," for example:

[0239] Mark "relevant": The target in the search results matches the request (e.g., the result "personnel activities" is indeed personnel).

[0240] Mark as "irrelevant": The search results do not match the request (e.g., "personnel activity" results are mixed with cranes);

[0241] 2. Supervision signal conversion: Converting feedback into label correction, such as the predicted probability of "irrelevant" samples. Forced to be reduced;

[0242] 3. Incremental training: For every 100 feedback points collected, use... Retraining the student model Update parameters :

[0243] right Large parameters (such as "crane" identification parameters). This restricts its variation and prevents it from being forgotten;

[0244] right Smaller parameters (such as the "new cargo" identification parameter) allow for more significant updates, improving accuracy for new tasks;

[0245] 4. Model synchronization: after the update The old model is replaced by pushing the data to all nodes through an edge node synchronization mechanism.

[0246] For example: The initial model had low accuracy in retrieving "people wearing orange life jackets" (easily confused with "orange containers"), and after users marked it as "irrelevant" multiple times:

[0247] Parameters related to "orange life jacket" Smaller (not important in historical tasks), allows for updates;

[0248] After training, these parameter adjustments increased the feature vector distance between "personnel" and "containers" from 0.6 to 0.85, improving the retrieval accuracy to 90%.

[0249] Meanwhile, the "crane" identification parameters are due to Large but small changes, accuracy remains stable (from 95% to 94%).

[0250] Summary of Algorithm Collaboration Relationships

[0251] 1. S2 Internal Collaboration: Target mask output by GMM → DeepSORT extracts detection boxes and tracks IDs → ID counts target quantity → Density calculation → Dynamically determine the concentration ratio R;

[0252] 2. S2 and S4 Collaboration: Target Features Output by DeepSORT →S4 training samples→Knowledge distillation to generate student model ;

[0253] 3. S4 and S5 collaboration: Output feature vector →S5 secondary index→and query vector Calculate similarity → Return search results;

[0254] 4. S5 and S6 Collaboration: User feedback on search results → Supervision signal from S6 → EWC algorithm optimization →Optimized Improve the retrieval accuracy of S5.

[0255] Overall step-by-step connection logic

[0256] S1→S2: The original video clips (temporary buffer) acquired by S1 are used as input to S2. Time synchronization of S1 ensures that the target timelines of different nodes in S2 can be aligned.

[0257] S2→S3: The condensed video and metadata (with a unique identifier) ​​generated by S2 are used as the synchronization object of S3. The dynamic condensation ratio of S2 determines the amount of video data transmitted in S3 (the higher the ratio, the smaller the amount of data transmitted).

[0258] S3→S4: The entire ship's metadata set after S3 synchronization serves as the training dataset for S4. The target features and labels in the metadata provide supervision information for model training.

[0259] S4→S5: The lightweight model (student model) deployed in S4 provides feature extraction and semantic encoding capabilities for S5's retrieval requests. The feature vector output by the model is the basis for S5 to build its index.

[0260] S5→S6: User feedback (relevant / irrelevant) from S5 serves as a supervisory signal for S6, driving incremental model optimization;

[0261] S6→S4: The optimized model parameters of S6 are fed back into S4 to update the student models of each node and improve the accuracy of subsequent searches.

[0262] By following three main lines—data (raw video → condensed video → metadata), model (training → deployment → optimization), and feedback (user evaluation → model update)—each step forms a closed loop, ensuring that the system can achieve efficient condensation and accurate retrieval of ship video recordings with limited resources at edge nodes.

[0263] All algorithms form a closed loop through feature vectors, metadata, and feedback signals, ensuring that edge nodes can achieve efficient video summarization and accurate retrieval with limited computing power.

[0264] This embodiment provides a ship video recording summarization and intelligent retrieval system based on edge nodes. Applying any one of the above-described methods for ship video recording summarization and intelligent retrieval, the edge node-based ship video recording summarization and intelligent retrieval system includes:

[0265] The acquisition module is used to deploy edge nodes in various monitoring areas of the ship, acquire ship video data after initializing the equipment parameters, and store the raw video data in the temporary buffer of the edge nodes;

[0266] The edge processing module is used to detect moving targets in the original video recordings in the temporary buffer at edge nodes using an improved Gaussian mixture model to obtain target masks. Based on the target masks, the module tracks the moving targets using an improved DeepSORT algorithm to obtain target IDs and motion trajectories. The module counts the number of targets based on their IDs and combines the time information of the motion trajectories. It then uses a dynamic compression ratio adjustment algorithm based on target density to filter valid frames and reassemble them to generate a compressed video. Simultaneously, it generates metadata containing target IDs, motion trajectories, and timestamps. The compressed video and metadata are stored in the long-term storage area, and the original video recordings in the temporary buffer are deleted.

[0267] The synchronization module is used to synchronize the index information of condensed video and metadata on each edge node through a publish-subscribe model. If missing data is detected locally, it requests transmission and storage from the publishing node to ensure that the long-term storage area of ​​each edge node contains the condensed video and metadata of the entire ship.

[0268] The training and deployment module is used to collect synchronous metadata from each edge node at the central control node to build a training dataset. The motion trajectory in the metadata is used to extract target features. A lightweight intelligent retrieval model is trained through a knowledge distillation algorithm and the model is deployed to each edge node.

[0269] The feedback module allows users to input search requests through the central control system. Edge nodes call a lightweight intelligent search model, which combines motion trajectory time information in metadata with a hierarchical indexing algorithm to match the corresponding condensed video clips in the local long-term storage area and return the search results.

[0270] The optimization module is used to collect user feedback on search results, incrementally optimize the lightweight intelligent search model through the elastic weight consolidation algorithm, and synchronize the optimized model to each edge node to form a closed loop.

[0271] In some embodiments, for target motion trajectory jitter caused by ship turbulence (such as trajectory deviation caused by personnel walking due to ship swaying, or swaying during cargo hoisting), a surface fitting algorithm is introduced to smooth and optimize the motion trajectory, reduce noise interference, improve the spatial continuity and temporal consistency of the trajectory, and provide more reliable trajectory data for subsequent video condensation, metadata construction, and retrieval model training.

[0272] After "Obtaining the motion trajectory using the improved DeepSORT algorithm" in S2, add the sub-step "Optimizing the trajectory through surface fitting":

[0273] 1. Trajectory data preprocessing: Extract the target's time-series position coordinates from the raw motion trajectory output by DeepSORT. ,in For timestamps, Let k be the target position at time k.

[0274] 2. Surface Fitting Model Construction: A bivariate polynomial surface fitting method (such as a quadratic surface) is used. Using time t and spatial coordinates (x, y) as independent variables, the coefficients are solved using the least squares method. This makes the fitted surface as close as possible to the original trajectory;

[0275] 3. Optimize trajectory generation: Calculate each timestamp based on the fitted surface. Corresponding optimized position And set a residual threshold (such as the Euclidean distance between the original position and the fitted position). (pixels), if the residual exceeds the threshold, the original position is retained (to avoid overfitting and losing the true mutation trajectory, such as a person suddenly turning).

[0276] 4. Output optimized trajectory: Output the optimized trajectory As the final trajectory, it replaces the original trajectory for subsequent steps.

[0277] Interaction with other steps

[0278] 1. Interaction with the dynamic concentration ratio adjustment algorithm in S2:

[0279] The optimized trajectory provides a smoother trend in positional changes, making the target density calculation ( It is more stable. For example, the original trajectory may cause the same target to be counted repeatedly due to jitter (such as a person's swaying being misjudged as multiple targets). The optimized trajectory can correct the number of targets. ,make sure Accurate calculations lead to a more reasonable dynamic concentration ratio R.

[0280] 2. Interaction with metadata generation in S2:

[0281] The motion trajectory recorded in the metadata is updated to the optimized trajectory. The continuity of its timestamps and location coordinates enhances the reliability of metadata, providing accurate data support for "time-location based retrieval" in S5 (such as "finding targets active in the port side of the deck from 14:00 to 14:05").

[0282] 3. Interaction with model training in S4:

[0283] The optimized trajectory features (such as direction of motion and speed changes) are used as part of the training samples, enabling the lightweight intelligent retrieval model to learn the motion patterns of targets more accurately (such as "crane hoisting trajectory" and "person walking path"), and reducing model misjudgments caused by trajectory noise.

[0284] 4. Interaction with hierarchical indexes in S5:

[0285] In the secondary index, feature vectors are constructed based on the location coordinates of the optimized trajectory, so that targets with similar motion trajectories are clustered into one class in the index, improving the matching accuracy during retrieval (such as when searching for "goods with similar hoisting trajectories", targets with similar optimized trajectory features can be quickly located).

[0286] In some embodiments, step S2, after obtaining the motion trajectory using the improved DeepSORT algorithm, further includes a step of optimizing the trajectory using a surface fitting algorithm: based on the timestamp and position coordinates of the original motion trajectory, a bivariate polynomial surface fitting model is used, and the fitting coefficients are solved using the least squares method to generate the optimized motion trajectory. The residual threshold of the optimized trajectory is dynamically adjusted according to the degree of ship turbulence. The optimized trajectory is used to calculate the target density, generate metadata, and for subsequent model training.

[0287] In some embodiments, the interaction process between the surface fitting algorithm and the dynamic concentration ratio adjustment algorithm is as follows: the optimized motion trajectory corrects the statistical error of the target quantity, thereby increasing the target density. More accurate calculations, thus ensuring average density Matching with the dynamic concentration ratio R avoids incorrect adjustment of the concentration ratio due to jitter in the original trajectory.

[0288] In some embodiments, in step S4, the optimized motion trajectory is used to extract the motion pattern features of the target (such as motion direction and velocity change rate). These features are fused with the target category features and used as training samples, enabling the lightweight intelligent retrieval model to learn the association between the target category and the motion pattern, thereby improving the retrieval accuracy of "targets with specific motion patterns".

[0289] In some embodiments, in step S5, the secondary index of the hierarchical index is combined with the position coordinates of the optimized trajectory to construct a feature vector, so that targets with similar motion trajectories are clustered in the index. During retrieval, the matching condensed video clip is quickly located by comparing the motion pattern (such as "arc motion trajectory") in the query request with the trajectory features in the index.

[0290] By optimizing motion trajectories through surface fitting algorithms, the stability of target detection and tracking is further improved, making the video compression ratio more in line with the needs of actual scenarios. The accuracy of metadata and the reliability of model training are enhanced, ultimately improving the spatiotemporal matching accuracy of intelligent retrieval, which is especially suitable for high-turbulence ship scenarios.

[0291] I. Experimental Design

[0292] 1. Experimental Objective

[0293] The synergistic effect of the algorithm combination of "improved Gaussian mixture model + DeepSORT + dynamic concentration ratio adjustment", "knowledge distillation + hierarchical indexing" and "surface fitting + closed-loop optimization" in this scheme is verified. Compared with traditional methods, the advantages of the scheme in terms of video concentration efficiency, retrieval accuracy and scene adaptability are demonstrated.

[0294] 2. Experimental Environment

[0295] • Hardware: Edge nodes use NVIDIA Jetson Orin NX (8-core CPU + 1024-core GPU) and are equipped with 1920×1080 resolution cameras; central control nodes use Intel Xeon E5 servers;

[0296] Software: Python 3.8, OpenCV 4.5, PyTorch 1.12;

[0297] Dataset: Collected video footage of three ship scenarios (calm sea, moderate waves and heavy loading and unloading), each scenario lasting 24 hours, including target types: personnel (5-20 people), cranes (1-3 units), and containers (10-50 containers).

[0298] 3. Comparison group settings

[0299] Control group 1: Traditional fixed-ratio concentration (10:1) + manual retrieval;

[0300] Control group 2: Single algorithm combination (Gaussian mixture model + fixed concentration ratio + original model without distillation);

[0301] Experimental group: The complete algorithm combination of this scheme (improved GMM + DeepSORT + dynamic concentration + surface fitting + knowledge distillation + hierarchical indexing + closed-loop optimization).

[0302] II. Experimental Procedure

[0303] 1. Video summarization performance test

[0304] (1) The original videos of the three scenarios were processed using control group 1, control group 2, and experimental group, respectively;

[0305] (2) Calculate the condensed storage volume (GB) and key information retention rate (the proportion of key events such as personnel operation and equipment operation marked by manual annotations that are retained in the condensed video) for each group.

[0306] (3) Analyze the compatibility between dynamic concentration ratio and target density.

[0307] 2. Target tracking and trajectory optimization test

[0308] (1) In the mid-wave turbulence scenario, compare the number of target tracking ID switching between control group 2 (without surface fitting) and experimental group (with surface fitting) (the more ID switching, the more severe the trajectory breakage).

[0309] (2) Calculate the trajectory error between the two groups (the average Euclidean distance between the tracked trajectory and the manually labeled real trajectory, in pixels).

[0310] 3. Intelligent Search Performance Test

[0311] (1) Construct 100 search requests (such as "deck personnel activities from 14:00 to 16:00 in mid-wave scenario" and "crane hoisting trajectory in cargo-intensive scenario").

[0312] (2) Compare the retrieval accuracy (the proportion of relevant segments in the retrieval results), recall (the proportion of all relevant segments retrieved), and response time (the time from the input of the request to the return of the result) of the three groups.

[0313] (3) Test the changes in retrieval performance before and after closed-loop optimization of the experimental group (with 50 user feedbacks as the optimization trigger condition).

[0314] III. Experimental Data and Analysis

[0315] 1. Video summarization performance (Table 1)

[0316] Table 1. Examples of video compression performance comparison experiments

[0317] Scene Control group 1 (fixed 10:1) Control group 2 (single algorithm) Experimental group (complete algorithm) Concentrated storage capacity (GB) 240 180 120-200 (Dynamic) Key information retention rate 75% 82% 95%

[0318] analyze:

[0319] The experimental group, through the collaboration of "DeepSORT tracking target ID → dynamic concentration ratio adjustment", automatically reduced the concentration ratio (5:1) in the cargo-dense scenario (high target density) to retain more details; and increased the ratio (20:1) in the calm sea (low target density) to reduce storage volume, thus achieving a balance between "storage efficiency and information integrity".

[0320] The control group 2, which had no dynamic adjustment, had a higher rate of key information loss (18%) than the experimental group (5%) in high target density scenarios.

[0321] 2. Target tracking and trajectory optimization (Table 2)

[0322] Table 2. Examples of comparative experiments on target tracking and trajectory optimization

[0323] index Control group 2 (no surface fitting) Experimental group (with surface fitting) ID switching count (24 hours) 120 times 35 times Trajectory error (pixels) 15.6 6.2

[0324] analyze:

[0325] The experimental group solved the problem of target occlusion and positional change caused by mid-wave turbulence by combining "improved GMM to generate stable target masks → DeepSORT to track initial trajectories → surface fitting to optimize jittery trajectories".

[0326] The combination of surface fitting and DeepSORT reduces trajectory error by 60%, providing reliable data for subsequent effective frame selection (based on trajectory time information) and retrieval index construction (based on trajectory features) of condensed video.

[0327] 3. Intelligent search performance (Table 3)

[0328] Table 3. Examples of Intelligent Retrieval Performance Comparison Experiments

[0329] index Control group 1 (manually retrieved) Control group 2 (single algorithm) Experimental group (before optimization) Experimental group (optimized) accuracy 100% (human error-free) 65% 85% 92% Recall rate 30% (high rate of missed detection by manual methods) 58% 80% 88% Response time 30-60 minutes 8-10 seconds 1.5 to 2 seconds 1.2 to 1.8 seconds

[0330] analyze:

[0331] The synergy of the experimental group's "knowledge distillation lightweight model + hierarchical index" shortened the response time by 80% compared to the control group 2 (the model reasoning speed was increased by 3 times and the index retrieval efficiency was increased by 5 times).

[0332] Closed-loop optimization (user feedback → EWC algorithm model adjustment) further improved accuracy by 7%, proving the effectiveness of the closed loop of "retrieval results → model optimization → improved retrieval accuracy";

[0333] The trajectory features optimized by surface fitting improved the retrieval accuracy of "trajectory-type requests" (such as crane hoisting paths) (from 78% before optimization to 89%).

[0334] IV. Summary of Collaborative Effects

[0335] 1. Video compression chain collaboration: Improved stable target detection in GMM provides reliable input for DeepSORT, while DeepSORT's trajectory data drives dynamic compression ratio adaptive adjustment. The combination of these three elements improves storage efficiency by 40% while achieving a critical information retention rate of 95%.

[0336] 1. Trajectory-retrieval chain collaboration: Surface fitting optimization of trajectory data improves metadata quality, providing more accurate training samples for knowledge distillation models. Combined with hierarchical indexing, it enables fast matching with a retrieval response time of <2 seconds.

[0337] 1. Closed-loop optimization and collaboration: User feedback feeds back into the model through the EWC algorithm, which continuously improves the retrieval accuracy as the system is used, thus solving the model adaptability problem caused by the dynamic changes in the ship scenario.

[0338] In summary, the algorithms in this solution form an organic whole through data transmission and feedback, which is significantly better than a single algorithm or traditional methods, and is suitable for complex ship monitoring scenarios.

[0339] In summary, several specific embodiments of the present invention have been disclosed. Without contradiction, the various embodiments can be freely combined to form new embodiments. That is, embodiments that are alternative solutions can be freely substituted for each other, but cannot be combined with each other; embodiments that are not alternative solutions can be combined with each other. These new embodiments are also part of the substantive content of the present invention.

[0340] The above embodiments describe several specific implementations of the present invention. However, those skilled in the art should understand that various changes or modifications can be made to these implementations without departing from the principles and essence of the present invention, but all such changes and modifications fall within the protection scope of the present invention.

Claims

1. A method for ship video recording summarization and intelligent retrieval based on edge nodes, characterized in that, include: S1: Deploy edge nodes in each monitoring area of ​​the ship, initialize the equipment parameters, collect ship video data, and store the raw video data in the temporary buffer of the edge nodes; S2: Edge nodes perform moving target detection on the original video recording in the temporary buffer area using an improved Gaussian mixture model to obtain the target mask; Based on the target mask, the improved DeepSORT algorithm is used to track moving targets and obtain the target ID and motion trajectory; The number of targets is counted based on the target ID. Combined with the time information of the motion trajectory, the effective frames are selected and recombined to generate a condensed video by a dynamic compression ratio adjustment algorithm based on target density. At the same time, metadata containing target ID, motion trajectory and timestamp is generated. The condensed video and metadata are stored in the long-term storage area, and the original recording in the temporary cache area is deleted. S3: Each edge node synchronizes the index information of condensed video and metadata through a publish-subscribe model. If missing data is detected locally, it requests transmission and storage from the publishing node to ensure that the long-term storage area of ​​each edge node contains the condensed video and metadata of the entire ship. S4: The central control node collects the synchronous metadata of each edge node to build a training dataset. The motion trajectory in the metadata is used to extract target features. A lightweight intelligent retrieval model is trained through the knowledge distillation algorithm and the model is deployed to each edge node. S5: Users input search requests through the central control system, and edge nodes call the lightweight intelligent search model. Combining motion trajectory time information in metadata with hierarchical indexing algorithms, the corresponding condensed video clips are matched in the local long-term storage area, and the search results are returned. S6: Collect user feedback on search results, incrementally optimize the lightweight intelligent search model through the elastic weight consolidation algorithm, and synchronize the optimized model to each edge node to form a closed loop.

2. The method for ship video recording summarization and intelligent retrieval based on edge nodes according to claim 1, characterized in that, In step S2, the dynamic compression ratio adjustment algorithm based on target density interacts with the improved DeepSORT algorithm. Specifically, the target ID output by the improved DeepSORT algorithm is used to count the number of targets in each frame, calculate the average target density within the time window, and determine the dynamic compression ratio based on the average target density. The higher the average target density, the smaller the compression ratio.

3. The method for ship video recording summarization and intelligent retrieval based on edge nodes according to claim 1, characterized in that, In step S4, the knowledge distillation algorithm includes constructing a teacher model and a student model. The teacher model is a high-precision YOLOv8 and BERT combination model, and the student model is a lightweight MobileNetV3 and simplified BERT combination model. During training, the output of the teacher model is used as a guide to optimize the student model by fusing classification loss and distillation loss, so that the student model can be adapted to the computing power of the edge nodes.

4. The method for ship video recording summarization and intelligent retrieval based on edge nodes according to claim 1, characterized in that, In step S5, the hierarchical indexing algorithm includes a first-level index and a second-level index. The first-level index divides the metadata into partitions according to the edge node ID and time interval. The second-level index indexes the target feature vector in the metadata in each partition through a KD tree. During retrieval, the candidate partition is first located through the first-level index, and then the target feature vector similar to the retrieval request is quickly matched through the second-level index.

5. The method for ship video recording summarization and intelligent retrieval based on edge nodes according to claim 1, characterized in that, In step S6, the elastic weight consolidation algorithm interacts with the user feedback in step S5, specifically: determining the importance of model parameters to historical tasks based on user feedback and generating a Fisher information matrix; during incremental training, limiting the variation of important parameters by adding a parameter protection term to the loss function to prevent the model from forgetting historical knowledge.

6. The method for ship video recording summarization and intelligent retrieval based on edge nodes according to claim 1, characterized in that, In step S3, the interaction process of the publish-subscribe mode is as follows: each edge node publishes the index information of the local condensed video and metadata to the message queue, other nodes subscribe to the message queue and compare the local index. If it is missing, they send a request containing the file identifier to the publishing node. The publishing node transmits the data through the breakpoint resume protocol. The receiving node updates the local storage after verifying the integrity.

7. The method for ship video recording summarization and intelligent retrieval based on edge nodes according to claim 1, characterized in that, In step S2, the improved Gaussian mixture model is optimized for ship scenarios. By dynamically adjusting the number of Gaussian components and the update rate, the background modeling adapts to scenarios of wave reflection and equipment vibration, and the target mask can accurately distinguish moving targets from the undulating background.

8. The method for ship video recording summarization and intelligent retrieval based on edge nodes according to claim 1, characterized in that, In step S5, the retrieval request includes a text description or a target feature image. The edge node converts the text description into a semantic vector and the target feature image into a visual feature vector through a lightweight intelligent retrieval model. Then, it calculates the similarity with the target feature vector in the metadata and filters out condensed video clips with similarity reaching a threshold.

9. The method for ship video recording summarization and intelligent retrieval based on edge nodes according to claim 1, characterized in that, In step S6, the triggering condition for the closed-loop optimization is: when the number of collected user feedback reaches a preset threshold, the elastic weight consolidation algorithm is started to incrementally optimize the model. The optimized model is updated to each edge node through the synchronization mechanism in step S3 until the retrieval accuracy meets the preset requirements.

10. A ship video recording summarization and intelligent retrieval system based on edge nodes, characterized in that, The edge node-based ship video recording summarization and intelligent retrieval method according to any one of claims 1 to 9, wherein the edge node-based ship video recording summarization and intelligent retrieval system comprises: The acquisition module is used to deploy edge nodes in various monitoring areas of the ship, acquire ship video data after initializing the equipment parameters, and store the raw video data in the temporary buffer of the edge nodes; The edge processing module is used to detect moving targets in the original video recordings in the temporary buffer at edge nodes using an improved Gaussian mixture model to obtain target masks. Based on the target masks, the module tracks the moving targets using an improved DeepSORT algorithm to obtain target IDs and motion trajectories. The module counts the number of targets based on their IDs and combines the time information of the motion trajectories. It then uses a dynamic compression ratio adjustment algorithm based on target density to filter valid frames and reassemble them to generate a compressed video. Simultaneously, it generates metadata containing target IDs, motion trajectories, and timestamps. The compressed video and metadata are stored in the long-term storage area, and the original video recordings in the temporary buffer are deleted. The synchronization module is used to synchronize the index information of condensed video and metadata on each edge node through a publish-subscribe model. If missing data is detected locally, it requests transmission and storage from the publishing node to ensure that the long-term storage area of ​​each edge node contains the condensed video and metadata of the entire ship. The training and deployment module is used to collect synchronous metadata from each edge node at the central control node to build a training dataset. The motion trajectory in the metadata is used to extract target features. A lightweight intelligent retrieval model is trained through a knowledge distillation algorithm and the model is deployed to each edge node. The feedback module allows users to input search requests through the central control system. Edge nodes call a lightweight intelligent search model, which combines motion trajectory time information in metadata with a hierarchical indexing algorithm to match the corresponding condensed video clips in the local long-term storage area and return the search results. The optimization module is used to collect user feedback on search results, incrementally optimize the lightweight intelligent search model through the elastic weight consolidation algorithm, and synchronize the optimized model to each edge node to form a closed loop.

Images