Ship navigation monitoring video super-resolution method and system based on hybrid attention

Abstract

The present invention discloses a hybrid attention-based super-resolution method and system for ship navigation monitoring video, which relates to the field of video processing technology. The method comprises: using two cameras with different precisions to capture and preprocess ship navigation video sequences, constructing a hybrid attention module for each of the two ship navigation video sequences, using the hybrid attention module to extract video frames, obtain key frames of high-precision video and low-precision video, and adaptively alternately superimpose them to generate a superimposed video sequence; constructing a super-resolution reconstruction network to enhance the superimposed video sequence, and outputting a super-resolution video sequence through adversarial training. The method utilizes the detailed information of the high-precision camera and the wide-area motion information of the low-precision camera to significantly improve the super-resolution quality of ship videos, and is suitable for high-precision monitoring needs in complex maritime environments.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and more specifically, to a ship navigation monitoring video super-resolution method and system based on hybrid attention. Background Art

[0002] Vessel navigation monitoring systems play a vital role in marine transportation, port management, maritime search and rescue, and other fields. With the rapid development of computer vision and video surveillance technologies, video-based vessel navigation monitoring has become a key tool for modern maritime regulation. However, in practical applications, due to limitations in camera hardware performance, transmission bandwidth, and environmental interference (such as fog, waves, and low light), captured vessel navigation videos often suffer from low resolution, high noise levels, and blurred details, severely impacting the accuracy of subsequent tasks such as target detection, tracking, and behavior analysis.

[0003] Traditional video super-resolution methods are primarily based on interpolation or reconstruction-based approaches, but these methods often rely on hand-crafted prior knowledge and struggle to adapt to complex ocean environments. In recent years, deep learning-based super-resolution techniques have made significant progress, particularly with the introduction of convolutional neural networks (CNNs) and generative adversarial networks (GANs), which have significantly improved super-resolution performance for both images and videos. Existing methods are limited by motion blur and complex background interference in ship navigation videos, as well as their difficulty meeting real-time requirements. In recent years, attention mechanisms have been introduced to super-resolution tasks, improving reconstruction performance by focusing on key areas. Hybrid attention combines channel attention, spatial attention, and temporal attention mechanisms to better model spatiotemporal dependencies and multi-scale features in videos. Therefore, combining deep learning and hybrid attention mechanisms to improve video resolution is a pressing issue. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a ship navigation monitoring video super-resolution method and system based on hybrid attention, which combines deep learning and hybrid attention mechanism to improve video resolution while enhancing the detailed features of ship targets and adapt to complex marine environments.

[0005] A first aspect of the present invention provides a ship navigation monitoring video super-resolution method based on hybrid attention, comprising the following steps:

[0006] Use two cameras with different precision to collect ship navigation video sequences and preprocess them to obtain two preprocessed ship navigation video sequences;

[0007] A hybrid attention module is constructed for each of the two ship navigation video sequences, and the hybrid attention module is used to extract high-precision video frames and low-precision video frames to obtain key frames of the high-precision video and the low-precision video;

[0008] Adaptively and alternately superimpose the key frames of the high-precision video and the low-precision video to generate a superimposed video sequence;

[0009] A super-resolution reconstruction network is constructed, the superimposed video sequence is used as a model input, the superimposed video sequence is enhanced, and a super-resolution video sequence is output through adversarial training.

[0010] In this solution, two cameras with different precision are used to capture the ship navigation video sequence and perform preprocessing, specifically:

[0011] Configuring a high-precision camera and a low-precision camera, performing hardware synchronization on the high-precision camera and the low-precision camera, and performing monitoring alignment based on timestamps to obtain two ship navigation video sequences respectively;

[0012] The SIFT feature detection algorithm is used to detect scale space extreme values ​​in two ship navigation video sequences, locate key points in the video frames, calculate the main direction of the key points, and generate SIFT feature descriptors. In addition, the ORB feature detection algorithm is used to quickly locate key points and generate BRIEF feature descriptors.

[0013] Generate feature point sets of high-precision video frames and low-precision video frames based on a mixture of SIFT feature descriptors and BRIEF feature descriptors, and perform SIFT matching on the feature point sets using Euclidean distance and ORB matching using Hamming distance respectively;

[0014] Obtain key point matching results and eliminate false matches, optimize key point matching pairs to achieve spatial alignment and eliminate perspective differences.

[0015] In this scheme, a hybrid attention module is constructed for the two ship navigation video sequences respectively, and the hybrid attention module of the high-precision video sequence is used to perform high-precision video frame extraction, specifically:

[0016] For high-precision video sequences, a hybrid attention module is constructed using spatial attention and semantic attention. The pre-processed ship navigation video sequence captured by the high-precision camera is used as the module input. The Laplacian gradient function is used to evaluate the clarity in the spatial attention branch of the hybrid attention module, and the spatial attention is configured according to the local clarity evaluation results of the video frame.

[0017] In the semantic attention branch of the hybrid attention module, a MobileNetV2-based encoder is used to encode and map the video frames into an embedding space to obtain single-frame visual features, a pre-trained classification network is constructed using the single-frame visual features, and class activation mapping is used to extract the visual semantics corresponding to the video frames;

[0018] Obtaining the mission objective of the ship navigation video, using global average pooling to obtain the relationship between the visual features of a single frame and the mission objective category in the class activation map, projecting the visual features of the single frame using the weights of the classification network output layer to obtain a class activation map with the mission objective category label, activating the class activation map using the Softmax function, and configuring the semantic attention of the video frame based on the obtained probability;

[0019] The spatial attention and semantic attention of the video frames are combined, and the video frames that meet the preset weight threshold are selected according to the combined attention weight and output as the key frames of the ship navigation video sequence captured by the high-precision camera.

[0020] In this solution, a hybrid attention module of low-precision video sequences is used to extract low-precision video frames, specifically:

[0021] For low-precision video sequences, a hybrid attention module is constructed using temporal attention and motion attention. The pre-processed ship navigation video sequence captured by the low-precision camera is used as the module input, and the optical flow vector of the task target in the video frame is calculated in the hybrid attention module.

[0022] In the temporal attention branch, the optical flow vector of each video frame is encoded and represented, and the GRU unit is used for temporal modeling to capture the long-term dependency of the optical flow vector sequence. The temporal attention weight is generated according to the hidden state of the GRU unit.

[0023] In the motion attention branch, the amplitude of each optical flow vector is calculated, and the average motion intensity of the video frame is obtained based on the amplitude of the optical flow vector. The main direction of the optical flow vector is also obtained, and the angle between the motion direction of all optical flow vectors in the video frame and the main direction is calculated to perform motion consistency evaluation and obtain the motion direction consistency evaluation result;

[0024] The motion attention weight of the video frame is generated by the average motion intensity and motion direction consistency evaluation results of the video frame. The temporal attention weight and motion attention weight of the video frame are combined, and the video frames that meet the preset weight threshold are screened and output as the key frames of the ship navigation video sequence captured by the low-precision camera.

[0025] In this solution, the key frames of high-precision video and low-precision video are adaptively and alternately superimposed to generate a superimposed video sequence, specifically:

[0026] Obtain key frames of high-precision video sequences and low-precision video sequences, use the motion offset between the key frames of the two video sequences to perform motion compensation alignment, and obtain the absolute value of the difference in motion features between the high-precision video key frames and the low-precision video key frames;

[0027] When the absolute value of the motion feature difference is greater than the preset threshold, the key frame with the larger motion feature value is selected; otherwise, the high-precision video key frame and the low-precision video key frame are selected according to the odd-even alternating strategy;

[0028] The superimposed video sequence is obtained to establish an inter-frame structural similarity index monitoring for visual consistency check. When the preset visual consistency standard is met, the superimposed video sequence is output.

[0029] In this solution, a super-resolution reconstruction network is constructed, the superimposed video sequence is used as the model input, the superimposed video sequence is enhanced, and a super-resolution video sequence is output, specifically:

[0030] The superimposed video sequences are grouped according to different time steps. The target frames are selected in each group based on the motion feature values. The spatiotemporal features are obtained by using 3D dense blocks with residual connections in each group. The spatiotemporal features within the group are integrated to obtain group-level features.

[0031] Convolve the group-level features of each group to obtain a single-channel feature map, connect the single-channel feature maps along the time axis, introduce a self-attention mechanism to use softmax to calculate the self-attention weight for each position across channels, and use the self-attention weight to perform element-wise multiplication of the group-level features of different groups at the same position;

[0032] The result of multiplying the cascaded component features and the self-attention weights is imported into the 3D dense block to integrate the local features of different groups. A 2D dense block is added on top of the 3D dense block to further fuse the features and generate an aggregated feature map.

[0033] The aggregated feature map is enhanced by hybrid attention using channel attention, spatial attention and temporal attention, and the enhanced aggregated feature map is fused using a feature pyramid structure. The fused features are upsampled to obtain a high-resolution image consistent with the video frame scale, and are connected along the time axis to obtain a super-resolution video sequence.

[0034] In this solution, adversarial training is introduced into the super-resolution reconstruction network, specifically:

[0035] A discriminator module is introduced into the super-resolution reconstruction network. The discriminator module is divided into a spatial discriminator branch and a temporal discriminator branch. The acquired high-resolution image is imported into the spatial discriminator for feature recognition to determine whether the input high-resolution image is a real high-resolution frame or a generated super-resolution frame.

[0036] The acquired super-resolution video sequence is imported into the temporal discriminator for feature recognition to determine whether the input super-resolution video sequence has real temporal dynamic characteristics;

[0037] Obtain the discriminant matrix output by the spatial discriminator branch and the temporal discriminator branch, and guide the training of the super-resolution reconstruction network according to the discriminant matrix according to the adversarial mechanism. After iterative training, the training is terminated when the discriminator module can no longer distinguish between the generated super-resolution video sequence and the real super-resolution video sequence, and the trained super-resolution reconstruction network is obtained.

[0038] The second aspect of the present invention provides a ship navigation monitoring video super-resolution system based on hybrid attention, including a data acquisition and preprocessing unit, a hybrid attention frame extraction unit, a key frame superposition unit, a super-resolution reconstruction unit and a post-processing optimization unit;

[0039] The data acquisition and preprocessing unit uses two cameras with different precisions to acquire and preprocess the ship navigation video sequence to obtain two preprocessed ship navigation video sequences;

[0040] The hybrid attention frame extraction unit constructs a hybrid attention module for the two ship navigation video sequences respectively, and uses the hybrid attention module to perform high-precision video frame extraction and low-precision video frame extraction to obtain the key frame with the most information;

[0041] The key frame superposition unit aligns the key frames of the high-precision video and the low-precision video, and performs adaptive alternating superposition to generate a superimposed video sequence;

[0042] The super-resolution reconstruction unit constructs a super-resolution reconstruction network, uses a hybrid attention mechanism to enhance the superimposed video sequence, and reconstructs the super-resolution video sequence through adversarial training;

[0043] The post-processing optimization unit performs post-processing optimization on the super-resolution result of the ship navigation video to enhance edges and reduce inter-frame jitter, and lightweights the super-resolution reconstruction network to achieve real-time super-resolution.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] This method significantly improves the super-resolution quality of ship videos by combining dual-camera data with a hybrid attention mechanism, significantly enhancing the efficiency of ship video analysis while maintaining real-time performance. By combining deep learning and a hybrid attention mechanism, this method is expected to enhance the detailed features of ship targets while improving video resolution, adapt to complex marine environments, and provide a high-quality data foundation for subsequent ship detection, identification, and behavior analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments or exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or exemplary descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained according to these drawings without paying any creative work.

[0047] Figure 1 The flowchart of the ship navigation monitoring video super-resolution method based on hybrid attention is shown;

[0048] Figure 2 A flow chart showing the alternating superposition of key frames of high-precision video and low-precision video is shown;

[0049] Figure 3 A flowchart of constructing a super-resolution reconstruction network for video super-resolution is shown;

[0050] Figure 4 The block diagram of the ship navigation monitoring video super-resolution system based on hybrid attention is shown. DETAILED DESCRIPTION

[0051] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0053] Figure 1 A flow chart of the ship navigation monitoring video super-resolution method based on hybrid attention is shown.

[0054] like Figure 1 As shown, in a first embodiment of the present invention, a ship navigation monitoring video super-resolution method based on hybrid attention is provided, comprising:

[0055] S102, using two cameras with different precisions to capture and preprocess ship navigation video sequences to obtain two preprocessed ship navigation video sequences;

[0056] S104, constructing a hybrid attention module for each of the two ship navigation video sequences, and using the hybrid attention module to extract high-precision video frames and low-precision video frames to obtain key frames of the high-precision video and the low-precision video;

[0057] S106, adaptively and alternately superimposing key frames of the high-precision video and the low-precision video to generate a superimposed video sequence;

[0058] S108, constructing a super-resolution reconstruction network, taking the superimposed video sequence as a model input, enhancing the superimposed video sequence, and outputting a super-resolution video sequence through adversarial training.

[0059] It should be noted that a high-precision camera and a low-precision camera are configured. The high-precision camera provides clear but limited detail information, while the low-precision camera provides smoother but blurred motion information. The high-precision and low-precision cameras are synchronized by hardware and aligned based on timestamps to obtain two video sequences of ship navigation. The SIFT feature detection algorithm is used to extract features from the two video sequences of ship navigation. The Gaussian difference pyramid is used to detect scale space extreme values, eliminate low-contrast and edge-responsive points, locate key points in the video frames, calculate the main directions of the key points, and generate SIFT feature descriptors. In addition, the ORB feature detection algorithm is used for feature extraction, and FAST corner detection is used to quickly locate key points and generate the BRIEF feature descriptor. A hybrid of the SIFT and BRIEF feature descriptors generates feature point sets for high-precision and low-precision video frames. These feature point sets are matched using Euclidean distance for SIFT and Hamming distance for ORB, respectively. KNN is used to filter nearest neighbor matching pairs in SIFT matching, and cross-validation is used in ORB matching to improve matching accuracy. Key point matching results are obtained and false matches are eliminated. The average error of matching point pairs is calculated, and key point matching pairs are optimized to achieve spatial alignment and eliminate perspective differences. When the error is less than a preset threshold, alignment is determined to have failed, triggering rematching. If the initial alignment fails, the image is downsampled and rematched. By combining the accuracy of SIFT and the speed of ORB, performance and efficiency are balanced, improving the quality of video preprocessing. Guided filtering is also used during the preprocessing process to reduce wave interference.

[0060] It should be noted that hybrid attention modules were constructed for the two ship navigation video sequences. For the high-precision video sequence, a hybrid attention module was constructed using spatial attention and semantic attention. The preprocessed ship navigation video sequence captured by the high-precision camera was used as the module input. The Laplacian gradient function was used for clarity evaluation in the spatial attention branch of the hybrid attention module. The spatial attention was configured based on the local clarity evaluation results of the video frames, and the high-sharpness video frames were selected based on the spatial attention.

[0061] In the semantic attention branch of the hybrid attention module, a MobileNetV2-based encoder is used to encode video frames and map them to an embedding space, obtaining single-frame visual features. These single-frame visual features are then used to construct a pre-trained classification network. Preferably, the pre-trained classification network can be constructed using a lightweight CNN. Class activation mapping (CMAP) is used to extract the visual semantics corresponding to the video frames. The CMAP utilizes a global average pooling (GAP) layer to capture the relationship between feature maps and different classes, generating a visual activation map. The task target (e.g., a ship target) of the ship navigation video is obtained. The CMAP utilizes global average pooling to obtain the relationship between single-frame visual features and the task target category. The single-frame visual features are projected using the weights of the classification network output layer to obtain a class activation map with the task target category label. The CMAP is activated using a softmax function. Semantic attention is then assigned to the video frames based on the obtained probabilities, prioritizing frames with a high ship profile. The spatial and semantic attention of the video frames are combined, and based on the combined attention weights, video frames that meet a preset weight threshold are selected and output as keyframes of the ship navigation video sequence captured by a high-precision camera, providing high-quality details.

[0062] For low-precision video sequences, a hybrid attention module is constructed using temporal attention and motion attention. The preprocessed ship navigation video sequence captured by the low-precision camera is used as the module input. The Pyramidal Lucas-Kanade algorithm is used in the hybrid attention module to perform optical flow estimation to construct the optical flow field, and the optical flow vector of the task target in the video frame is calculated. In the temporal attention branch, the optical flow vector of each video frame is encoded and represented, and the GRU unit is used for temporal modeling to capture the long-term dependency of the optical flow vector sequence. The temporal attention weight is generated according to the hidden state of the GRU unit. The generated attention weight can accurately reflect the key motion areas in the video.

[0063] In the motion attention branch, for each optical flow vector Calculate its amplitude , obtain the average motion intensity of the video frame according to the amplitude of the optical flow vector , is the number of valid optical flow vectors, and obtains the main direction of the optical flow vector The main direction represents the overall motion direction of the ship. The angle between the motion direction of all optical flow vectors in the video frame and the main direction is calculated. The number of vectors with an angle greater than a preset angle is obtained. The ratio of the obtained vector number to the effective optical flow vector data is calculated to perform motion consistency evaluation and obtain the motion direction consistency evaluation result. The motion attention weight of the video frame is generated by evaluating the average motion intensity and motion direction consistency of the video frame. , 、 To adjust the parameters, the more intense the movement and the more consistent the direction, the higher the corresponding motion weight. The temporal attention weight and motion attention weight of the video frame are combined, and the video frames that meet the preset weight threshold are screened as the key frames of the ship navigation video sequence captured by the low-precision camera for output. Frames with significant motion (such as ship turning and acceleration) are selected to provide motion compensation information.

[0064] Figure 2 A flow chart showing the alternate superposition of key frames of high-precision video and low-precision video is shown.

[0065] According to an embodiment of the present invention, key frames of high-precision video and low-precision video are adaptively and alternately superimposed to generate a superimposed video sequence, specifically as follows:

[0066] S202, obtaining key frames of a high-precision video sequence and a low-precision video sequence, performing motion compensation alignment using a motion offset between the key frames of the two video sequences, and obtaining an absolute value of a motion feature difference between the high-precision video key frames and the low-precision video key frames;

[0067] S204, when the absolute value of the motion feature difference is greater than a preset threshold, selecting a key frame with a larger motion feature value; otherwise, selecting a high-precision video key frame and a low-precision video key frame according to an odd-even alternating strategy;

[0068] S206 , obtaining a superimposed video sequence and establishing an inter-frame structural similarity index monitoring to perform a visual consistency check. When a preset visual consistency standard is met, the superimposed video sequence is output.

[0069] It should be noted that the motion offset between the key frames of the high-precision video sequence and the low-precision video sequence is estimated by the optical flow method, and warping alignment is performed. The motion scene is determined according to the absolute value threshold of the motion feature difference. The threshold can be adjusted dynamically and appropriately lowered under low-light conditions. When violent motion is detected, the key frame with a larger motion feature value is selected. For example, when a ship suddenly turns, the low-precision camera can better capture fast motion and automatically select the low-precision video key frame and perform anti-blurring processing. When the ship sails steadily, the high-precision camera provides a clearer picture and gives priority to the high-precision video key frame. In the absence of significant motion differences, the high-precision video sequence and the low-precision video sequence are alternately selected in a fixed order. The odd-even alternation strategy is used to ensure that the key frames of the high-precision video sequence and the low-precision video sequence appear strictly alternately.

[0070] Figure 3 A flowchart of constructing a super-resolution reconstruction network for video super-resolution is shown.

[0071] According to an embodiment of the present invention, a super-resolution reconstruction network is constructed, the superimposed video sequence is used as a model input, the superimposed video sequence is enhanced, and a super-resolution video sequence is output, specifically:

[0072] S302, feature grouping the superimposed video sequence according to different time steps, selecting target frames in each group according to motion feature values, obtaining spatiotemporal features from 3D dense blocks using residual connections in each group, and integrating the spatiotemporal features within the group to obtain group-level features;

[0073] S304: Convolve the group-level features of each group to obtain a single-channel feature map, connect the single-channel feature maps along the time axis, introduce a self-attention mechanism to calculate the self-attention weight using softmax for each position across channels, and use the self-attention weight to perform element-wise multiplication of the group-level features of different groups at the same position;

[0074] S306: The result of multiplying the cascaded component features and the self-attention weight is imported into the 3D dense block to integrate the local features of different groups, and a 2D dense block is added on top of the 3D dense block to further fuse the features and generate an aggregated feature map;

[0075] S308, using channel attention, spatial attention and temporal attention to perform mixed attention enhancement on the aggregated feature map, using a feature pyramid structure to perform feature fusion on the enhanced aggregated feature map, upsampling the fused features to obtain a high-resolution image consistent with the video frame scale, and connecting along the time axis to obtain a super-resolution video sequence.

[0076] It should be noted that a super-resolution reconstruction network is constructed, and grouping is performed in the reconstruction of the superimposed video sequence and group attention is introduced to align and integrate the information of video frames with different degrees of motion. Feature grouping of the superimposed video sequence provides different complementary information for the recovery of the target frame. The attention mechanism is used to adaptively utilize the complementary information to restore the missing details of the target frame, thereby enhancing the performance of video super-resolution. After grouping, the self-attention mechanism is used to extract the temporal features of the group-level features to obtain the contextual information of the group. When the area of ​​a certain group is occluded, complementary information is extracted from the remaining groups to restore the unclear local details, thereby achieving mutual complementation. Each layer of the 3D dense block inputs the concatenation of feature maps from all previous layers, capturing both spatial details and temporal motion. Multi-scale feature reuse is achieved through dense connections. A 2D dense block is added on top of the 3D dense block to further fuse features, and the information of each group is deeply integrated to generate an aggregated feature map.

[0077] The aggregated feature map is enhanced using a hybrid attention mechanism using channel attention, spatial attention, and temporal attention. Channel attention dynamically adjusts the importance weights of feature channels, enhancing characteristic vessel channels (such as specific color channels like the hull number and waterline), while suppressing interfering channels like wave reflections. Spatial attention focuses on key spatial regions of the image, automatically locking onto the ship's region of interest, enhancing the hull's edges, and suppressing irrelevant sea surface areas. Temporal attention models inter-frame motion dependencies to enhance motion-significant frames (such as those during turns and acceleration), suppress the contribution of blurred frames, and improve track continuity. While maintaining real-time processing performance, the hybrid attention mechanism is used to provide targeted enhancement of key vessel features (such as the hull number and hull markings), while maintaining stable output due to spatiotemporal consistency constraints.

[0078] It should be noted that a discriminator module is introduced into the super-resolution reconstruction network. The discriminator module consists of a spatial discriminator branch and a temporal discriminator branch. The acquired high-resolution image is fed into the spatial discriminator, where it performs downsampling without pooling through strided convolution. This discriminator performs feature discrimination to determine whether the input high-resolution image is a true high-resolution frame or a generated super-resolution frame. Adversarial training is used to force the superimposed video reconstruction network to produce more realistic texture details, particularly improving high-frequency content such as ship numbers and flags. The acquired super-resolution video sequence is fed into the temporal discriminator, where a 3D convolutional neural network is used for feature discrimination to determine whether the input super-resolution video sequence exhibits realistic temporal dynamic characteristics. Adversarial training is also used to identify inter-frame jitter and inconsistent motion blur in the generated video. The discriminant matrix output by the spatial and temporal discriminator branches is obtained. This discriminant matrix is ​​then used to guide the training of the super-resolution reconstruction network using an adversarial mechanism. After iterative training, the discriminator module is unable to distinguish between the generated super-resolution video sequence and the true super-resolution video sequence, resulting in the trained super-resolution reconstruction network.

[0079] It should be noted that the super-resolution video sequences output by the super-resolution reconstruction network are optimized to improve real-time performance. For example, Unsharp Masking is used to enhance edges, and Kalman filtering is employed to reduce inter-frame jitter. Furthermore, knowledge distillation or channel pruning is used to lightweight the super-resolution reconstruction network to reduce computational complexity and achieve real-time super-resolution.

[0080] The super-resolution video sequence output by the super-resolution reconstruction network is used to obtain the ship detection area using the YOLOv5 network for ship identification. The navigation trajectory of the target ship is generated based on the ship identification result, and the navigation trajectory is spatiotemporally encoded. The prediction head performs track prediction based on the spatiotemporal encoding. The track prediction sequence within a preset time is compared with the historical berthing trajectory sequence of the ship in the port database. The dynamic time warping algorithm is used to obtain the time warping distance between the sequences. The trajectory similarity is characterized by the time warping distance. When the trajectory similarity is greater than the preset similarity threshold, the target ship is marked as berthing, and the mark is updated in real time to achieve real-time berthing analysis.

[0081] Figure 4 The block diagram of the ship navigation monitoring video super-resolution system based on hybrid attention is shown.

[0082] The second embodiment of the present invention provides a ship navigation monitoring video super-resolution system 4 based on hybrid attention, which includes a data acquisition and preprocessing unit 401, a hybrid attention frame extraction unit 402, a key frame superposition unit 403, a super-resolution reconstruction unit 404 and a post-processing optimization unit 405;

[0083] The data acquisition and preprocessing unit uses two cameras with different precisions to acquire and preprocess the ship navigation video sequence to obtain two preprocessed ship navigation video sequences;

[0084] The hybrid attention frame extraction unit constructs a hybrid attention module for the two ship navigation video sequences respectively, and uses the hybrid attention module to perform high-precision video frame extraction and low-precision video frame extraction to obtain the key frame with the most information;

[0085] The key frame superposition unit aligns the key frames of the high-precision video and the low-precision video, and performs adaptive alternating superposition to generate a superimposed video sequence;

[0086] The super-resolution reconstruction unit constructs a super-resolution reconstruction network, uses a hybrid attention mechanism to enhance the superimposed video sequence, and reconstructs the super-resolution video sequence through adversarial training;

[0087] The post-processing optimization unit performs post-processing optimization on the super-resolution result of the ship navigation video to enhance edges and reduce inter-frame jitter, and lightweights the super-resolution reconstruction network to achieve real-time super-resolution.

[0088] The third embodiment of the present invention provides a computer-readable storage medium, which includes a ship navigation monitoring video super-resolution method program based on mixed attention. When the ship navigation monitoring video super-resolution method program based on mixed attention is executed by a processor, it implements the steps of the ship navigation monitoring video super-resolution method based on mixed attention.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0090] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0091] Alternatively, if the integrated units described above are implemented as software functional units and sold or used as standalone products, they can also be stored on a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product, stored on a storage medium, includes instructions for enabling a computer device (such as a personal computer, server, or network device) to execute all or part of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as removable storage devices, ROM, RAM, magnetic disks, or optical disks.

[0092] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. A ship navigation monitoring video super-resolution method based on hybrid attention, characterized in that: The following steps are involved: Use two cameras with different precision to collect ship navigation video sequences and preprocess them to obtain two preprocessed ship navigation video sequences; A hybrid attention module is constructed for each of the two ship navigation video sequences, and the hybrid attention module is used to extract high-precision video frames and low-precision video frames to obtain key frames of the high-precision video and the low-precision video; Adaptively and alternately superimpose the key frames of the high-precision video and the low-precision video to generate a superimposed video sequence; Constructing a super-resolution reconstruction network, taking the superimposed video sequence as a model input, enhancing the superimposed video sequence, and outputting a super-resolution video sequence through adversarial training; Construct a super-resolution reconstruction network, take the superimposed video sequence as the model input, enhance the superimposed video sequence, and output a super-resolution video sequence, specifically: The superimposed video sequences are grouped according to different time steps. The target frames are selected in each group based on the motion feature values. The spatiotemporal features are obtained by using 3D dense blocks with residual connections in each group. The spatiotemporal features within the group are integrated to obtain group-level features. Convolve the group-level features of each group to obtain a single-channel feature map, connect the single-channel feature maps along the time axis, introduce a self-attention mechanism to use softmax to calculate the self-attention weight for each position across channels, and use the self-attention weight to perform element-wise multiplication of the group-level features of different groups at the same position; The result of multiplying the cascaded component features and the self-attention weights is imported into the 3D dense block to integrate the local features of different groups. A 2D dense block is added on top of the 3D dense block to further fuse the features and generate an aggregated feature map. The aggregated feature map is enhanced by hybrid attention using channel attention, spatial attention and temporal attention, and the enhanced aggregated feature map is fused using a feature pyramid structure. The fused features are upsampled to obtain a high-resolution image consistent with the video frame scale, and are connected along the time axis to obtain a super-resolution video sequence.

2. The ship navigation monitoring video super-resolution method based on hybrid attention according to claim 1 is characterized in that: Two cameras with different precision are used to collect the ship navigation video sequence and preprocess it, specifically: Configuring a high-precision camera and a low-precision camera, performing hardware synchronization on the high-precision camera and the low-precision camera, and performing monitoring alignment based on timestamps to obtain two ship navigation video sequences respectively; The SIFT feature detection algorithm is used to detect scale space extreme values ​​in two ship navigation video sequences, locate key points in the video frames, calculate the main direction of the key points, and generate SIFT feature descriptors. In addition, the ORB feature detection algorithm is used to quickly locate key points and generate BRIEF feature descriptors. Generate feature point sets of high-precision video frames and low-precision video frames based on a mixture of SIFT feature descriptors and BRIEF feature descriptors, and perform SIFT matching on the feature point sets using Euclidean distance and ORB matching using Hamming distance respectively; Obtain key point matching results and eliminate false matches, optimize key point matching pairs to achieve spatial alignment and eliminate perspective differences.

3. The ship navigation monitoring video super-resolution method based on hybrid attention according to claim 1 is characterized in that: A hybrid attention module is constructed for the two ship navigation video sequences respectively, and the hybrid attention module of the high-precision video sequence is used to perform high-precision video frame extraction, specifically: For high-precision video sequences, a hybrid attention module is constructed using spatial attention and semantic attention. The pre-processed ship navigation video sequence captured by the high-precision camera is used as the module input. The Laplacian gradient function is used to evaluate the clarity in the spatial attention branch of the hybrid attention module, and the spatial attention is configured according to the local clarity evaluation results of the video frame. In the semantic attention branch of the hybrid attention module, a MobileNetV2-based encoder is used to encode and map the video frames into an embedding space to obtain single-frame visual features, a pre-trained classification network is constructed using the single-frame visual features, and class activation mapping is used to extract the visual semantics corresponding to the video frames; Obtaining the mission objective of the ship navigation video, using global average pooling to obtain the relationship between the visual features of a single frame and the mission objective category in the class activation map, projecting the visual features of the single frame using the weights of the classification network output layer to obtain a class activation map with the mission objective category label, activating the class activation map using the Softmax function, and configuring the semantic attention of the video frame based on the obtained probability; The spatial attention and semantic attention of the video frames are combined, and the video frames that meet the preset weight threshold are selected according to the combined attention weight and output as the key frames of the ship navigation video sequence captured by the high-precision camera.

4. The ship navigation monitoring video super-resolution method based on hybrid attention according to claim 3 is characterized in that: Use the hybrid attention module of low-precision video sequences to perform low-precision video frame extraction, specifically: For low-precision video sequences, a hybrid attention module is constructed using temporal attention and motion attention. The pre-processed ship navigation video sequence captured by the low-precision camera is used as the module input, and the optical flow vector of the task target in the video frame is calculated in the hybrid attention module. In the temporal attention branch, the optical flow vector of each video frame is encoded and represented, and the GRU unit is used for temporal modeling to capture the long-term dependency of the optical flow vector sequence. The temporal attention weight is generated according to the hidden state of the GRU unit. In the motion attention branch, the amplitude of each optical flow vector is calculated, and the average motion intensity of the video frame is obtained based on the amplitude of the optical flow vector. The main direction of the optical flow vector is also obtained, and the angle between the motion direction of all optical flow vectors in the video frame and the main direction is calculated to perform motion consistency evaluation and obtain the motion direction consistency evaluation result; The motion attention weight of the video frame is generated by the average motion intensity and motion direction consistency evaluation results of the video frame. The temporal attention weight and motion attention weight of the video frame are combined, and the video frames that meet the preset weight threshold are screened and output as the key frames of the ship navigation video sequence captured by the low-precision camera.

5. The ship navigation monitoring video super-resolution method based on hybrid attention according to claim 1 is characterized in that: The key frames of the high-precision video and the low-precision video are adaptively and alternately superimposed to generate a superimposed video sequence, specifically: Obtain key frames of high-precision video sequences and low-precision video sequences, use the motion offset between the key frames of the two video sequences to perform motion compensation alignment, and obtain the absolute value of the difference in motion features between the high-precision video key frames and the low-precision video key frames; When the absolute value of the motion feature difference is greater than the preset threshold, the key frame with the larger motion feature value is selected; otherwise, the high-precision video key frame and the low-precision video key frame are selected according to the odd-even alternating strategy; The superimposed video sequence is obtained to establish an inter-frame structural similarity index monitoring for visual consistency check. When the preset visual consistency standard is met, the superimposed video sequence is output.

6. The ship navigation monitoring video super-resolution method based on hybrid attention according to claim 1 is characterized in that: Adversarial training is introduced into the super-resolution reconstruction network, specifically: A discriminator module is introduced into the super-resolution reconstruction network. The discriminator module is divided into a spatial discriminator branch and a temporal discriminator branch. The acquired high-resolution image is imported into the spatial discriminator for feature recognition to determine whether the input high-resolution image is a real high-resolution frame or a generated super-resolution frame. The acquired super-resolution video sequence is imported into the temporal discriminator for feature recognition to determine whether the input super-resolution video sequence has real temporal dynamic characteristics; Obtain the discriminant matrix output by the spatial discriminator branch and the temporal discriminator branch, and guide the training of the super-resolution reconstruction network according to the discriminant matrix according to the adversarial mechanism. After iterative training, the training is terminated when the discriminator module can no longer distinguish between the generated super-resolution video sequence and the real super-resolution video sequence, and the trained super-resolution reconstruction network is obtained.

7. A ship navigation monitoring video super-resolution system based on mixed attention, characterized in that: Implementing the ship navigation monitoring video super-resolution method based on hybrid attention as described in any one of claims 1 to 6, comprising a data acquisition and preprocessing unit, a hybrid attention frame extraction unit, a key frame superposition unit, a super-resolution reconstruction unit and a post-processing optimization unit; The data acquisition and preprocessing unit uses two cameras with different precisions to acquire and preprocess the ship navigation video sequence to obtain two preprocessed ship navigation video sequences; The hybrid attention frame extraction unit constructs a hybrid attention module for the two ship navigation video sequences respectively, and uses the hybrid attention module to perform high-precision video frame extraction and low-precision video frame extraction to obtain the key frame with the most information; The key frame superposition unit aligns the key frames of the high-precision video and the low-precision video, and performs adaptive alternating superposition to generate a superimposed video sequence; The super-resolution reconstruction unit constructs a super-resolution reconstruction network, uses a hybrid attention mechanism to enhance the superimposed video sequence, and reconstructs the super-resolution video sequence through adversarial training; The post-processing optimization unit performs post-processing optimization on the super-resolution result of the ship navigation video to enhance edges and reduce inter-frame jitter, and lightweights the super-resolution reconstruction network to achieve real-time super-resolution.

Images