An underwater wireless optical semantic communication method and system based on image and video transmission

Abstract

This invention discloses an underwater wireless optical semantic communication method and system based on image and video transmission. Implemented using an optical transmitter and an optical receiver, the method includes: acquiring a video frame sequence and performing Gaussian denoising; constructing a discrete histogram for the denoised video frame sequence by statistically analyzing grayscale distribution, and extracting keyframes through similarity analysis; extracting basic features from the keyframes, and generating semantic segmentation maps of the keyframes using pyramid pooling and attention mechanisms, and forming a reduced-data-volume segmentation map group through difference calculation, inputting it into the optical transmitter to convert it into an optical signal, and transmitting it to the underwater medium; converting and superimposing the optical signal received by the optical receiver after transmission through the underwater medium to restore the semantic segmentation map group; recovering the semantic segmentation map group based on an enhanced recurrent adversarial generative network to obtain a visual image; and generating a complete video frame sequence based on the visual image of the corresponding keyframes through a frame interpolation network, thereby achieving video frame recovery.

Description

Technical Field

[0001] This invention belongs to the field of underwater wireless optical communication technology, and particularly relates to an underwater wireless optical semantic communication method and system based on image and video transmission. Background Technology

[0002] With the deepening development of applications such as marine resource development and seabed observation network construction, underwater video transmission systems are facing an ever-increasing demand for high-bandwidth, low-latency communication. Typical high-definition video (such as 4K / 30fps) raw data rates can reach the 6Gbps level, posing a severe challenge to underwater communication systems.

[0003] Current underwater communication solutions mainly include wired and wireless transmission. Wired transmission relies on submarine optical or electrical cables, which, while providing a stable high-speed transmission channel, suffers from inherent drawbacks such as high deployment and maintenance costs and poor flexibility. Its applicability is severely limited, especially in deep-sea or complex seabed terrain areas. In the field of wireless communication, existing technologies mainly include radio frequency (RF) communication, acoustic communication, and optical communication. RF communication is significantly affected by the conductivity of seawater; the attenuation coefficient of electromagnetic waves in seawater is as high as 100 dB / m, and the effective transmission distance is typically no more than 10 meters. While acoustic communication can achieve kilometer-level transmission, its usable bandwidth is usually limited to tens to hundreds of kbps, and it suffers from second-level transmission delays, completely failing to meet the requirements for real-time high-definition video transmission.

[0004] In comparison, Underwater Wireless Optical Communication (UWOC) has significant advantages: its theoretical bandwidth can reach the Gbps level, transmission latency is only in the milliseconds level, and the equipment is small in size and low in power consumption. However, in practical applications, UWOC systems face two major technical bottlenecks: First, seawater channels suffer from severe scattering and turbulence effects, leading to rapid attenuation of the optical signal; second, the amount of raw video data far exceeds the actual carrying capacity of the UWOC system. For example, in typical underwater environments, the effective transmission bandwidth of a UWOC system is usually limited to the range of 100Mbps-1Gbps, which is orders of magnitude different from the amount of raw video data.

[0005] Semantic communication technology, which has emerged in recent years, offers a new approach to solving this problem. Unlike traditional communication methods, semantic communication extracts and transmits semantic features of information rather than raw data, which can significantly reduce the amount of data transmitted. Research shows that deep learning-based semantic coding can compress video data by more than 90% while maintaining the integrity of key information.

[0006] This invention proposes an underwater video transmission system based on semantic compression and optical communication. The system analyzes acquired video data and utilizes algorithms for keyframe extraction and semantic encoding, thereby achieving efficient data compression. The processed data is transmitted via underwater optical communication technology, where semantic decoding and frame interpolation are performed at the receiving end to ultimately reconstruct the video information. This processing method significantly reduces bandwidth requirements and effectively solves the problem of transmission efficiency limitations caused by channel characteristics in existing underwater wireless video transmission. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and to propose an underwater wireless optical semantic communication method based on image and video transmission. This invention also discloses an underwater wireless optical semantic communication system based on image and video transmission.

[0008] In view of this, the present invention proposes an underwater wireless optical semantic communication method based on image and video transmission, implemented using an optical transmitter and an optical receiver, the method comprising:

[0009] Step 1: Obtain the video frame sequence and perform Gaussian denoising processing;

[0010] Step 2: For the denoised video frame sequence, construct a discrete histogram by statistically analyzing the gray level distribution, and extract keyframes through similarity analysis;

[0011] Step 3: Extract basic features from keyframes, combine them with pyramid pooling and attention mechanisms to generate semantic segmentation images of keyframes, and use difference calculation to form semantic segmentation image groups with reduced data volume. Input the images into the optical transmitter to convert them into optical signals and then transmit them to the underwater medium.

[0012] Step 4: Convert and superimpose the optical signal received by the optical receiver after transmission through the underwater medium into a semantic segmentation graph group;

[0013] Step 5: Restore the semantic segmentation graph group based on the enhanced recurrent adversarial generative network to obtain the visual image;

[0014] Step 6: Based on the visual images of the corresponding keyframes, generate a complete video frame sequence through a frame interpolation network; thereby realizing the restoration of video frames.

[0015] Preferably, step 2 includes:

[0016] Step 2-1: Convert the denoised video frame sequence into a grayscale image v = {v1, v2, ...}, and designate the first frame v1 as the keyframe I1;

[0017] Step 2-2: Statistically analyze the grayscale distribution of video frames, construct a discrete histogram, and then convert the histogram frequencies into a probability distribution:

[0018]

[0019] Where i represents the i-th frame in the time series, k is the interval index of the histogram of the i-th frame, indicating the position of the current interval, N is the number of intervals in the histogram of the i-th frame, representing the total number of intervals into which the histogram is divided, and H... i [k] is the frequency of the k-th interval in the histogram of the i-th frame, P i [k] represents the probability in the k-th interval of the histogram of the i-th frame;

[0020] Steps 2-3: Calculate the sum of the square roots of the product of the probability distributions of frame A and frame B using the following formula to obtain the corresponding similarity BC:

[0021]

[0022] Among them, P A [k],P B [k] represents the probability of the histogram of frame A and the histogram of frame B in the kth interval; the BC value ranges from [0,1], where 0 indicates no overlap and 1 indicates that the two histograms completely overlap.

[0023] Step 2-4: Perform similarity analysis. When BC is less than the decision threshold, it is judged as a significant difference. Define this frame as a keyframe and go to step 2-3. Continue until the grayscale images v = {v1, v2, ...} have been traversed. Finally, output the RGB image sequence I = {I1, I2, ...} corresponding to the extracted keyframes.

[0024] Preferably, step 3 includes:

[0025] Basic feature extraction is performed using a ResNet-50 backbone network, and the output is... The feature map; where H and W are the height and width of the original image of the video frame, respectively;

[0026] The feature enhancement module consists of a pyramid pooling module and an attention mechanism module. The pyramid pooling module... The feature map undergoes multi-scale pooling: it is divided into 1×1, 2×2, 3×3, and 6×6 sub-regions, and average pooling is performed on each sub-region to capture contextual information of different ranges. Each sub-region is reduced to 512 channels by 1×1 convolution, then restored to its original size by bilinear interpolation, and finally concatenated with the original feature map to form a new feature map. The attention mechanism module includes a channel attention mechanism and a spatial attention mechanism. The channel attention mechanism generates channel weight vectors through global average pooling and dynamically adjusts channel importance by combining a sigmoid activation function. The spatial attention mechanism generates a spatial weight matrix through local max pooling and average pooling, fuses spatial information using 7×7 convolution, and outputs weighted high discriminative features.

[0027] Feature fusion and channel adjustment are performed sequentially. The feature map is upsampled to the original input resolution H×W×C by bilinear interpolation to obtain a semantic segmentation image, where C is the number of target categories.

[0028] The semantic segmentation images obtained from each keyframe are combined to form a semantic segmentation graph group G.

[0029] Preferably, step 3 further includes:

[0030] Set the first segmentation graph in the semantic segmentation graph group G as the group baseline segmentation graph;

[0031] Calculate the difference between the two semantic segmentation maps sequentially;

[0032] The baseline segmentation map and the subsequent difference are combined to form a segmentation map group S with reduced data volume. This group is then input into an optical transmitter, converted into an optical signal, and transmitted to the underwater medium.

[0033] Preferably, step 4 includes: converting the optical signal received by the optical receiver into a segmentation map group S, and performing superposition and restoration processing sequentially to obtain the restored semantic segmentation map group G.

[0034] Preferably, the enhanced recurrent adversarial generative network in step 5 includes a generator that adopts an improved U-Net architecture. By combining residual blocks and skip connections, it enhances feature extraction capabilities at multiple scales. In the encoder part, it uses multiple convolutional layers for feature extraction and skip connections to pass low-level feature information to higher-resolution layers so that the feature information extracted by the encoder can be reused during the decoding process.

[0035] Preferably, the enhanced recurrent adversarial generative network in step 5 further includes a discriminator during the training phase. The discriminator adopts the PatchGAN architecture and is improved by introducing multiple dilated convolutions to enhance its ability to capture global information.

[0036] The loss function during the training phase is:

[0037] Loss = Loss GAN +λ·Loss cycle +Loss identity

[0038] Among them, Loss GAN This represents the adversarial loss, Loss cycle Loss represents the loss due to cycle consistency. identity Let λ represent the identity loss, and λ be the weighting coefficient of the cycle consistency loss.

[0039] Preferably, the frame interpolation network in step 6 includes: a latent feature extraction module, a diffusion model module, and an upsampling module; wherein,

[0040] The latent feature extraction module includes a convolutional layer, a pooling layer, and a fully connected layer. It is used to obtain a D-dimensional latent feature vector based on the visual image of the corresponding key frame, and to concatenate the latent vectors corresponding to two key frames respectively through a conditional encoder to obtain a synthetic latent vector.

[0041] The diffusion model module is used to gradually add noise to the synthesized latent vector through a forward diffusion process and gradually remove noise through a reverse diffusion process to generate the latent vector of the intermediate frame.

[0042] The upsampling module includes an initial convolutional layer, a transposed convolutional layer, and a ReLU activation function, which are used to convert the latent vectors of intermediate frames into a low-resolution feature map, and then enlarge the size to restore the resolution of the original image of the video frame.

[0043] Preferably, the number of intermediate frames generated based on the frame interpolation network is matched with the number of frame intervals carried by the first keyframe, thereby generating a complete video frame sequence and realizing the recovery of video frames.

[0044] On the other hand, the present invention provides an underwater wireless optical semantic communication system based on image and video transmission, implemented using an optical transmitter and an optical receiver, the system comprising:

[0045] The acquisition and denoising module is used to acquire video frame sequences and perform Gaussian denoising processing.

[0046] The keyframe extraction module is used to extract keyframes from the denoised video frame sequence by statistically analyzing the gray-level distribution, constructing a discrete histogram, and performing similarity analysis.

[0047] The semantic extraction module is used to extract basic features from keyframes. In conjunction with pyramid pooling and attention mechanisms, it generates semantic segmentation maps of keyframes. The segmentation map group with reduced data volume is formed by difference calculation and input into the optical transmitter to be converted into an optical signal and then emitted into the underwater medium.

[0048] The overlay and restoration module is used to convert and overlay the optical signal received by the optical receiver after transmission through the underwater medium into a semantic segmentation map group.

[0049] The semantic restoration module is used to restore semantic segmentation graphs based on an enhanced recurrent adversarial generative network to obtain visual images; and

[0050] The frame interpolation module is used to generate a complete video frame sequence based on the visual image of the corresponding keyframe through a frame interpolation network, thereby realizing the recovery of video frames.

[0051] Compared with the prior art, the advantages of the present invention are:

[0052] 1. Application of semantic compression technology: By performing semantic analysis and encoding on keyframes through a semantic extraction module, data is further compressed while retaining important information, improving the efficiency and quality of data transmission. This significantly reduces bandwidth requirements, enabling efficient and stable transmission of video information even under bandwidth constraints.

[0053] 2. Integration with underwater optical communication technology: Utilizing underwater optical communication technology as the medium for data transmission. This technology has advantages such as high bandwidth, low latency, low power consumption, small size, strong anti-interference ability, and good confidentiality, effectively solving the problem of transmission efficiency being limited by channel characteristics in existing underwater wireless video transmission.

[0054] 3. System architecture optimization: The entire system combines semantic compression and optical communication to achieve a complete process from video acquisition, key frame extraction, semantic encoding, underwater transmission, semantic recovery to frame interpolation, optimizing the data transmission process and improving transmission efficiency and stability.

[0055] 4. Meets the needs of complex underwater environments: The system can meet the operational needs of underwater robots and other equipment in complex underwater environments, providing a new solution for the field of underwater communication, especially in scenarios such as deep-sea exploration and underwater robot operations, where it can provide real-time high-definition video communication support. Attached Figure Description

[0056] Figure 1 This is a flowchart of the underwater wireless optical semantic communication method based on image and video transmission according to the present invention;

[0057] Figure 2 yes Figure 1 Semantic extraction flowchart;

[0058] Figure 3 yes Figure 2 A structural diagram of feature enhancement and semantic information extraction;

[0059] Figure 4 yes Figure 2 Flowchart of the CBAM hybrid attention mechanism;

[0060] Figure 5 yes Figure 4 Schematic diagram of channel attention mechanism

[0061] Figure 6 yes Figure 4 A schematic diagram of the spatial attention mechanism;

[0062] Figure 7 yes Figure 1 A schematic diagram of semantic information extraction;

[0063] Figure 8 This is a flowchart of an image semantic restoration method based on enhanced recurrent adversarial generative networks;

[0064] Figure 9 yes Figure 1 The structure diagram of the frame interpolation network;

[0065] Figure 10 yes Figure 1 A schematic diagram of the frame interpolation principle. Detailed Implementation

[0066] This invention aims to provide an underwater video transmission method and system based on semantic compression and optical communication. By deeply integrating semantic coding technology with an underwater wireless optical communication (UWOC) system, it solves the problem of low efficiency in underwater high-definition video transmission caused by channel characteristics and limited bandwidth resources in the prior art. Specifically, this invention addresses the core contradictions of physical layer interference such as scattering attenuation and turbulence disturbances in complex underwater optical channels, as well as the fact that video streams in the traditional "full-volume transmission" mode far exceed the carrying capacity of the UWOC system. It achieves three major innovative goals by constructing a full-link optimization framework encompassing video semantic feature extraction, intelligent compression, optical communication transmission, and semantic reconstruction: First, it utilizes deep learning algorithms to perform semantic-level understanding and keyframe selection of video content, compressing the original data volume by more than 90%, thus reducing the transmission bandwidth requirement to within the effective carrying capacity of the existing UWOC system. Second, through joint optimization design of semantic coding and channel coding, it adaptively adjusts the compression rate and modulation format when channel conditions change dynamically, improving the robustness of video transmission in complex underwater environments. Third, through a frame interpolation network, it recovers discontinuous frame video sequences, ultimately constructing a new generation of underwater video transmission system with high energy efficiency and intelligent sensing characteristics, providing real-time high-definition video communication support for scenarios such as deep-sea exploration and underwater robot operations.

[0067] This invention discloses an underwater video transmission method based on semantic compression and optical communication. The method includes source data input, keyframe extraction, semantic extraction, optical transmitter, optical receiver, semantic recovery, and frame interpolation. A schematic diagram of the invention is attached. Figure 1 As shown.

[0068] More specifically, the source data input refers to the raw data input.

[0069] The optical transmitter and optical receiver are the medium for data transmission, and have advantages such as high bandwidth, low latency, low power consumption, small size, strong anti-interference ability and good confidentiality.

[0070] The semantic extraction involves semantically encoding the keyframe sequence to generate semantic information for the keyframes.

[0071] The semantic recovery involves semantically decoding the demodulated data to recover the keyframes.

[0072] The frame interpolation process generates intermediate frames through an interpolation network based on the recovered keyframes, ultimately restoring the complete video information.

[0073] The entire method significantly reduces bandwidth requirements through semantic compression technology, effectively solving the problem of transmission efficiency constraints caused by channel characteristics in existing underwater wireless video transmission. Simultaneously, by utilizing underwater optical communication technology, it achieves efficient and stable transmission of underwater video information, meeting the operational needs of underwater robots and other equipment in complex underwater environments.

[0074] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0075] Example 1

[0076] Embodiment 1 of the present invention proposes an underwater wireless optical semantic communication method based on image and video transmission, which specifically includes the following steps:

[0077] Step 101: Using the camera as the input source of the source data, obtain the original video frame sequence V = {v1, v2, ...}. To ensure the accuracy of subsequent keyframe extraction, Gaussian denoising processing is performed on the obtained original video frame sequence to reduce the interference of noise on keyframe judgment.

[0078] Step 102: Input the original video frame sequence after Gaussian denoising into the keyframe extraction module, and output the keyframe sequence. This module dynamically adjusts the frame interval of the keyframes according to the channel capacity. It should be noted that the interval between adjacent keyframes should be combined with the keyframe that appears earlier in time for transmission to ensure the integrity and effectiveness of the transmission.

[0079] In one embodiment of this application, the keyframe extraction module comprises a grayscale conversion module, a histogram normalization module, and a keyframe decision module. The grayscale conversion module converts the input video frame sequence into grayscale images v = {v1′, v2′, ...}, defining the first frame v1′ as the keyframe I1. In the histogram normalization module, the grayscale distribution of the video frames is first statistically analyzed to construct a discrete histogram, and then the histogram frequency is converted into a probability distribution using formula (1). In the keyframe decision module, the sum of the square roots of the probability distributions of subsequent frames and the keyframe is first calculated using formula (2) to obtain the corresponding similarity BC = {BC...}. 2-1 ,BC 3-1 The BC value ranges from [0,1], where 1 indicates that the two histograms completely overlap (completely similar), and 0 indicates that they do not overlap (completely dissimilar). Then, similarity analysis is performed. When BC is less than the decision threshold, it is judged as a significant difference, and this frame is defined as a keyframe. The next keyframe is then calculated in this way.

[0080]

[0081] Where i represents the i-th frame in the time series, k is the interval index of the histogram of the i-th frame, indicating the position of the current interval. N is the number of intervals in the histogram of the i-th frame, representing the total number of intervals into which the histogram is divided. P i Let H be the normalized probability distribution of the i-th frame. i [k] is the frequency of the k-th interval in the histogram of the i-th frame, P i [k] represents the probability in the k-th interval of the histogram of the i-th frame.

[0082]

[0083] In the embodiments of this application, considering the real-time performance of the video and the robustness of the system, the frame interval of the aforementioned key frames cannot exceed the maximum frame interval P. P is negatively correlated with the current channel capacity; when the channel capacity is large, the value of P can be decreased, and when the channel capacity is small, the value of P can be increased. During the decision-making process, if no key frame appears within the range of P, then the (P+1)th frame should be designated as the next key frame.

[0084] Step 103: Semantic encoding of the keyframe sequence was implemented. The original image underwent efficient compression and semantic parsing through a feature extraction module, a feature enhancement module, and a semantic information extraction module. (See attached image.) Figure 2 As shown, the model framework first inputs the input image (size H×W×3) into the ResNet-50 backbone network for basic feature extraction. ResNet-50 employs a five-stage architecture: the first stage reduces the feature map size to [size missing] using 7×7 convolutions (64 channels, stride 2) and 3×3 max pooling (stride 2). The second to fifth stages extract deep features sequentially using 3, 4, 6, and 3 residual blocks, respectively, resulting in a final output size of [size missing]. The feature maps are as follows. Among them, the residual blocks are divided into two types: Conv Block and Identity Block. The Conv Block adjusts the number of channels through 1×1 convolution to match the subsequent input, while the Identity Block maintains network stability and avoids the gradient vanishing problem through identity mapping.

[0085] The feature enhancement module consists of a pyramid pooling module (PPM) and an attention mechanism module. The PPM module takes the feature map output by ResNet-50 as input and first performs multi-scale pooling: dividing the feature map into 1×1, 2×2, 3×3, and 6×6 sub-regions, and performing average pooling on each to capture contextual information at different scales. Each sub-region is then reduced to 512 channels via 1×1 convolution and restored to its original size through bilinear interpolation. Finally, it is spliced ​​with the original feature map to form For dimensional enhancement features, see Appendix Figure 3 Appendix Figure 4 The flowchart below shows the CBAM hybrid attention mechanism. Subsequently, the attention mechanism module optimizes the enhanced features: the channel attention mechanism generates channel weight vectors through global average pooling and dynamically adjusts channel importance using a sigmoid activation function, as shown in the attached diagram. Figure 5 The spatial attention mechanism generates a spatial weight matrix through local max pooling and average pooling, fuses spatial information using 7×7 convolution, and finally outputs weighted, highly discriminative features, as shown in the appendix. Figure 6 As shown.

[0086] The semantic information extraction module receives the optimized feature map. Feature fusion, channel adjustment, and upsampling are performed sequentially. First, multi-scale features are integrated using a 3×3 convolution, outputting a feature map of size H / 8×W / 8×512. Then, a 1×1 convolution is used to adjust the number of channels to match the number of target classes (e.g., class C), generating... The initial segmentation results are obtained; finally, the feature map is upsampled to the original input resolution (H×W×3) through bilinear interpolation to obtain the final semantic segmentation image, as shown in the attached figure. Figure 7 As shown.

[0087] To optimize model performance, a combination strategy of cross-entropy loss, Dice loss, and Focal loss is adopted during training. Cross-entropy loss improves classification accuracy by minimizing the difference between the predicted probability distribution and the true label, as shown in formula (3); Dice loss maximizes the overlap between the predicted region and the true region to enhance segmentation consistency, as shown in formula (4); Focal loss alleviates the class imbalance problem by dynamically adjusting the weights of easy and difficult samples, as shown in formula (5).

[0088]

[0089] Where N is the size of the dataset, M is the number of categories, and y ic It is an indicator function (0 or 1) used to indicate whether sample i belongs to category c, p ic Cross-entropy loss represents the model's predicted probability that sample i belongs to category c. The goal of cross-entropy loss is to minimize the difference between the predicted probability distribution and the true label, thereby improving the model's classification accuracy.

[0090]

[0091] Here, X represents the model's prediction, and Y represents the true result. The goal of Dice loss is to maximize the overlap between the predicted and true results, thereby improving the accuracy of segmentation.

[0092] L Focal =-α t (1-p t ) γ log(p t (5)

[0093] Where, α t These are class weights, used to adjust the ratio of positive to negative samples, p t γ is the classification probability of the identified object, and γ is the focusing parameter used to adjust the weight of hard examples. The goal of Focal loss is to improve the model's performance when dealing with imbalanced datasets by reducing the weight of easily classified samples and increasing the weight of hard-classified samples.

[0094] Step 104: To reduce the amount of information transmitted, the semantic segmentation graphs output by semantic coding are further processed. Specifically, a semantic segmentation graph group is first defined, with the i-th semantic segmentation graph denoted as gi, and a group of semantic segmentation graphs represented as G = {g1, g2, ..., g...} n The semantic segmentation map G contains i keyframes. During communication transmission, the original semantic segmentation map group G is not transmitted directly; instead, a segmentation map group S = {s1, s2, ..., s...} that has undergone data reduction processing is transmitted. n}

[0095] In the specific processing, the baseline segmentation map is first determined by setting the first segmentation map g1 in the semantic segmentation map group G as the group's baseline segmentation map. Then, the difference between subsequent semantic segmentation maps is calculated to reduce the amount of data. Specifically, the first semantic segmentation map g1 is used as the baseline segmentation map s1, i.e., s1 = g1; for the subsequent i-th semantic segmentation map g... i (i≥2), calculate its relationship with the previous semantic segmentation graph g.i-1 The difference is used as the i-th difference segmentation image s in the segmentation image group S. i That is, s2 = g2 - g1, s3 = g3 - g2, and so on, the subsequent s i =g i -g i-1 .

[0096] Using this method, identical elements in a set of semantic segmentation graphs only need to be sent once during transmission, effectively reducing the amount of information transmitted and improving data transmission efficiency. Simultaneously, because the semantic segmentation graphs and their difference information of the keyframes are preserved, the receiving end can accurately reconstruct the original semantic segmentation graph set G through a simple accumulation operation.

[0097] Step 105: The encoded digital signal needs to be input into the optical transmitter, converted into an optical signal, and then transmitted into the underwater medium.

[0098] Step 106: The optical receiver is responsible for receiving these optical signals and converting them back into digital signals.

[0099] Step 107: The receiving end needs to restore the received information to the original semantic segmentation graph group. This process is the reverse of step 104. The receiving end first receives the segmentation graph group S = {s1, s2, ..., s...} after the data volume has been reduced. n To achieve restoration, an empty semantic segmentation graph group G′ is created to store the restored semantic segmentation graphs. The first segmentation graph s1 in the received segmentation graph group S is directly used as the first semantic segmentation graph g1′ in the restoration group G′, i.e., g1′ = s1.

[0100] Next, the subsequent segmentation images are restored sequentially. Starting from the second segmentation image, each difference segmentation image s is then processed. i (i≥2) and the previous restored semantic segmentation graph g′ i-1 The segments are accumulated to recover the original semantic segmentation map. Specifically, the second restored semantic segmentation map g2′ = g1′ + s2; the third restored semantic segmentation map g3′ = g2′ + s3; and so on, until all segmentation maps are restored, resulting in a complete semantic segmentation map group G′.

[0101] Using this method, the receiving end can restore the transmitted reduced data group S to the original semantic segmentation graph group G, thereby recovering the semantic information consistent with the original data.

[0102] Step 108: The receiving end uses the semantic decoding module to restore the semantic information into a visually realistic image and outputs a keyframe sequence I = {I0, I1…}. This invention proposes an image semantic restoration method based on an enhanced recurrent adversarial generative network (EN-CycleGAN). By introducing cycle consistency loss and adversarial loss, the generator is prompted to learn more detailed information at multiple scales, thereby improving the accuracy and visual effect of semantic restoration.

[0103] The algorithm flow of this image semantic method is attached. Figure 8 As shown, assume the data in the X and Y domains are X and Y, respectively, with data distributions PX and PY. Generator G maps an image x in the X domain to an image G(x) in the Y domain, while generator F establishes a mapping relationship from the Y domain to the X domain, i.e., F(y). During training, the model needs to simultaneously learn the mapping relationship between G and F, satisfying the cycle consistency requirements F(G(x))≈x and G(F(y))≈y. This cycle-consistent mapping mechanism effectively avoids the problem of the generator mapping all X domain images to the same image, ensuring continuous training until the generated images achieve the expected results, thereby effectively restoring image details and realistic visual effects while preserving semantic features.

[0104] Regarding the generator architecture, to address the loss of spatial information and semantic features in image segmentation and semantic restoration tasks, this invention employs an improved U-Net architecture. U-Net enhances feature extraction capabilities at multiple scales by combining residual blocks and skip connections. Its encoder utilizes multiple convolutional layers for feature extraction, and skip connections pass low-level feature information to higher-resolution layers for reuse during decoding. Building upon this, this invention improves the skip connections of U-Net by adding residual block structures to learn richer detailed features, further enhancing image detail restoration capabilities.

[0105] To improve the performance of Generative Adversarial Networks (GANs), this invention modifies the PatchGAN architecture by introducing dilated convolutions to enhance its ability to capture global information. Dilated convolutions control the distance between convolutional windows by adjusting the dilation rate, thus expanding the receptive field. This allows the discriminator to capture global image information over a larger area, improving its discrimination ability and enabling it to capture image features at multiple scales, thereby enhancing its ability to judge the authenticity of generated images. The improved discriminator structure contains multiple dilated convolutional layers, which expand the receptive field.

[0106] To optimize the image semantic restoration model, the loss function combines adversarial loss, cycle consistency loss and identity loss, as shown in Equation (6).

[0107] Loss = Loss GAN+λ·Loss cycle +Loss identity (6)

[0108] Loss GA N represents Generative Adversarial Loss, Loss cycle This represents the cycle consistency loss. identity Let λ represent the identity loss, and λ be the weighting coefficient of the cycle consistency loss.

[0109] Adversarial loss is used to optimize the performance of the generator and discriminator, ensuring that the images generated by the generator are more visually realistic. The principle is to make the images generated by the generator difficult for the discriminator to distinguish through adversarial training, as shown in formula (7).

[0110] L GAN =E y [logD Y (y)]+E x [log(1-D Y (G(z))]+E x [logD X (x)]+E y [log(1-D X (F(y)))](7)

[0111] Among them, D Y and D X Here, G and F are the discriminator, used to distinguish between real and generated data, and the generator, G, and F are the generators. By minimizing the adversarial loss, the generator can produce more realistic images.

[0112] Cycle consistency loss is used to ensure that the generator's output, after inverse mapping, is as close as possible to the original input. Its principle is to prevent the generator from mapping all inputs to the same image, thereby improving the model's diversity and robustness, as shown in Equation (8).

[0113] L Cycle =E x [||F(G(x))-x||1]+E y [||G(F(y))-y||1] (8)

[0114] This loss ensures that the generator's output can be restored to a state close to the original input after inverse mapping, thereby improving the stability and reliability of the model.

[0115] The identity loss is used to prevent the generator from modifying the tone and details in the original image. The principle is to ensure that the generator retains the basic features of the input image when processing it by minimizing the identity loss, as shown in Equation (9).

[0116] L Identity =E y [|D Y (G(y))-y||1]+E x [||D x (G(x))-x||1] (9)

[0117] By minimizing the identity loss, the generator is able to recover the detailed features of an image while preserving its content. Using this method, the receiver can reconstruct the visual image from the recovered semantic information.

[0118] Step 109: This invention describes a video generation method based on a frame interpolation network, which gradually generates a complete video frame sequence by inputting a keyframe sequence. The network consists of three main modules: a latent feature extraction module, a diffusion model module, and an upsampling module, as shown in the attached diagram. Figure 9 As shown.

[0119] To generate intermediate frames I of keyframes I0 and I1 0.5 For example, I0 and I1 are first input into the latent feature extraction network to obtain latent feature vectors z0 and z1, respectively. The image input size of the latent feature extraction network is H×W×C (height, width, number of channels). The latent feature extraction network consists of convolutional layers, pooling layers, and fully connected layers. The output size of the convolutional layer is shown in formula (10), where H in and W in These are the height and width of the input, padding is the padding size, kernel_size is the kernel size, and stride is the stride.

[0120]

[0121] The image output size of the pooling layer is given by formula (11), where pool_size is the size of the pooling window and stride is the stride.

[0122]

[0123] Input image X∈R H×W×C The output of the convolutional layer is expressed as Equation (12), and the output of the pooling layer is expressed as Equation (13). The output of the fully connected layer is expressed as Equation (14), and the final output is a D-dimensional latent feature vector Z∈R. D Through the latent feature extraction module, keyframes I0 and I1 are transformed into latent feature vectors z0 and z1, respectively, with a dimension of 1×1×D.

[0124] X i =ReLU(Conv(X,W) i ,b i (12)

[0125] X i =MaxPool(X i (13)

[0126] X i =ReLU(fully_connected(X,W) i ,b i (14)

[0127] The latent vectors z0 and z1 from two images are concatenated using a conditional encoder to obtain the synthetic latent vector z. 0-1 This noise is then input into the diffusion model. The core of the diffusion model is to progressively add noise through a forward diffusion process and progressively remove noise through a reverse diffusion process to generate the latent vector of the intermediate frame. In the diffusion model, the forward diffusion process progressively adds noise to the latent vector z, generating z... t See formula (15), where β t These are the coefficients for adding noise, where N represents a Gaussian distribution, and β... t I indicates that the covariance matrix is ​​a diagonal matrix with diagonal elements β. t Off-diagonal elements are 0; through a reverse diffusion process, a new latent vector is generated from the noise, as shown in formula (16), where μ θ and ∑ θ These are the prediction functions for the mean and covariance, respectively. N represents a Gaussian distribution, t represents the time step, and c represents the conditional information. In the actual generation of intermediate frames, the synthesized latent vector is input into the pre-trained diffusion model, and the latent vector z of the intermediate frame is obtained through inverse diffusion. 0.5 (Dimensions are 1×1×D).

[0128]

[0129] p θ (z t-1 |z t ,c)=N(z t-1 μ θ (z t ,c,t),∑ θ (z t ,c,t)) (16)

[0130] The latent vectors that generate intermediate frames are then used by an upsampling module to generate the final keyframe images, restoring them to the resolution of the original image.

[0131] The latent vector input to the initial convolutional layer is transformed into a low-resolution feature map with dimension . Where N is the number of subsequent upsampling steps, and C′ is a large number of channels to carry sufficient feature information. This process can be represented by formula (17), where X0 represents the initial feature map, and W0 and b0 are the weights and biases of the convolutional layer, respectively.

[0132] X0 = Conv(z) 0.5 (17)

[0133] Initial features Figure X 0 is input into the upsampling unit. Each upsampling unit consists of a transposed convolutional layer and a ReLU activation function, the purpose of which is to double the size of the feature map. The transposed convolutional kernel size is 4×4, the stride is 2, and the padding is 1. The size of the output feature map can be calculated by formula (18), where H in and W in These are the height and width of the input feature map, H, respectively. out and W out H represents the height and width of the output feature map. in and W in These are the height and width of the input feature map, H, respectively. out and W out It represents the height and width of the output feature map.

[0134] H out =(H in -1)×stride-2×padding+kernel_size

[0135] W out =(W in -1)×stride-2×padding+kernel_size (18)

[0136] As the upsampling process progresses, the resolution of the feature map gradually approaches the resolution of the target image, while the number of channels gradually decreases. After N upsampling iterations, the resolution of the feature map reaches W×H. At this point, a convolutional layer is used to convert the number of channels in the feature map to C, which is the number of channels in the target image. This convolutional layer uses the Tanh activation function to limit the output value to the range [-1, 1], and then maps it to the pixel value range of the image through a linear transformation. The final image generation process can be represented by formula (19), where X last X represents the feature map output by the last upsampling unit. final W represents the final generated image. final and b finalThese represent the weights and biases of the final convolutional layer. Through the above process, the latent vector z of the intermediate frame... 0.5 The images are progressively upsampled and converted into intermediate frame images with a target resolution of W×H×C.

[0137] X final =Tanh(Conv(X) last W final ,b final (19)

[0138] Based on the above method, intermediate frames are generated step by step from the initial two keyframes until the number of intermediate frames generated matches the number of frame intervals carried by the first keyframe. (See Appendix) Figure 10 .

[0139] Example 2

[0140] Embodiment 2 of the present invention provides an underwater wireless optical semantic communication system based on image and video transmission, implemented using an optical transmitter and an optical receiver. The system includes:

[0141] The acquisition and denoising module is used to acquire video frame sequences and perform Gaussian denoising processing.

[0142] The keyframe extraction module is used to extract keyframes from the denoised video frame sequence by statistically analyzing the gray-level distribution, constructing a discrete histogram, and performing similarity analysis.

[0143] The semantic extraction module is used to extract basic features from keyframes. In conjunction with pyramid pooling and attention mechanisms, it generates semantic segmentation maps of keyframes. The segmentation map group with reduced data volume is formed by difference calculation and input into the optical transmitter to be converted into an optical signal and then emitted into the underwater medium.

[0144] The overlay and restoration module is used to convert and overlay the optical signal received by the optical receiver after transmission through the underwater medium into a semantic segmentation map group.

[0145] The semantic restoration module is used to restore semantic segmentation graphs based on an enhanced recurrent adversarial generative network to obtain visual images;

[0146] The frame interpolation module is used to generate a complete video frame sequence based on the visual image of the corresponding keyframe through a frame interpolation network, thereby realizing the recovery of video frames.

[0147] It is worth noting that in the embodiments of the above system, the modules included are divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional module are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0148] Overview:

[0149] This application achieves efficient, accurate, and highly adaptable video processing and transmission through a series of innovative technical means. First, the edge device controls the camera to acquire the original video frame sequence and then performs Gaussian denoising to effectively reduce noise interference, providing high-quality foundational data for subsequent keyframe extraction. The keyframe extraction module dynamically adjusts the interval between adjacent keyframes based on channel status and communication rate, and cleverly combines the interval information with the preceding keyframes for transmission, ensuring the integrity and effectiveness of the transmission. Its internal grayscale conversion, histogram normalization, and keyframe decision-making processes work closely together, accurately determining keyframes based on probability distribution and similarity analysis. Simultaneously, considering video real-time performance and system robustness, the maximum frame interval is reasonably controlled to ensure the scientific and practical nature of keyframe extraction.

[0150] In the semantic encoding stage, a ResNet-50 backbone network is used to extract basic features. This, combined with a feature enhancement module consisting of a pyramid pooling module and an attention mechanism, comprehensively captures contextual information and optimizes features, improving their discriminativeness and effectiveness. The subsequent semantic information extraction module, through feature fusion, channel adjustment, and upsampling operations, combined with training optimization strategies using various loss functions, generates accurate semantic segmentation images, providing efficient and semantically clear output for semantic encoding.

[0151] For underwater wireless optical communication data transmission, a method is proposed to reduce the data volume by performing difference calculations on semantic segmentation graph groups, which greatly reduces the amount of transmitted information and improves transmission efficiency. Simultaneously, this method ensures that the receiving end can completely reconstruct the original semantic segmentation graph group through a simple accumulation operation, thereby guaranteeing the accuracy and reliability of data transmission in underwater wireless optical communication.

[0152] In the semantic restoration stage, the semantic decoding module based on a recurrent adversarial generative network (ROGGNN) leverages ROG consistency loss and adversarial loss to enable the generator to learn rich detail information, achieving high-quality semantic restoration and accurately reconstructing semantic information into visually realistic images. Meanwhile, the video generation method utilizes a frame interpolation network, through the synergistic effect of latent feature extraction, a diffusion model, and an upsampling module, to progressively generate a complete video frame sequence from a keyframe sequence. This effectively achieves efficient video generation and restoration, significantly improving the overall performance and effectiveness of the video processing system, demonstrating significant technical advantages and practical application value.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An underwater wireless optical semantic communication method based on image and video transmission, implemented using an optical transmitter and an optical receiver, the method comprising: Step 1: Obtain the video frame sequence and perform Gaussian denoising processing; Step 2: For the denoised video frame sequence, construct a discrete histogram by statistically analyzing the gray level distribution, and extract keyframes through similarity analysis; Step 3: Extract basic features from keyframes, combine them with pyramid pooling and attention mechanisms to generate semantic segmentation images of keyframes, and use difference calculation to form semantic segmentation image groups with reduced data volume. Input the images into the optical transmitter to convert them into optical signals and then transmit them to the underwater medium. Step 4: Convert and superimpose the optical signal received by the optical receiver after transmission through the underwater medium into a semantic segmentation graph group; Step 5: Restore the semantic segmentation graph group based on the enhanced recurrent adversarial generative network to obtain the visual image; Step 6: Based on the visual images of the corresponding keyframes, generate a complete video frame sequence through a frame interpolation network; thereby realizing the restoration of video frames.

2. The underwater wireless optical semantic communication method based on image and video transmission according to claim 1, characterized in that, Step 2 includes: Step 2-1: Convert the denoised video frame sequence into a grayscale image v = {v1, v2, ...}, and designate the first frame v1 as the keyframe I1; Step 2-2: Statistically analyze the grayscale distribution of video frames, construct a discrete histogram, and then convert the histogram frequencies into a probability distribution: Where i represents the i-th frame in the time series, k is the interval index of the histogram of the i-th frame, indicating the position of the current interval, N is the number of intervals in the histogram of the i-th frame, representing the total number of intervals into which the histogram is divided, and H... i [k] is the frequency of the k-th interval in the histogram of the i-th frame, P i [k] represents the probability in the k-th interval of the histogram of the i-th frame; Steps 2-3: Calculate the sum of the square roots of the product of the probability distributions of frame A and frame B using the following formula to obtain the corresponding similarity BC: Among them, P A [k],P B [k] represents the probability of the histogram of frame A and the histogram of frame B in the kth interval; the BC value ranges from [0,1], where 0 indicates no overlap and 1 indicates that the two histograms completely overlap. Step 2-4: Perform similarity analysis. When BC is less than the decision threshold, it is judged as a significant difference. Define this frame as a keyframe and go to step 2-3. Continue until the grayscale images v = {v1, v2, ...} have been traversed. Finally, output the RGB image sequence I = {I1, I2, ...} corresponding to the extracted keyframes.

3. The underwater wireless optical semantic communication method based on image and video transmission according to claim 1, characterized in that, Step 3 includes: Basic feature extraction is performed using a ResNet-50 backbone network, and the output is... The feature map; where H and W are the height and width of the original image of the video frame, respectively; The feature enhancement module consists of a pyramid pooling module and an attention mechanism module. The pyramid pooling module... The feature map undergoes multi-scale pooling: it is divided into 1×1, 2×2, 3×3, and 6×6 sub-regions, and average pooling is performed on each sub-region to capture contextual information of different ranges. Each sub-region is reduced to 512 channels by 1×1 convolution, then restored to its original size by bilinear interpolation, and finally concatenated with the original feature map to form a new feature map. The attention mechanism module includes a channel attention mechanism and a spatial attention mechanism. The channel attention mechanism generates channel weight vectors through global average pooling and dynamically adjusts channel importance by combining a sigmoid activation function. The spatial attention mechanism generates a spatial weight matrix through local max pooling and average pooling, fuses spatial information using 7×7 convolution, and outputs weighted high discriminative features. Feature fusion and channel adjustment are performed sequentially. The feature map is upsampled to the original input resolution H×W×C by bilinear interpolation to obtain a semantic segmentation image, where C is the number of target categories. The semantic segmentation images obtained from each keyframe are combined to form a semantic segmentation graph group G.

4. The underwater wireless optical semantic communication method based on image and video transmission according to claim 1, characterized in that, Step 3 also includes: Set the first segmentation graph in the semantic segmentation graph group G as the group baseline segmentation graph; Calculate the difference between the two semantic segmentation maps sequentially; The baseline segmentation map and the subsequent difference are combined to form a segmentation map group S with reduced data volume. This group is then input into an optical transmitter, converted into an optical signal, and transmitted to the underwater medium.

5. The underwater wireless optical semantic communication method based on image and video transmission according to claim 4, characterized in that, Step 4 includes: converting the optical signal received by the optical receiver into a segmentation map group S, and performing superposition and restoration processing sequentially to obtain the restored semantic segmentation map group G.

6. The underwater wireless optical semantic communication method based on image and video transmission according to claim 5, characterized in that, The enhanced recurrent adversarial generative network in step 5 includes a generator that employs an improved U-Net architecture and enhances feature extraction capabilities at multiple scales by combining residual blocks and skip connections. In the encoder section, multiple convolutional layers are used for feature extraction. Skip connections are used to pass low-level feature information to higher-resolution layers so that the feature information extracted by the encoder can be reused during the decoding process.

7. The underwater wireless optical semantic communication method based on image and video transmission according to claim 1, characterized in that, The enhanced recurrent adversarial generative network in step 5 also includes a discriminator during the training phase. The discriminator adopts the PatchGAN architecture and is improved by introducing multiple dilated convolutions to enhance its ability to capture global information. The loss function Loss during the training phase is: Loss=Loss GAN +λ·Loss cycle +Loss identity Among them, Loss GAN This represents the adversarial loss, Loss cycle Loss represents the loss due to cycle consistency. identity Let λ represent the identity loss, and λ be the weighting coefficient of the cycle consistency loss.

8. The underwater wireless optical semantic communication method based on image and video transmission according to claim 1, characterized in that, The frame interpolation network in step 6 includes: a latent feature extraction module, a diffusion model module, and an upsampling module; wherein... The latent feature extraction module includes a convolutional layer, a pooling layer, and a fully connected layer. It is used to obtain a D-dimensional latent feature vector based on the visual image of the corresponding key frame, and to concatenate the latent vectors corresponding to two key frames respectively through a conditional encoder to obtain a synthetic latent vector. The diffusion model module is used to gradually add noise to the synthesized latent vector through a forward diffusion process and gradually remove noise through a reverse diffusion process to generate the latent vector of the intermediate frame. The upsampling module includes an initial convolutional layer, a transposed convolutional layer, and a ReLU activation function, which are used to convert the latent vectors of intermediate frames into a low-resolution feature map, and then enlarge the size to restore the resolution of the original image of the video frame.

9. The underwater wireless optical semantic communication method based on image and video transmission according to claim 8, characterized in that, The number of intermediate frames generated by the frame interpolation network is matched with the number of frame intervals carried by the first keyframe, thereby generating a complete video frame sequence and realizing the recovery of video frames.

10. An underwater wireless optical semantic communication system based on image and video transmission, implemented using an optical transmitter and an optical receiver, characterized in that, The system includes: The acquisition and denoising module is used to acquire video frame sequences and perform Gaussian denoising processing. The keyframe extraction module is used to extract keyframes from the denoised video frame sequence by statistically analyzing the gray-level distribution, constructing a discrete histogram, and performing similarity analysis. The semantic extraction module is used to extract basic features from keyframes. In conjunction with pyramid pooling and attention mechanisms, it generates semantic segmentation maps of keyframes. The segmentation map group with reduced data volume is formed by difference calculation and input into the optical transmitter to be converted into an optical signal and then emitted into the underwater medium. The overlay and restoration module is used to convert and overlay the optical signal received by the optical receiver after transmission through the underwater medium into a semantic segmentation map group. The semantic restoration module is used to restore semantic segmentation graphs based on an enhanced recurrent adversarial generative network to obtain visual images; and The frame interpolation module is used to generate a complete video frame sequence based on the visual image of the corresponding keyframe through a frame interpolation network, thereby realizing the recovery of video frames.

Images