Sonar image semantic communication method and device, equipment, storage medium and computer program product

Abstract

The invention relates to the technical field of link communication, and discloses a sonar image semantic communication method and device, equipment, a storage medium and a computer program product, and the method comprises the steps: obtaining an original sonar image, carrying out the semantic coding of the original sonar image according to a semantic network, and extracting semantic features; combining the semantic features with the signal-to-noise ratio information of the channel to obtain channel coding output vector data; compressing the channel coding output vector data based on a quantization module to obtain target integer data, and sending the target integer data to a network receiving end through a channel; and the network receiving end recovers the received target integer data into the channel coding output vector data based on the quantization factor, and performs channel decoding and semantic decoding on the channel coding output vector data to obtain a target sonar image. A sonar image is efficiently returned to a land base station by using a semantic communication technology, and the challenge brought to information return due to scarcity of sonar image data and limitation of channel bandwidth is solved.

Description

Technical Field

[0001] The present application relates to the field of link communication technology, and in particular to a sonar image semantic communication method, device, equipment, storage medium and computer program product. Background Art

[0002] When transmitting large amounts of ocean data such as sonar images or videos through a sea surface-satellite-land link using low-orbit satellites as relays, the bandwidth of the link channel is limited, the channel quality changes dynamically, and the communication quality is poor. Summary of the invention

[0003] The main purpose of the present application is to provide a sonar image semantic communication method, device, equipment, storage medium and computer program product, which aims to solve the technical problems of limited link channel bandwidth, dynamic channel quality changes and poor communication quality when transmitting far sea ocean data of a large amount of ocean data such as sonar images or videos using low-orbit satellites as relays to realize the sea surface-satellite-land link.

[0004] To achieve the above-mentioned purpose, the present application proposes a sonar image semantic communication method, which is applied to a network sending end and includes: Acquire an original sonar image, and perform semantic encoding on the original sonar image according to a semantic network to extract semantic features; Combining the semantic features with the signal-to-noise ratio information of the channel to obtain channel coding output vector data; Compressing the channel coding output vector data based on the quantization module to obtain target integer data, and sending the target integer data to a network receiving end through a channel; The network receiving end restores the received target integer data into the channel coding output vector data based on the quantization factor, and performs channel decoding and semantic decoding on the channel coding output vector data to obtain the target sonar image.

[0005] Optionally, the step of acquiring an original sonar image, and semantically encoding the original sonar image according to a semantic network to extract semantic features includes: Obtain original sonar images collected by marine operating equipment; Dividing the original sonar image into blocks according to a network hyperparameter setting table preset in the semantic network to obtain small-sized sonar image blocks; The sonar image block is transmitted to a multi-layer Swin Transformer module to extract the semantic features of the original sonar image.

[0006] Optionally, the step of transmitting the sonar image block to a multi-layer Swin Transformer module to extract semantic features of the original sonar image includes: The divided sonar image blocks are sent as input to the multi-layer Swin Transformer module; In the multi-layer Swin Transformer module, a fixed-window multi-head attention mechanism and a moving-window multi-head attention mechanism are alternately used to capture the local and global initial semantic features of the sonar image block; The initial semantic features are calculated and adjusted through residual connection and layer normalization, and the adjusted initial semantic features are integrated to obtain the semantic features of the original sonar image.

[0007] Optionally, before the step of acquiring the original sonar image collected by the marine operation equipment, the step further includes: Selecting a synthetic aperture radar data set to perform pre-training on the semantic network to obtain pre-training network parameters; Selecting a fine-tuning method based on transfer learning, adjusting the pre-trained network parameters using a preset sonar image data set to obtain target network parameters; The target network parameters are applied in the semantic network to construct the network hyperparameter setting table and the Swin Transformer module.

[0008] Optionally, the step of combining the semantic feature with the signal-to-noise ratio information of the channel to obtain channel coding output vector data includes: Collect the signal-to-noise ratio information of the current channel; Acquire global statistical information of the semantic feature on the entire feature map based on a global average pooling function, and concatenate the global statistical information with the signal-to-noise ratio information to obtain context information; The context information is fed into a fully connected neural network, and a set of weight factors is calculated based on the semantic features based on a signal-to-noise ratio attention module; The semantic features are multiplied by the corresponding weight factors respectively to obtain channel coding output vector data.

[0009] In addition, to achieve the above-mentioned purpose, the present application also proposes a sonar image semantic communication method, which is applied to a network receiving end and includes: Receiving target integer data sent by a network sending end, the network sending end is used to perform semantic encoding on the original sonar image, extract semantic features, and compress the semantic features into target integer data after combining them with signal-to-noise ratio information of the channel; Restoring the target integer data into channel coding output vector data based on the quantization factor; Channel decoding is performed on the channel coding output vector data according to the signal-to-noise ratio information to reconstruct the semantic features of the sonar image; The semantic features of the sonar image are semantically decoded to reconstruct the target sonar image.

[0010] In addition, to achieve the above-mentioned purpose, the present application also proposes a sonar image semantic communication device, the sonar image semantic communication device comprising: A feature extraction module is used to obtain an original sonar image, and to perform semantic encoding on the original sonar image according to a semantic network to extract semantic features; A data generation module, used for combining the semantic features with the signal-to-noise ratio information of the channel to obtain channel coding output vector data; A data transmission module, used for compressing the channel coding output vector data based on the quantization module to obtain target integer data, and sending the target integer data to a network receiving end through a channel; The image reconstruction module is used to restore the received integer data to the channel coding output vector data based on the quantization factor, and perform channel decoding and semantic decoding on the channel coding output vector data to obtain a target sonar image.

[0011] In addition, to achieve the above-mentioned purpose, the present application also proposes a sonar image semantic communication device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sonar image semantic communication method as described above.

[0012] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the sonar image semantic communication method described above are implemented.

[0013] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the sonar image semantic communication method as described above are implemented.

[0014] The present application discloses obtaining an original sonar image, and semantically encoding the original sonar image according to a semantic network to extract semantic features; combining the semantic features with the signal-to-noise ratio information of the channel to obtain channel coding output vector data; compressing the channel coding output vector data based on a quantization module to obtain target integer data, and sending the target integer data to a network receiving end through a channel; the network receiving end restores the received target integer data to the channel coding output vector data based on a quantization factor, and performs channel decoding and semantic decoding on the channel coding output vector data to obtain a target sonar image. By constructing a semantic coding and decoding network, the semantics of the original signal are extracted to capture details more efficiently, and the sonar image is efficiently transmitted back to the land base station using semantic communication technology, and the integrity of the semantics is more accurately maintained during the transmission process, thereby improving the transmission efficiency of the communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 This is a flow chart of the first embodiment of the sonar image semantic communication method of the present application; Figure 2 This is the structure diagram of the SNR attention module for this application; Figure 3 A sea surface-satellite-land link diagram for sonar image data transmission in this application; Figure 4 This is a flow chart of the second embodiment of the sonar image semantic communication method of the present application; Figure 5 This is a structural diagram of the semantic coding network for this application; Figure 6 This is the internal structure diagram of the Swin Transformer module for this application; Figure 7 This is a schematic diagram of the SAR image target for this application; Figure 8 This is a schematic diagram of the forward-looking sonar image target for this application; Fig. 9 This is a flow chart of the third embodiment of the sonar image semantic communication method of the present application; Fig.10This is the flow chart of sonar image semantic communication for this application; Fig.11 This is the structure diagram of the semantic decoding network for this application; Fig.12 This is a comparison chart of the pre-training effect for this application; Fig.13 This is a comparison chart of MS-SSIM experimental results; Fig.14 This is a comparison chart of PSNR experimental results; Fig.15 This is a schematic diagram of the module structure of the sonar image semantic communication device according to an embodiment of the present application; Fig.16 Schematic diagram of the device structure of the hardware operating environment involved in the sonar image semantic communication method in the embodiment of the present application.

[0018] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0020] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0021] The main solution of the embodiment of the present application is: obtaining an original sonar image, and semantically encoding the original sonar image according to a semantic network to extract semantic features; combining the semantic features with the signal-to-noise ratio information of the channel to obtain channel coding output vector data; compressing the channel coding output vector data based on a quantization module to obtain target integer data, and sending the target integer data to a network receiving end through a channel; the network receiving end restores the received target integer data to the channel coding output vector data based on a quantization factor, and performs channel decoding and semantic decoding on the channel coding output vector data to obtain a target sonar image.

[0022] Since the data transmission from the sea can usually be realized through sea surface nodes and low Earth Orbit (LEO) satellites as a medium, at present, the data transmission service between ships and shore bases on the sea surface mainly relies on traditional maritime communication systems, such as the Global Maritime Distress and Safety System, the navigation telex system, and the automatic identification system. However, these systems can only provide low data rate communication. In addition, the harsh marine propagation environment and limited communication coverage have caused serious obstacles to the communication from the sea surface to the land. LEO can be used as a relay to realize the sea surface-satellite-land link for the data transmission from the sea surface to the satellite-land link because of its wide coverage. However, due to the limited channel bandwidth of the link and the dynamic change of channel quality, the communication quality is poor, especially for the rapid transmission of large amounts of ocean data such as sonar images or videos. In addition, semantic communication is a potential solution to the channel bandwidth limitation. Despite this, most of the existing semantic communication methods based on deep learning perform model training for a fixed channel environment, which leads to a rapid decrease in communication performance when the channel conditions change. In addition, deep learning requires a large amount of training data to make the model generalizable. However, due to factors such as the high cost of collecting ocean sonar images, the sonar image dataset is very limited, and the existing semantic communication methods based on deep learning are difficult to apply to the transmission of sonar images.

[0023] The present application provides a sonar image semantic communication method, which is oriented to the sea-satellite-land communication link and uses semantic communication technology to efficiently transmit sonar images back to land base stations, solving the challenges brought by the scarcity of sonar image data and the limited channel bandwidth for information transmission.

[0024] Based on this, the present application embodiment provides a sonar image semantic communication method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the sonar image semantic communication method of the present application.

[0025] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions or an electronic device capable of implementing the above functions, etc. The following takes the network sending end as an example to illustrate this embodiment and the following embodiments.

[0026] In this embodiment, the sonar image semantic communication method is applied to a network sending end, including: Step S10, obtaining an original sonar image, and semantically encoding the original sonar image according to a semantic network to extract semantic features.

[0027] It should be noted that the original sonar image is an image of sound wave reflection in the water or other media obtained by the sonar equipment. It contains a large amount of raw data and information, usually presented in the form of an image, used to represent underwater terrain, object distribution, etc. The semantic network is a network structure used to extract the semantic information of the original sonar image. Semantic encoding is the process of converting raw data or information into a code or representation with specific semantic meaning, which involves converting the pixel value, texture, shape and other features of the image into a feature vector or label with specific semantic meaning. Semantic features refer to features in the image with specific semantic meanings, including the undulations of the underwater terrain, the shape and size of the object, the intensity of the reflected sound waves, etc.

[0028] It can be understood that the semantic communication method improves the transmission efficiency of communication by extracting semantics from the original signal for more efficient compression and more accurately maintaining the integrity of semantics during transmission. Compared with traditional digital communication methods, this method is more suitable for bandwidth-limited communication scenarios such as sea-satellite-land.

[0029] Step S20, combining the semantic features with the signal-to-noise ratio information of the channel to obtain channel coding output vector data.

[0030] It should be noted that the quality of the sea-satellite-ground communication channel often changes dynamically. In addition to the path loss caused by long-distance propagation, the absorption loss of the atmosphere and weather changes also affect the signal quality. The uncertainty of these external environments requires that the communication system should have the ability to adaptively adjust according to the channel conditions. Combining the semantic features with the signal-to-noise ratio (SNR) information of the channel, resources can be adaptively allocated according to the differences in channel conditions to achieve the best data transmission under limited bandwidth.

[0031] Of course, in order to make the solution more robust in a dynamic channel environment, the step S20 may include: Collect the signal-to-noise ratio information of the current channel; obtain the global statistical information of the semantic feature on the entire feature map based on the global average pooling function, and splice the global statistical information with the signal-to-noise ratio information to obtain context information; transmit the context information to the fully connected neural network, and calculate a set of weight factors according to the semantic features based on the signal-to-noise ratio attention module; multiply the semantic features with the corresponding weight factors respectively to obtain channel coding output vector data.

[0032] It should be noted that global average pooling is a special pooling operation that averages the entire feature map to obtain global statistical information for each channel, which can reduce the dimension of the feature map while retaining important global information. The weight factor is a coefficient used in a neural network to adjust the importance of different features or neurons.

[0033] It should be understood that in the case of poor channel quality (low SNR), certain key features need to be amplified by weight factors to ensure that they are not drowned out by noise during transmission. Conversely, in the case of good channel quality (high SNR), the strength of the features can be appropriately reduced to reduce unnecessary data transmission. The weight factors enable the model to differentiate between different semantic features, improving the transmission quality of important features while potentially reducing the priority of less critical features.

[0034] It is understandable that the use of global average pooling and fully connected networks can effectively reduce computational overhead and achieve rapid coding adjustments. And the generated weighting factors make the model more robust in a dynamic channel environment. For ease of understanding, taking the actual situation as an example, but not limiting this application, in one example, reference Figure 2 , Figure 2 This is the structure diagram of the SNR attention module of this application. In order to cope with the challenge of dynamic changes in channel quality, a method that can adaptively adjust channel coding according to the signal-to-noise ratio (SNR) is proposed. In the process of semantic model optimization, the SNR value range of the simulated channel is set to [1, 13] dB, covering a wide range of scenarios from low SNR to high SNR. By training the neural network in the entire SNR range, the adaptability of the model under different SNR conditions has been significantly improved. Based on the principle of attention mechanism, an SNR attention module is proposed, in which FC is a fully connected layer, and Relu and Sigmoid are activation functions. The SNR attention mechanism can weight the input features, thereby enhancing the model's ability to capture key information. It receives the feature output from the semantic encoding and the SNR information of the channel as input at the same time. Based on these inputs, the SNR attention module calculates a set of weight factors through a neural network, which assign different importance to different features according to the current channel conditions.

[0035] set up represents the semantic encoding feature, where is the number of channels of the feature, is the size of the feature. Similarly, let represents the weighted features generated by the SNR attention module. In order to effectively utilize the input features and adjust according to the signal-to-noise ratio, the SNR attention module first uses a global average pooling function, which can capture the global statistical information of the entire feature map, and then concatenates it with the signal-to-noise ratio to form context information. This context information is fed into a fully connected neural network to generate a weight factor. Finally, by combining the features Multiply it with the weight factor to get the channel coded feature .

[0036] Step S30, compressing the channel coding output vector data based on the quantization module to obtain target integer data, and sending the target integer data to the network receiving end through the channel.

[0037] It should be noted that the bandwidth of the sea-satellite-ground channel is limited, which requires that the size of the data must be reduced during the transmission process. In the semantic encoding process of sonar images, although the features have been mapped into a compact semantic expression form through dimensionality reduction, their data still exists in the form of 32-bit floating point numbers, which takes up a large storage space. The purpose of the quantization module is to quantize and round the low-dimensional semantic vector while maintaining key semantic information, and compress the semantic vector into a smaller bit stream to adapt to bandwidth limitations.

[0038] In one example, the data type of the low-dimensional semantic representation is converted from a 32-bit floating point type to an 8-bit integer type by quantization, reducing the number of transmission bits to one-fourth. The quantization function used is:

[0039] in, is the low-dimensional semantics output by the semantic encoder, is the quantitative result, is the scale parameter to be learned, is the rounding function. Before semantic decoding at the receiving end, the received semantics needs to be dequantized and restored to floating-point data. The dequantization function is:

[0040] in, To restore the vector to floating point data, is a bias term to eliminate part of the additive noise.

[0041] Step S40: The network receiving end restores the received target integer data into the channel coding output vector data based on the quantization factor, and performs channel decoding and semantic decoding on the channel coding output vector data to obtain a target sonar image.

[0042] It should be noted that the target sonar image is not exactly the same as the original sonar image. Before decoding, the received signal usually needs to be preprocessed, such as filtering and amplification, to improve the quality and detectability of the signal. The decoding algorithm can restore the original information vector after channel coding. These information vectors may still contain some redundant information, but most of the noise and interference have been removed.

[0043] In one example, reference Figure 3 , Figure 3 This is a sea-satellite-land link diagram for the transmission of sonar image data for this application. When constructing the sea-satellite-ground communication link, LEO is selected as the data transmission medium for sonar images. The process is described as follows: 1. After obtaining underwater sonar image data, marine operating equipment (such as ships, etc.) uses semantic communication technology to semantically encode the sonar image and upload the data to the low-orbit satellite through a wireless channel. 2. The low-orbit satellite uses a wide communication coverage range to forward the received data to the ground base station. 3. The ground base station uses semantic communication technology to semantically decode the received data to reconstruct the sonar image data.

[0044] In this embodiment, the original sonar image is obtained, and the original sonar image is semantically encoded according to the semantic network to extract semantic features; the semantic features are combined with the signal-to-noise ratio information of the channel to obtain channel coding output vector data; the channel coding output vector data is compressed based on the quantization module to obtain target integer data, and the target integer data is sent to the network receiving end through the channel; the network receiving end restores the received target integer data to the channel coding output vector data based on the quantization factor, and performs channel decoding and semantic decoding on the channel coding output vector data to obtain the target sonar image. By constructing a semantic coding and decoding network, the semantics of the original signal are extracted to capture details more efficiently, and the sonar image is efficiently transmitted back to the land base station using semantic communication technology, and the integrity of the semantics is more accurately maintained during the transmission process, thereby improving the transmission efficiency of the communication.

[0045] Reference Figure 4 , Figure 4 This is a flow chart of the second embodiment of the sonar image semantic communication method of the present application. Based on the above-mentioned first embodiment, the second embodiment of the sonar image semantic communication method of the present application is proposed. In the second embodiment, the step S10 includes: Step S101, obtaining the original sonar image collected by the marine operation equipment.

[0046] It should be noted that marine operation equipment refers to equipment used to perform various operations in the marine environment, including sonar systems, underwater robots and marine survey vessels.

[0047] Step S102, dividing the original sonar image into blocks according to a network hyperparameter setting table preset in the semantic network to obtain small-sized sonar image blocks.

[0048] It should be noted that dividing the original sonar image into small-sized sonar image blocks can better capture the imaging relationship between the target object and the acoustic shadow, thereby improving the model's ability to understand the semantics of the sonar image.

[0049] It should be understood that the current applications of sonar image semantic communication are relatively limited, and the existing image semantic communication network structures are mostly based on convolutional neural networks or Vision Transformer (ViT) to extract semantic information. These network structures are mainly designed for optical images, and have the following limitations when applied to sonar images: the main targets in sonar images are usually small, and there is a lack of design for small-scale target features; the uniqueness of sonar images lies in their acoustic shadow features, which requires the semantic extraction network to capture both global information and local details.

[0050] In one example, based on the imaging characteristics of sonar images, the hyperparameter settings of the Swin-Transformer network are shown in Table 1.

[0051] Table 1 Network hyperparameter settings

[0052] Among them, the patch size is set to 2×2 to meet the needs of semantic feature extraction of small-sized targets in sonar images; the window size is 8×8, which has significant advantages in processing small-sized targets and detailed features in sonar images, and can extract target features more accurately.

[0053] Step S103: transmitting the sonar image block to a multi-layer Swin Transformer module to extract semantic features of the original sonar image.

[0054] It can be understood that the preprocessed and segmented sonar image blocks are used as input, and the semantic features of the image are extracted through multiple levels of Transformer modules using the SwinTransformer structure. Each module contains a self-attention mechanism and a multi-layer perceptron (MLP) to capture features at different scales. This improves the global information understanding and detail capture capabilities of the sonar image.

[0055] For ease of understanding, taking the actual situation as an example, but not limiting the present application, in one example, reference Figure 5 , Figure 5This is a structural diagram of the semantic encoding network of this application. First, the source image is divided into small blocks and embedded into the feature space through the Patch Embedding stage. Then, these image blocks are processed in turn through multiple Swin Transformer modules, each of which is followed by a downsampling step to gradually extract and fuse the multi-scale semantic features of the image. Specifically, the image is first processed by a Swin Transformer module 1, followed by downsampling; then it passes through Swin Transformer module 2 and is downsampled again; this process is repeated three times in Swin Transformer module 3, and each module is followed by a downsampling step; finally, the image is processed by Swin Transformer module 4 to complete the extraction of semantic features and finally output the semantic representation of the image. This process achieves effective encoding of deep semantic information of sonar images through a hierarchical structure and gradual downsampling.

[0056] Furthermore, the global information understanding and detail capturing capabilities of the sonar image are significantly improved. The step S103 may include: The divided sonar image blocks are transmitted as input to a multi-layer Swin Transformer module; in the multi-layer Swin Transformer module, a fixed window multi-head attention mechanism and a moving window multi-head attention mechanism are used alternately to capture the local and global initial semantic features of the sonar image blocks; the initial semantic features are calculated and adjusted through residual connection and layer normalization, and the adjusted initial semantic features are integrated to obtain the semantic features of the original sonar image.

[0057] It should be understood that in the Swin Transformer module, the fixed window multi-head attention mechanism (W-MSA, Window-based Multi-Head Self-Attention) and the moving window multi-head attention mechanism (SW-MSA, Shifted Window Multi-Head Self-Attention) are used alternately. The fixed window multi-head attention mechanism allows the model to capture detailed features in a local area, while the moving window multi-head attention mechanism helps the model to cross the local window and capture a wider range of contextual information.

[0058] For ease of understanding, taking the actual situation as an example, but not limiting the present application, in one example, reference Figure 6 , Figure 6This is the internal structure of the Swin Transformer module in this application. In the figure, a fixed-window multi-head attention module (W-MSA) and a moving-window multi-head attention module (SW-MSA) are used alternately in one module. , after the following calculation, we get :

[0059]

[0060]

[0061]

[0062] Among them, MLP (Multilayer Perceptron) is a multi-layer perceptron structure, and LN (LayerNormalization) is a layer normalization operation. In W-MSA, the image block (Tokens) first undergoes layer normalization (LN), then is processed by a fixed-window multi-head attention mechanism, and then again passes through LN and residual connections to generate the output In SW-MSA, the image patch also first passes through LN, but is then processed by a moving window multi-head attention mechanism, which increases the receptive field by moving the window at different positions, and then passes through LN and residual connection again to generate the output These two mechanisms are used alternately to capture local and cross-window dependencies, thereby extracting multi-scale features of the image.

[0063] Of course, in order to solve the problem of under-fitting of the feature extraction network caused by the scarcity of sonar image data sets, before step S101, the following steps are also included: A synthetic aperture radar data set is selected to be pre-trained on the semantic network to obtain pre-trained network parameters; a fine-tuning method is selected based on transfer learning, and the pre-trained network parameters are adjusted using a preset sonar image data set to obtain target network parameters; the target network parameters are applied to the semantic network to construct the network hyperparameter setting table and the Swin Transformer module.

[0064] It should be understood that transfer learning is an effective way to solve the problem of insufficient training data for deep learning. It uses rich data from other related fields to pre-train the model parameters of the neural network, and then migrates the trained model to the new model to optimize the learning efficiency of the model. SAR (Synthetic Aperture Radar) images are selected as pre-training data. The main reason is that both sonar images and SAR images are grayscale images, and they show certain similarities in shallow features, such as the edge of the target and the distribution of speckle noise. The targets of SAR images and sonar images are mainly small targets.

[0065] In one example, reference Figure 7 and Figure 8 , Figure 7 This is a schematic diagram of the SAR image target for this application. Figure 8 This is a schematic diagram of the forward-looking sonar image target in this application. In the figure, the features of the two images are similar. When using transfer learning to transfer the knowledge of SAR data pre-training to the sonar image processing task, the domain of the image distribution is assumed to be , the task can be expressed as ,in is the label space, is the target prediction function, then transfer learning can be defined as: Given a domain Learning tasks on , the task can be accessed from another domain Learning tasks on Get relevant learning experience. The purpose of transfer learning is to and Acquire knowledge and transfer it to , to improve the prediction function This shows that when the distribution difference between the two image domains is large, from The experience that can be learned is relatively limited, and the performance improvement of transfer learning is not obvious.

[0066] This scheme selects the SAR dataset SARDet-100K for pre-training on the proposed semantic network, and then applies the pre-trained network parameters to sonar images through transfer learning. Using a small amount of sonar images for training can obtain a semantic communication neural network with good generalization performance. The selected sonar image dataset and SAR image dataset information are shown in Table 2. The transfer learning method selects fine-tuning, and the sonar image dataset is used to fine-tune the model parameters based on the pre-trained model parameters. Table 2 Dataset information

[0067] As can be seen from the table, the target categories of both datasets include airplanes and ships. The identity of high-level semantics also reduces the semantic differences between the pre-trained datasets. Therefore, selecting this dataset for feature network pre-training helps the network learn the shallow and deep semantic features of sonar images.

[0068] In this embodiment, the original sonar image collected by the marine operation equipment is obtained; the original sonar image is divided into blocks according to the network hyperparameter setting table preset in the semantic network to obtain small-sized sonar image blocks; the sonar image blocks are transmitted to the multi-layer Swin Transformer module to extract the semantic features of the original sonar image. By introducing the sliding window mechanism, a smaller patch size is used to capture the relationship between the target and the acoustic shadow, and semantic features are extracted from different scales, which significantly improves the global information understanding and detail capture capabilities of the sonar image.

[0069] Reference Fig. 9 , Fig. 9 This is a flow chart of the third embodiment of the sonar image semantic communication method of the present application. Based on the above second embodiment, the third embodiment of the sonar image semantic communication method of the present application is proposed. In this embodiment, the sonar image semantic communication method is applied to a network receiving end, including: Step S40, receiving target integer data sent by a network sending end, wherein the network sending end is used to semantically encode the original sonar image, extract semantic features, and compress the semantic features into target integer data after combining them with the signal-to-noise ratio information of the channel.

[0070] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions or an electronic device capable of implementing the above functions, etc. The following takes the network receiving end as an example to illustrate this embodiment and the following embodiments.

[0071] In a specific example, refer to Fig.10 , Fig.10This is a flowchart of sonar image semantic communication for this application. First, the sonar image is used as the source input, and after the semantic coding stage, the image information is converted into semantic features. Then, these semantic features are quantized through channel coding and combined with the channel state information (SNR) to adapt to channel transmission. In the channel, these quantized data will be interfered by noise. After dequantization, the receiving end reconstructs the semantic information of the image through channel decoding and semantic decoding. Finally, the semantic information is converted back to image form through the semantic decoder to obtain the reconstructed sonar image. Throughout the process, SAR images are used for pre-training to improve the model's learning ability and generalization of sonar image features. This method utilizes the similarities between SAR images and sonar images in shallow features, as well as the common point that they are both mainly small targets, to achieve effective extraction and transmission of semantic information of sonar images when sonar image data is scarce.

[0072] Step S50: restoring the target integer data to channel coding output vector data based on the quantization factor.

[0073] It is understandable that before semantic decoding is performed at the receiving end, the received semantics need to be dequantized and the compressed data need to be restored using the quantization factor. , is the scale parameter to be learned, To restore the vector to floating point data, is a bias term to eliminate part of the additive noise.

[0074] Step S60: channel-decoding the channel-coded output vector data according to the signal-to-noise ratio information to reconstruct the semantic features of the sonar image.

[0075] It can be understood that after dequantization, the features are further processed using a semantic decoding network to reconstruct the semantic features of the sonar image. The semantic decoding network is usually similar to the semantic encoding network in structure, but in the opposite direction. It can recover the high-level semantic information of the image from the compressed semantic features.

[0076] Step S70: semantically decode the semantic features of the sonar image to reconstruct a target sonar image.

[0077] It should be understood that the steps of semantically decoding the semantic features of the sonar image and reconstructing the target sonar image are exactly the opposite of the encoding step, and those are upsampled after each module.

[0078] In one example, reference Fig.11 , Fig.11This is the structure diagram of the semantic decoding network of this application. First, the model receives the semantic information transmitted through the channel. The received semantic information is first processed by Swin Transformer module 4, and then processed by upsampling and Swin Transformer module 3 (repeated three times), upsampling again and Swin Transformer module 2, upsampling and Swin Transformer module 1, and finally the original sonar image is reconstructed through the last upsampling step.

[0079] In a specific scenario, refer to Fig.12 , Fig.13 and Fig.14 , Fig.12 This is a comparison chart of the pre-training effect for this application. Fig.13 and Fig.14 They are the comparison charts of MS-SSIM experimental results and PSNR experimental results, respectively. The performance of the present invention is verified by using AWGN (Additive White Gaussian Noise) simulation channel. In the process of optimizing the semantic model, the SNR value range of the simulation channel is set to [1, 13] dB, covering a wide range of conditions from low signal-to-noise ratio to high signal-to-noise ratio. The evaluation indicators of the proposed semantic communication method verified in this scheme are Peak Signal-to-Noise Ratio (PSNR) and Multi Scale Structural Similarity Index Measure (MS-SSIM). PSNR calculates the pixel value error between the semantically restored image and the original image. The larger the PSNR value, the closer the two images are, which is defined as:

[0080] Among them, m and n are the height and width of the image, is the pixel value of the original image, To restore the pixel values ​​of the image, is the maximum pixel value of the image. If the image pixel value is represented by 8 bits of binary, then =255.

[0081] MS-SSIM is an indicator to measure the similarity between two images. It compares the brightness, contrast and structure of the two images. The closer the MS-SSIM value is to 1, the more similar the two images are. It is defined as:

[0082] in, is a measure of brightness, is a measure of contrast, It is a structural measure. It is the contrast and structure measurement obtained after the image is downsampled j-1 times, and a total of M-1 downsampling times.

[0083] First, we verify the effectiveness of SAR images as pre-training data for sonar image tasks. We select the official training set of the optical image dataset COCO2014 as a comparison, which contains 82,783 images in 80 categories. Fig.12 As shown in the figure, the horizontal axis represents the SNR value of the simulated channel during the test, and the vertical axis represents the PSNR result. As can be seen from the figure, when tested on the sonar image dataset, the network model pre-trained with the SAR dataset achieved better PSNR results on different channel SNRs.

[0084] In order to verify the effectiveness of the proposed sonar image semantic communication method, the proposed method is compared with the Deep Joint Source-Channel Coding (Deep-JSCC) scheme based on SwinTransformer and the traditional separate source channel coding scheme BPG+LDPC (Better Portable Graphics+Low-Density Parity-Check, BPG image compression technology and LDPC error correction code). The experimental results are shown in Figure 2. Fig.13 and Fig.14 As shown in the figure, the results show that the proposed method is superior to the traditional digital communication method BPG+LDPC under various signal-to-noise ratio (SNR) conditions, especially under low signal-to-noise ratio conditions, where the latter can no longer effectively restore the image at the receiving end. The Deep-JSCC method takes fixed SNR conditions into consideration during optimization, but its stability under different SNR conditions is poor, and it can only achieve good communication performance when the channel conditions are similar to those during model training. In contrast, the method proposed in this scheme shows higher PSNR and MS-SSIM results under different SNR conditions, proving that it has better adaptability to dynamic changes in channel quality.

[0085] The compression rate is defined as the ratio of the number of transmitted bytes to the number of source bytes to measure the efficiency of data transmission. The smaller the compression rate, the fewer bytes are required to send the same source. According to the experimental data in Table 3, the sonar image communication method significantly reduces the number of required transmission bits while maintaining communication quality, and shows higher transmission efficiency than traditional communication methods. In addition, after adding the quantization module, the method further reduces the number of transmitted bytes while still maintaining excellent transmission quality. This result shows that the method is suitable for communication environments with limited channel bandwidth and can achieve efficient data transmission under limited bandwidth resources.

[0086] Table 3 PSNR performance comparison of different compression rate methods

[0087] In this embodiment, the target integer data sent by the network transmitter is received, and the network transmitter is used to semantically encode the original sonar image, extract semantic features, and compress the semantic features into target integer data after combining them with the signal-to-noise ratio information of the channel, and restore the target integer data into channel coding output vector data based on the quantization factor; channel-decode the channel coding output vector data according to the signal-to-noise ratio information to reconstruct the semantic features of the sonar image; and semantically decode the semantic features of the sonar image to reconstruct the target sonar image. By constructing a semantic encoding and decoding network, the received target data is decoded, the semantic features are restored, and the image is reconstructed, thereby realizing the rapid return of large amounts of ocean data such as sonar images or videos. The accuracy of image transmission is ensured.

[0088] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the sonar image semantic communication method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0089] This application also provides a sonar image semantic communication device, please refer to Fig.15 , the sonar image semantic communication device comprises: The feature extraction module 10 is used to obtain the original sonar image, and semantically encode the original sonar image according to the semantic network to extract semantic features; A data generation module 20, used to combine the semantic features with the signal-to-noise ratio information of the channel to obtain channel coding output vector data; A data transmission module 30, configured to compress the channel coding output vector data based on the quantization module to obtain target integer data, and send the target integer data to a network receiving end through a channel; The image reconstruction module 40 is used to restore the received integer data to the channel coding output vector data based on the quantization factor, and perform channel decoding and semantic decoding on the channel coding output vector data to obtain a target sonar image.

[0090] The sonar image semantic communication device provided by the present application adopts the sonar image semantic communication method in the above embodiment, which can solve the technical problems of limited bandwidth of the link channel, dynamic changes in channel quality, and poor communication quality when a large amount of ocean data such as sonar images or videos is transmitted back to the distant sea using low-orbit satellites as relays to realize the sea surface-satellite-land link. Compared with the prior art, the beneficial effects of the sonar image semantic communication device provided by the present application are the same as the beneficial effects of the sonar image semantic communication method provided by the above embodiment, and the other technical features of the sonar image semantic communication device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0091] The present application provides a sonar image semantic communication device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the sonar image semantic communication method in the above-mentioned embodiment 1.

[0092] Reference below Fig.16 , which shows a schematic diagram of the structure of a sonar image semantic communication device suitable for implementing the embodiment of the present application. The sonar image semantic communication device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.16 The sonar image semantic communication device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0093] like Fig.16As shown, the sonar image semantic communication device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. In RAM 1004, various programs and data required for the operation of the sonar image semantic communication device are also stored. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the sonar image semantic communication device to communicate with other devices wirelessly or by wire to exchange data. Although the sonar image semantic communication device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0094] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0095] The sonar image semantic communication device provided by the present application adopts the sonar image semantic communication method in the above embodiment, which can solve the technical problems of limited bandwidth of the link channel, dynamic changes in channel quality, and poor communication quality when a large amount of ocean data such as sonar images or videos is transmitted back to the distant sea ocean data using low-orbit satellites as relays to realize the sea surface-satellite-land link. Compared with the prior art, the beneficial effects of the sonar image semantic communication device provided by the present application are the same as the beneficial effects of the sonar image semantic communication method provided by the above embodiment, and the other technical features of the sonar image semantic communication device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0096] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0097] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0098] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the sonar image semantic communication method in the above-mentioned embodiment.

[0099] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0100] The computer-readable storage medium may be included in the sonar image semantic communication device; or may exist independently without being assembled into the sonar image semantic communication device.

[0101] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the sonar image semantic communication device, the sonar image semantic communication device executes the sonar image semantic communication method described above.

[0102] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0103] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0104] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0105] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned sonar image semantic communication method, and can solve the technical problems of limited bandwidth of the link channel, dynamic changes in channel quality, and poor communication quality when a large amount of ocean data such as sonar images or videos is transmitted back to the distant sea ocean data using low-orbit satellites as relays to realize the sea surface-satellite-land link. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the sonar image semantic communication method provided by the above-mentioned embodiment, and will not be repeated here.

[0106] The present application also provides a computer program product, including a computer program, which implements the steps of the sonar image semantic communication method as described above when executed by a processor.

[0107] The computer program product provided by the present application can solve the technical problems of limited bandwidth of the link channel, dynamic changes in channel quality, and poor communication quality when a large amount of ocean data such as sonar images or videos is transmitted back to the distant sea using low-orbit satellites as relays to realize the sea surface-satellite-land link. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the sonar image semantic communication method provided by the above-mentioned embodiment, and will not be elaborated here.

[0108] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A sonar image semantic communication method, characterized in that: The sonar image semantic communication method is applied to a network sending end, comprising: Acquire an original sonar image, and perform semantic encoding on the original sonar image according to a semantic network to extract semantic features; Combining the semantic features with the signal-to-noise ratio information of the channel to obtain channel coding output vector data; Compressing the channel coding output vector data based on the quantization module to obtain target integer data, and sending the target integer data to a network receiving end through a channel; The network receiving end restores the received target integer data into the channel coding output vector data based on the quantization factor, and performs channel decoding and semantic decoding on the channel coding output vector data to obtain the target sonar image.

2. The sonar image semantic communication method according to claim 1, characterized in that: The step of obtaining the original sonar image, and semantically encoding the original sonar image according to the semantic network to extract semantic features includes: Obtain original sonar images collected by marine operating equipment; Dividing the original sonar image into blocks according to a network hyperparameter setting table preset in the semantic network to obtain small-sized sonar image blocks; The sonar image block is transmitted to a multi-layer Swin Transformer module to extract the semantic features of the original sonar image.

3. The sonar image semantic communication method according to claim 2, characterized in that: The step of transmitting the sonar image block to a multi-layer Swin Transformer module to extract the semantic features of the original sonar image comprises: The divided sonar image blocks are sent as input to the multi-layer Swin Transformer module; In the multi-layer Swin Transformer module, a fixed-window multi-head attention mechanism and a moving-window multi-head attention mechanism are alternately used to capture the local and global initial semantic features of the sonar image block; The initial semantic features are calculated and adjusted through residual connection and layer normalization, and the adjusted initial semantic features are integrated to obtain the semantic features of the original sonar image.

4. The sonar image semantic communication method according to claim 2, characterized in that: Before the step of obtaining the original sonar image collected by the marine operation equipment, the method further includes: Selecting a synthetic aperture radar data set to perform pre-training on the semantic network to obtain pre-training network parameters; Selecting a fine-tuning method based on transfer learning, adjusting the pre-trained network parameters using a preset sonar image data set to obtain target network parameters; The target network parameters are applied in the semantic network to construct the network hyperparameter setting table and the SwinTransformer module.

5. The sonar image semantic communication method according to claim 1, characterized in that: The step of combining the semantic feature with the signal-to-noise ratio information of the channel to obtain channel coding output vector data comprises: Collect the signal-to-noise ratio information of the current channel; Acquire global statistical information of the semantic feature on the entire feature map based on a global average pooling function, and concatenate the global statistical information with the signal-to-noise ratio information to obtain context information; The context information is fed into a fully connected neural network, and a set of weight factors is calculated based on the semantic features based on a signal-to-noise ratio attention module; The semantic features are multiplied by the corresponding weight factors respectively to obtain channel coding output vector data.

6. A sonar image semantic communication method, characterized in that: The sonar image semantic communication method is applied to a network receiving end, comprising: Receiving target integer data sent by a network sending end, the network sending end is used to perform semantic encoding on the original sonar image, extract semantic features, and compress the semantic features into target integer data after combining them with signal-to-noise ratio information of the channel; Restoring the target integer data into channel coding output vector data based on the quantization factor; Channel decoding is performed on the channel coding output vector data according to the signal-to-noise ratio information to reconstruct the semantic features of the sonar image; The semantic features of the sonar image are semantically decoded to reconstruct the target sonar image.

7. A sonar image semantic communication device, characterized in that: The device comprises: A feature extraction module is used to obtain an original sonar image, and to perform semantic encoding on the original sonar image according to a semantic network to extract semantic features; A data generation module, used for combining the semantic features with the signal-to-noise ratio information of the channel to obtain channel coding output vector data; A data transmission module, used for compressing the channel coding output vector data based on the quantization module to obtain target integer data, and sending the target integer data to a network receiving end through a channel; The image reconstruction module is used to restore the received integer data to the channel coding output vector data based on the quantization factor, and perform channel decoding and semantic decoding on the channel coding output vector data to obtain a target sonar image.

8. A sonar image semantic communication device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sonar image semantic communication method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the sonar image semantic communication method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the sonar image semantic communication method according to any one of claims 1 to 6 are implemented.

Images