Semantic image transmission method and system based on lightweight network

Abstract

The invention discloses a semantic image transmission method and system based on a lightweight network. The method comprises the following steps: acquiring an input image; based on a Swin Transform architecture, quantization operation and non-linear operation integer approximation are introduced, and an image transmission network is obtained; the image transmission network comprises an encoder, a decoder and a channel modulation module; and transmitting the input image based on the image transmission network. The system comprises a memory and a processor used for executing the semantic image transmission method based on the lightweight network. By using the method, the calculation complexity and the energy consumption of the model are remarkably reduced, so that the method is more suitable for being deployed on edge equipment with limited calculation resources and energy consumption, and is more suitable for a high-speed image transmission scene. The method can be widely applied to the field of data transmission.

Description

Technical Field

[0001] The present invention relates to the field of data transmission, and in particular to a semantic image transmission method and system based on a lightweight network. Background Art

[0002] Autonomous driving technology has attracted widespread attention due to its potential to improve road safety and optimize traffic efficiency. Despite advanced perception systems, autonomous vehicles still pose potential safety risks due to the limited field of view of a single vehicle. Extending the field of view through image sharing can enhance driving safety, which makes efficient image data transmission essential. However, image transmission faces the problem of high latency when bandwidth is limited.

[0003] To address this challenge, current research focuses on compressing the amount of transmitted data. Compared with the traditional method of designing source coding and channel coding separately, joint source channel coding (JSCC) can reduce unnecessary redundant work, while semantic communication methods further reduce transmission delay by selectively propagating the transmission content. These methods use deep learning models to extract image features, among which the swintransformer model with outstanding performance in ViTs (Visiontransformers) is used as a feature extractor for semantic communication, providing ideal image transmission quality.

[0004] However, despite the superior performance of SwinTransformer in capturing multi-scale information and complex features, its computational complexity and model parameters are huge, and its application on edge devices for semantic communication still faces problems such as battery life, device space, and video memory requirements. These problems limit its widespread deployment in practical use. Summary of the invention

[0005] In view of this, in order to solve the technical problem that the swintransformer model calculation complexity and large model parameters used in the existing image transmission method cannot be widely applied to various scenarios, the present invention proposes a semantic image transmission method based on a lightweight network, and the method comprises the following steps:

[0006] Get the input image;

[0007] Based on the SwinTransformer architecture, quantization operations for linear operations and integer approximation operations for nonlinear operations are introduced to obtain an image transmission network that only uses integer calculations.

[0008] The image transmission network includes an encoder, a decoder and a channel modulation module;

[0009] The compressed feature map of the input image is transmitted based on the image transmission network.

[0010] In some embodiments, the step of transmitting the input image based on the image transmission network specifically includes:

[0011] Performing encoding processing on the input image based on the encoder to obtain a feature map;

[0012] Modulating the characteristic feature graph based on the channel modulation module to obtain a modulated feature graph;

[0013] Transmitting the modulated characteristic graph in a wireless channel;

[0014] The decoder receives the modulated feature map, performs demodulation processing and image restoration, and obtains a final image.

[0015] In some embodiments, the step of encoding the input image based on the encoder to obtain a feature map specifically includes:

[0016] Based on the quantized convolution layer, the input image is divided into image blocks and linearly embedded to obtain format-converted data;

[0017] Performing feature extraction on the data converted into the format through a multi-layer improved Swin Transformer Block to obtain a feature map;

[0018] The multi-layer improved Swin Transformer Block includes a quantized matrix multiplication layer, a quantized linear layer, an integer-approximated LayerNorm, an integer-approximated GELU, and an integer-approximated Softmax.

[0019] In some embodiments, an image block merging layer is also included between the Swin Transformer Blocks.

[0020] Through this preferred step, a patch merging layer is inserted between the Swin Transformer Block (STB), which gradually reduces the spatial dimension of the feature map and enhances the feature representation capability.

[0021] In some embodiments, the step of modulating the characteristic feature graph based on the channel modulation module to obtain the modulated feature graph specifically includes:

[0022] Accept the signal-to-noise ratio of the current channel state;

[0023] Based on the signal-to-noise ratio, a scaling factor is generated by sequentially performing a multi-layer quantized linear layer, a ReLU operation of integer approximation, and a Sigmoid operation of tail integer approximation;

[0024] The characteristic feature map is modulated according to the scaling signal to obtain a modulated feature map.

[0025] In some embodiments, it also includes:

[0026] The modulated feature map is compressed through a quantized linear layer.

[0027] Through this preferred step, the amount of data can be reduced, and the transmission bandwidth and storage requirements can be lowered.

[0028] In some embodiments, during the quantization operation, the quantization factor S x and the quantized integer representation I x As shown below:

[0029]

[0030]

[0031] Among them, X represents the input, X max represents the maximum quantization range of the input, k represents the number of quantization bits, Indicates that the value is rounded to the nearest integer, and clip means clipping X to the specified value range.

[0032] In some embodiments, the integer-approximated Softmax representation is as follows:

[0033]

[0034] in, Represents the quantized integer representation corresponding to the i-th index value, The value of d is the dimension of the hidden feature, express The quantized integer value obtained after integer operation approximation, the specific conversion steps are as follows: Since it is simpler to perform a shift operation on an integer calculation unit than to perform an exponential operation directly, the operation needs to be converted into an exponential form with base 2.

[0035] Since it is easier to perform shift operations on integer computing units than exponential operations, we need to Convert to base 2 operation and use the base conversion formula to process the input exponential operation:

[0036]

[0037] Where log2e can be approximated as a binary number

[0038] log2e≈(1.0111)2=(1.1000)2-(0.0001)2

[0039] So we have:

[0040]

[0041] in

[0042] because As the exponent term is not necessarily an integer, it may not be directly used for shift operations. Therefore, it can be divided into integer parts to find non-negative integers r and q such that And S x ·(-r)∈(-1,0].

[0043] Since when x∈(-1,0], so:

[0044]

[0045] Can be converted to S x [-(r>>1)+I1], where

[0046] Combining the above, we can get:

[0047]

[0048] Since both the Softmax numerator and denominator contain S x , the integer implementation of the division operation can be expressed as The inputs of this integer division function are the dividend, divisor and number of quantization bits. It is calculated as follows:

[0049]

[0050] The specific steps are explained as follows: In order to reduce the loss of precision in the integer division process, first change the divisor With the larger integer 2 M Multiply to expand the range of values, and then add the result to After multiplication, right shift (M+1-k) to adapt to the target bit width. In addition, since this process is equivalent to enlarging 2 k-1 times, scaling factor

[0051] In some embodiments, the integer approximation of the Sigmoid function is expressed as follows:

[0052]

[0053] The present invention also proposes a semantic image transmission system based on a lightweight network, comprising:

[0054] At least one transmitting end, including a processor and a memory, for storing and executing a program to extract semantic information of an image and to compress and transmit the image;

[0055] at least one receiving end, comprising a processor and a memory, for storing and executing a program to decode and restore received semantic information;

[0056] A communication channel, used to connect the sending end and the receiving end to transmit the semantic information;

[0057] When the at least one program is executed by the at least one processor, the at least one processor implements the semantic image transmission method based on a lightweight network as described above.

[0058] Based on the above scheme, the present invention provides a semantic image transmission method and system based on a lightweight network. By introducing quantization operations and nonlinear integer approximation operations, while ensuring the quality of image transmission, the calculation complexity and video memory requirements are significantly reduced, which is more conducive to practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a flowchart of the steps of a semantic image transmission method based on a lightweight network of the present invention;

[0060] Figure 2 Comparative performance analysis of semantic image transmission methods based on different channel conditions. DETAILED DESCRIPTION

[0061] The deep learning model required for semantic communication has a huge number of parameters and computational complexity. Factors such as hardware cost, deployment space, and battery life jointly limit the deployment of semantic communication on edge devices. In summary, existing technical methods still face many challenges in the practical application of semantic communication, such as high computational complexity, large memory requirements, and high energy consumption, especially when deployed on edge devices.

[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] It should be noted that, for the convenience of description, only the parts related to the relevant invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0064] It should be understood that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0065] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0066] In the description of the embodiments of the present application, "plurality" means two or more than two. The following terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0067] In addition, flow charts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0068] Reference Figure 1 , is a flowchart of an optional example of a semantic image transmission method based on a lightweight network proposed in the present invention. The method can be applied to a computer device. The semantic image transmission method proposed in this embodiment may include but is not limited to the following steps:

[0069] Step S1, obtaining an input image;

[0070] Step S2, based on the Swin Transformer architecture, introduce quantization operation and nonlinear operation integer approximation to obtain an image transmission network;

[0071] The image transmission network includes an encoder, a decoder and a channel modulation module;

[0072] Step S3: Transmit the input image based on the image transmission network.

[0073] In this embodiment, a lightweight semantic image transfer network (LSIT) is proposed to solve the high computational complexity and high video memory problems of full-precision Swin Transformer for feature extraction, especially for edge devices with limited computing resources and battery life. The network adopts an improved Swin Transformer architecture and is divided into three parts: encoder, decoder, and channel modulation module.

[0074] In some feasible embodiments, the step S3 specifically includes:

[0075] S3.1, encoding the input image based on the encoder to obtain a feature map;

[0076] S3.2, modulating the characteristic feature graph based on the channel modulation module to obtain a modulated feature graph;

[0077] S3.3, transmitting the modulated characteristic graph in a wireless channel;

[0078] S3.4. Receive the modulated feature map based on the decoder, perform demodulation processing and image restoration, and obtain a final image.

[0079] In some feasible embodiments, the step S3.1 specifically includes:

[0080] In the encoder part, LSIT first uses a quantized convolutional layer (QConv2d) to perform image block division and linear embedding steps to convert the image into a format that can be processed by deep learning. Next, we use a multi-layer improved STB (SwinTransformer Block) for feature extraction. In order to reduce the spatial dimension of the feature map and enhance the feature representation capability, the encoder intersperses image block merging layers (patch merging) between STBs to gradually reduce the spatial dimension of the feature map and enhance the feature representation capability.

[0081] In some feasible embodiments, the step S3.2 specifically includes:

[0082] After being processed by the encoder, the output feature map is sent to the channel modulation module (channel Modnet). This module modulates the feature map according to the channel state to adapt to different wireless channel conditions.

[0083] In some feasible embodiments, it also includes:

[0084] The modulated feature map is compressed through a quantized linear layer to reduce the amount of data, lower the transmission bandwidth and storage requirements, and is sent to the wireless channel after power normalization.

[0085] In some feasible embodiments, the step S3.4 specifically includes:

[0086] After receiving the data, the decoder first uses the quantized linear layer to decompress the feature map, and then demodulates the feature map through the channel modulation network. The demodulated feature map is restored through the Swin Transformer module, and then upsampled through the block splitting layer. After multi-stage processing, the final restored image is obtained.

[0087] In some feasible embodiments, the encoder in step S3.1 refers to a multi-layer improvement, SwinTransformer Block, which specifically includes:

[0088] The improved STB introduces quantization operations and integer approximation based on the traditional structure, uses the quantization matrix multiplication layer (Q_Matmul) and the quantization linear layer for quantization of linear operations, and also includes integer approximation LayerNorm, GELU and Softmax.

[0089] These improvements enable the STB to efficiently perform feature extraction or recovery on low-power integer computing units.

[0090] In some feasible embodiments, the channel modulation module specifically includes:

[0091] The channel modulation module (SNR Modulation) first receives the signal-to-noise ratio (SNR) of the current channel state as input, passes through three quantized linear layers and integer approximation ReLU operations, and finally generates a scaling factor of the same size as the input signal through integer approximation Sigmoid operations. These scaling factors are used to subsequently adjust the signal to make it more suitable for communication under specific channel conditions, thereby optimizing signal transmission performance. The channel modulation module takes the feature map extracted by the encoder or the feature map received after decompression from the wireless channel as input, and passes through 8 quantized linear layers in succession. The feature values ​​between every two layers are element-wise multiplied with the scaling factor output by SNR Modulation. After the signal passes through the last quantized linear layer, an integer approximation Sigmoid operation is performed to generate a scaling factor matrix of the same size as the input feature map. This scaling factor matrix is ​​element-wise multiplied with the initial input feature map to obtain the final output feature map.

[0092] Through this embodiment, it is ensured that the signal is transmitted stably and efficiently in the wireless channel, or the original signal is restored from the actual channel as accurately as possible.

[0093] Based on the above scheme, quantization technology is an important part of the LSIT network, which converts model parameters and activation values ​​from high precision (usually floating point numbers) to low precision integer representation. LSIT implements integer quantization of weights and activation values ​​in all linear operations in semantic communication scenarios, including quantization of activation values ​​(Q_Act), as well as quantization of weight parameters of linear layers, convolutional layers, and matrix multiplication layers. Q_Act is attached to all linear or nonlinear layers to quantize the output activation value from Int32 back to Int8 to ensure that the output of subsequent processes does not exceed the range. In order to solve the problem of non-differentiable quantization, LSIT uses a straight-through estimator for back propagation. In the specific implementation of integer quantization, the quantization scaling factor must be calculated first. Since LSIT chooses symmetric uniform quantization, assuming the input is X, the quantization range is [-X max ,X max ]. Therefore, when the number of quantization bits is k, the quantization factor S X It is expressed as:

[0094]

[0095] Among them, S X represents the quantization factor, X represents the input, and X max represents the maximum quantization range of the input, and k represents the number of quantization bits.

[0096] In order to ensure that S X After quantization to integers, we need to evenly clip the data and divide it by the scaling factor, and use the rounding operation to approximate it to an integer. X for:

[0097]

[0098] Since the scaling factor will remain unchanged once determined during the training phase, and LSIT considers inference on integer compute units, the scaling factor needs to be converted to binary number (DN) format, thereby converting the floating-point scaling factor to a format usable by integer compute units:

[0099]

[0100] Where b and c are positive integers. Therefore, by performing integer multiplication with b and shifting right by c bits, we can implement integer computation of the linear layer.

[0101] The quantization of the linear layer, convolutional layer, and matrix multiplication layer is similar. Taking the linear layer Y = WX + B as an example, after quantizing the activation value of the previous layer, we get I X and S X , and quantize the weights of the linear layer to obtain the quantized integer value and scaling factor I Wand S W The bias scaling factor defaults to the product of the weight and activation scaling factors. Thus:

[0102] Y=S Y I Y =WX+B=(S W ·S X )((I W I X +I B ))

[0103] Then there are:

[0104]

[0105] The corresponding implementation on the integer computing unit is:

[0106] I Y =[b′(I W I X +I B )]>>c'

[0107] Where >>c' means right shift by c' bits.

[0108] The second part is the integer approximation of the nonlinear layer:

[0109] In LSIT, nonlinear operation layers such as ReLU, GELU, Sigmoid, Softmax, and LayerNorm are approximated by integer operations to solve the problem of non-homogeneity of nonlinear operation layers. For the ReLU layer, since it is linear when the input is greater than 0, only the non-negative part needs to be retained by quantization. GELU, Sigmoid, and Softmax contain nonlinear subfunctions exp. Since it is easier to perform shift operations on integer computing units than exponential operations, it is necessary to Convert to base 2 operation and use the base conversion formula to process the input exponential operation:

[0110]

[0111] Where log2e can be approximated as a binary number

[0112] log2e≈(1.0111)2=(1.1000)2-(0.0001)2

[0113] So we have:

[0114]

[0115] in

[0116] because As the exponent term is not necessarily an integer, it may not be directly used for shift operations. Therefore, it can be divided into integer parts to find non-negative integers r and q such that And S x ·(-r)∈(-1,0].

[0117] Since when x∈(-1,0], so:

[0118]

[0119] Can be converted to S x [-(r>>1)+I1], where

[0120] Combining the above, we can get:

[0121]

[0122] Since both the Softmax numerator and denominator contain S x , the integer implementation of the division operation can be expressed as The inputs of this integer division function are the dividend, divisor and number of quantization bits. It is calculated as follows:

[0123]

[0124] The specific steps are explained as follows: In order to reduce the loss of precision in the integer division process, first change the divisor With the larger integer 2 M Multiply to expand the range of values, and then add the result to After multiplication, right shift (M+1-k) to adapt to the target bit width. In addition, since this process is equivalent to enlarging 2 k-1 times, scaling factor

[0125] For the Sigmoid function, it can be expressed as:

[0126]

[0127] Its form is similar to Softmax, so it can also be approximated using Intexp and Intdiv. According to existing papers, the GELU function can be approximately expressed as:

[0128] GELU(x)=x·σ(1.702x)

[0129] Among them, 1.702 can be converted into binary form, multiplied by the input x and used as the input of the Sigmoid function, and finally the calculation result can be multiplied by x.

[0130] In the inference phase, the integer arithmetic unit allows LayerNorm to directly calculate the mean and variance of the data, but does not support the square root of the standard deviation through the algorithm. Solving var(x) = 0 is equivalent to finding F(I i )=var(x)-I i The roots of , by Newton's iteration method:

[0131]

[0132] Therefore, Layernorm can be expressed as:

[0133] LayerNorm(S X ,I X )=Intdiv[S X I X -mean(S X I X ),I i+1 ,k]

[0134] When i=10, the ideal effect can be achieved.

[0135] Based on the above scheme, integer quantization is performed on the linear layer and the nonlinear layer, which significantly reduces the computational complexity and energy consumption of the model, making it more suitable for deployment on edge devices with limited computing resources and energy consumption. This method can reduce the dependence on complex calculations and improve the operating efficiency of the model on low-cost hardware. In the integer quantization process of the linear layer, the present invention greatly reduces the computational complexity and storage requirements by converting the weights and activation values ​​of the model from floating point numbers to low-precision integer representations. In the processing of the nonlinear layer, the present invention proposes an integer approximation method, which converts the operation of nonlinear functions into integer operations and shift operations, thereby avoiding frequent integer and floating point conversions in the reasoning process, thereby improving the operation efficiency. In addition, the application of the quantization-aware training method ensures that the model can still maintain high precision in the reasoning stage after integer quantization, ensuring the stable performance of the semantic communication system under different channel conditions. This series of technical means jointly support the efficient operation of integer quantization models on hardware that only supports integer operations, significantly reducing deployment costs and energy consumption, making semantic communication systems more feasible and extensive in practical applications.

[0136] Since the scheme only needs to use integer operations during the model inference process, the hardware requirements are relatively low. This means that it can run efficiently on hardware that only supports integer operations, reducing deployment costs and the demand for device space, and helping to promote it in a wider range of application scenarios. Despite the use of integer quantization methods, the scheme can still maintain a transmission quality almost equivalent to that of the full-precision model under different image resolutions, channel conditions and compression ratios, and sometimes even performs better. This shows that the scheme can effectively guarantee the quality of semantic image transmission under bandwidth constraints. This embodiment uses the peak signal-to-noise ratio (PSNR) as an evaluation indicator to compare the transmission quality of the lightweight version LSIT, the corresponding non-lightweight full-precision model WITT, and another semantic communication model JSCC. In order to reflect the robustness of the lightweight scheme, this embodiment has carried out extensive experimental comparisons on images of different resolutions, signal-to-noise ratios, channel types and compression rates.

[0137] In terms of image resolution, this embodiment trained and tested the model on the CIFAR10 dataset to observe its effect on low-resolution images; it was trained on the DIV2K dataset and tested on the KODAK dataset to measure its effect on high-resolution images. In addition, the embodiment is consistent with the non-lightweight full-precision model WITT in model architecture, and the number of levels varies with the resolution of the training image. For low-resolution images, the embodiment uses 2 levels, set to [N1, N2] = [2, 4], [C1, C2] = [128, 256], and the window size is 2. For high-resolution images, we use 4 levels, set to [N1, N2, N3, N4] = [2, 2, 6, 2], [C1, C2, C3, C4] = [128, 192, 256, 320], and the window size is 8.

[0138] In terms of channel types, LSIT conducted experiments on AWGN and Rayleigh channels under different signal-to-noise ratio conditions, and performed simulations under different compression ratio conditions (cbr).

[0139] The experimental results are as follows Figure 2 shown. Figure 2 (A) and Figure 2 (B) The comparison shows that LSIT has achieved good performance under different channel types and different signal-to-noise ratio conditions; Figure 2 (A) and Figure 2 (C) The comparison shows that LSIT maintains excellent performance at different image resolutions. Figure 2(D) shows the robust transmission performance of LSIT under different compression ratios. Experiments show that under different image resolutions, channel types, and compression ratios, when the signal-to-noise ratio is low, the difference in PSNR can always be maintained at around 0.5% of the full-precision model, and sometimes even better. In addition, compared with the full-precision model, the pure integer quantization model not only maintains stable image transmission quality and ensures the function of semantic communication, but also has significant cost advantages: it has lower cost in terms of computing unit area and energy consumption, as well as lower memory usage and computational complexity. Figure 2 In this paper, we are mainly concerned that the lightweight method will not cause serious degradation problems to the performance of the original model. After proving the feasibility and robustness of lightweighting, we need to provide indicators that lightweighting is beneficial to edge devices. Taking the full-precision model WITT before lightweighting and the LSIT after lightweighting as comparison objects, the number of bits (W / A) of weight parameters and activation values ​​are 32 bits and 8 bits respectively. In the inference stage, the lightweight model can also perform inference functions with less storage occupancy. In addition, the number of bit operations (BOPs) is used to measure the computational complexity during the inference of deep learning models, and is positively correlated with the power consumption of the model. From the above data, it can be seen that the lightweight LSIT has good improvements in memory usage, computational complexity, and energy consumption.

[0140] Table 1 Performance comparison between lightweight model LSIT and full-precision model WITT on edge devices

[0141] method W / A Memory usage / MB <![CDATA[BOPs / 10 12 ]]> Best PSNR performance WITT 32 / 32 4033 202.8 29.65 LSIT 8 / 8 1008 12.7 29.47

[0142] The current research mainly optimizes the image source process of autonomous driving target recognition, which enables the visual blind spot area to be acquired in a more cost-friendly way for the subsequent target recognition process. In addition, the current work considers the transmission of image data, and the subsequent consideration is to perform lightweight semantic communication on text data to achieve multimodal processes. In this way, multimodal fusion autonomous driving target recognition based on quantitative optimization can be achieved.

[0143] A semantic image transmission system based on a lightweight network, comprising:

[0144] At least one transmitting end, including a processor and a memory, for storing and executing a program to extract semantic information of an image and to compress and transmit the image;

[0145] at least one receiving end, comprising a processor and a memory, for storing and executing a program to decode and restore received semantic information;

[0146] A communication channel, used to connect the sending end and the receiving end to transmit the semantic information;

[0147] When the at least one program is executed by the at least one processor, the at least one processor implements the semantic image transmission method based on a lightweight network as described above.

[0148] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0149] A storage medium stores processor-executable instructions, which are used to implement a semantic image transmission method based on a lightweight network when executed by the processor.

[0150] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0151] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A semantic image transmission method based on a lightweight network, characterized in that: The following steps are involved: Get the input image; Based on the Swin Transformer architecture, quantization operations and integer approximation of nonlinear operations are introduced to obtain an image transmission network; The image transmission network includes an encoder, a decoder and a channel modulation module; The input image is transmitted based on the image transmission network.

2. The method for transmitting semantic images based on a lightweight network according to claim 1, characterized in that: The step of transmitting the input image based on the image transmission network specifically includes: Performing encoding processing on the input image based on the encoder to obtain a feature map; Modulating the characteristic feature graph based on the channel modulation module to obtain a modulated feature graph; Transmitting the modulated characteristic graph in a wireless channel; The decoder receives the modulated feature map, performs demodulation processing and image restoration, and obtains a final image.

3. The method for transmitting semantic images based on a lightweight network according to claim 2, characterized in that: The step of encoding the input image based on the encoder to obtain a feature map specifically includes: Based on the quantized convolution layer, the input image is divided into image blocks and linearly embedded to obtain format-converted data; Performing feature extraction on the data converted into the format through a multi-layer improved Swin Transformer Block to obtain a feature map; The multi-layer improved Swin Transformer Block includes a quantized matrix multiplication layer, a quantized linear layer, an integer-approximated LayerNorm, an integer-approximated GELU, and an integer-approximated Softmax.

4. The method for transmitting semantic images based on a lightweight network according to claim 3, characterized in that: An image block merging layer is also included between the SwinTransformer Blocks.

5. The method for transmitting semantic images based on a lightweight network according to claim 4, characterized in that: The step of modulating the characteristic feature graph based on the channel modulation module to obtain the modulated feature graph specifically includes: Accept the signal-to-noise ratio of the current channel state; Based on the signal-to-noise ratio, a scaling factor is generated by sequentially performing a quantized linear layer, an integer-approximated ReLU operation, and an integer-approximated Sigmoid operation; The characteristic feature map is modulated according to the scaling signal to obtain a modulated feature map.

6. The method for transmitting semantic images based on a lightweight network according to claim 5, characterized in that: Also includes: The modulated feature map is compressed through a quantized linear layer.

7. The method for transmitting semantic images based on a lightweight network according to claim 6, characterized in that: In the quantization operation, the quantization factor S x and the quantized integer representation I x As shown below: Among them, X represents the input, X max represents the maximum quantization range of the input, k represents the number of quantization bits, Indicates that the value is rounded to the nearest integer, and clip means clipping X to the specified value range.

8. The method for transmitting semantic images based on a lightweight network according to claim 7, characterized in that: The integer approximation Softmax is expressed as follows: in, Represents the quantized integer representation corresponding to the i-th index value, The value of d is the dimension of the hidden feature, express The quantized integer value obtained after integer operation approximation.

9. The method for transmitting semantic images based on a lightweight network according to claim 8, characterized in that: The integer approximation of the Sigmoid function is expressed as follows:

10. A semantic image transmission system based on a lightweight network, characterized in that: include: At least one transmitting end, including a processor and a memory, for storing and executing a program to extract semantic information of an image and to compress and transmit the image; at least one receiving end, comprising a processor and a memory, for storing and executing a program to decode and restore received semantic information; A communication channel, used to connect the sending end and the receiving end to transmit the semantic information; When the at least one program is executed by the at least one processor, the at least one processor implements the semantic image transmission method based on a lightweight network as described in any one of claims 1 to 9.

Images