A lightweight network-based semantic image transmission method and system

Abstract

The application discloses a kind of based on lightweight network's semantic image transmission method and system, the method includes: obtaining input image;Based on Swin Transformer architecture, introduce quantization operation and nonlinear operation integer approximation, obtain image transmission network;The image transmission network includes encoder, decoder and channel modulation module;Based on the image transmission network transmission input image.The system includes: including memory and for executing above based on lightweight network's semantic image transmission method processor.By using the application, the calculation complexity and energy consumption of the model are significantly reduced, making it more suitable for deployment on edge devices with limited computing resources and energy consumption, and more suitable for high-speed image transmission scenarios.The application can be widely applied in data transmission field.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data transmission, in particular to a semantic image transmission method and system based on a lightweight network. BACKGROUND

[0002] Autonomous driving technology has attracted widespread attention due to its potential in improving road safety and optimizing traffic efficiency. Despite having advanced perception systems, autonomous vehicles still have potential safety risks due to the limited field of view of a single vehicle. Extending the field of view through image sharing can enhance driving safety, which makes efficient image data transmission crucial. However, under the condition of limited bandwidth, image transmission faces the problem of high delay.

[0003] To address this challenge, current research mainly focuses on compressing the amount of data transmitted. Compared to traditional methods that separately design source coding and channel coding, joint source-channel coding (JSCC) can reduce unnecessary redundancy, and semantic communication methods further reduce transmission delay by selectively transmitting content. These methods use deep learning models for feature extraction of images, among which the swintransformer model in ViTs (Vision transformers) performs outstandingly as a feature extractor for semantic communication, providing ideal image transmission quality.

[0004] However, although swintransformer performs outstandingly in capturing multi-scale information and complex features, its computational complexity and large model parameters still pose problems in its application on edge devices, such as endurance, device space, and memory requirements. These problems limit its widespread deployment in practical use. SUMMARY

[0005] Therefore, in order to solve the technical problem that the model calculation complexity and large model parameters of the swintransformer applied in the existing image transmission method, which further leads to the inability to be widely applied in various scenarios, the present application proposes a semantic image transmission method based on a lightweight network, which comprises the following steps:

[0006] obtaining an input image;

[0007] based on the SwinTransformer architecture, introducing linear operation quantization and nonlinear operation integer approximation operation, obtaining an image transmission network with only integer calculation;

[0008] the image transmission network comprises an encoder, a decoder and a channel modulation module;

[0009] transmitting the compressed feature map of the input image based on the image transmission network.

[0010] In some embodiments, the step of transmitting the input image based on the image transmission network specifically comprises:

[0011] Based on the encoding of the input image by the encoder, a feature map is obtained;

[0012] Based on the modulation of the feature map by the channel modulation module, a modulated feature map is obtained;

[0013] The modulated feature map is transmitted in a wireless channel;

[0014] Based on the reception of the modulated feature map by the decoder, demodulation processing and image recovery are performed to obtain a final image.

[0015] In some embodiments, the step of encoding the input image by the encoder to obtain a feature map specifically comprises:

[0016] Based on the image block division and linear embedding of the input image by the quantization convolution layer, format-converted data is obtained;

[0017] The format-converted data is feature-extracted by a multi-layer improved Swin Transformer Block to obtain a feature map;

[0018] The multi-layer improved Swin Transformer Block includes a quantization matrix multiplication layer, a quantization linear layer, an integer approximate LayerNorm, an integer approximate GELU, and an integer approximate Softmax.

[0019] In some embodiments, between the Swin Transformer Blocks, an image block merging layer is also included.

[0020] Through this preferred step, image block merging layers (patch merging) are interspersed between Swin Transformer Blocks (STB), gradually reducing the spatial dimension of the feature map and enhancing the feature representation capability

[0021] In some embodiments, the step of modulating the feature map by the channel modulation module to obtain a modulated feature map specifically comprises:

[0022] Accepting the signal-to-noise ratio of the current channel state;

[0023] Based on the signal-to-noise ratio, sequentially passing through a multi-layer quantization linear layer, an integer approximate ReLU operation, and a final integer approximate Sigmoid operation to generate a scaling coefficient;

[0024] modulating the feature feature map according to the scaling signal to obtain a modulated feature map.

[0025] In some embodiments, further comprising:

[0026] compressing the modulated feature map by a quantization linear layer.

[0027] Through the preferred step, the amount of data can be reduced, and the transmission bandwidth and storage requirements can be reduced.

[0028] In some embodiments, in the quantization operation, the quantization factor S x and the quantized integer representation I x As follows:

[0029]

[0030]

[0031] wherein X represents the input, X max represents the maximum quantization range of the input, k represents the number of quantization bits, represents rounding the numerical value to the nearest integer, and clip represents clipping X to the specified numerical range.

[0032] In some embodiments, the integer approximate Softmax representation is as follows:

[0033]

[0034] wherein represents the quantized integer representation corresponding to the i-th index value, d represents the dimension of the hidden feature, represents the quantized integer value obtained after the integer operation approximation, and the specific conversion steps are as follows: since the bit shift operation on the integer calculation unit is simpler than the direct exponential operation, it is necessary to convert the operation to the form of the base 2 exponential.

[0035] Since the bit shift operation on the integer calculation unit is easier than the exponential operation, it is necessary to convert to the operation with base 2, and the input exponential operation is processed using the base conversion formula:

[0036]

[0037] wherein log2e can be approximated to the binary number

[0038] log2e≈(1.0111)2=(1.1000)2-(0.0001)2

[0039] Thus we have:

[0040]

[0041] where

[0042] Because As the exponent term is not necessarily an integer, it can not be directly used for a shift operation. It can thus be split into an integer part to find a non-negative integer r and q such that and S x ·(-r)∈(-1,0].

[0043] Since when x∈(-1,0], So:

[0044]

[0045] can be rewritten as S x ·[-(r>>1)+I1], where

[0046] Combining the above, we have:

[0047]

[0048] Since the Softmax numerator denominator contains S x , the integer implementation of the division operation can be expressed as The input of this integer division function is the dividend, divisor and quantization bit number respectively. Its calculation method is as follows:

[0049]

[0050] The specific steps are explained as follows: in order to reduce the loss of precision in the process of integer division, first multiply the divisor with a larger integer 2 M to expand the numerical range, then multiply the result with and right shift (M+1-k) to adapt to the target bit width. In addition, since this process is equivalent to amplifying 2 k-1 times, the scaling factor

[0051] In some embodiments, the integer approximation of the Sigmoid function is represented as follows:

[0052]

[0053] The present application also proposes a semantic image transmission system based on a lightweight network, comprising:

[0054] at least one sending end comprising a processor and a memory for storing and executing a program to realize semantic information extraction and compressed transmission of images;

[0055] at least one receiving end comprising a processor and a memory for storing and executing a program to decode and restore received semantic information;

[0056] a communication channel for connecting the sending end and the receiving end to transmit the semantic information;

[0057] When the at least one program is executed by the at least one processor, the at least one processor realizes a semantic image transmission method based on a lightweight network as described above.

[0058] Based on the above scheme, the present application provides a semantic image transmission method and system based on a lightweight network. By introducing quantization operation and nonlinear operation integer approximation operation, the calculation complexity and video memory requirement are significantly reduced while ensuring the image transmission quality, which is more conducive to practical application. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is a step flow chart of a semantic image transmission method based on a lightweight network of the present application;

[0060] Figure 2 Performance comparison and analysis of semantic image transmission methods based on different channel conditions. DETAILED DESCRIPTION

[0061] The deep learning model required for semantic communication has a large number of parameters and high computational complexity. Hardware cost, deployment space and endurance ability jointly limit the deployment of semantic communication on edge devices. In summary, the existing technical methods still face many challenges in the practical application of semantic communication, such as high computational complexity, large memory requirement and high energy consumption, especially when deployed on edge devices.

[0062] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0063] It should be noted that, for the convenience of description, only the parts related to the application are shown in the drawings. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0064] It should be understood that the "system", "apparatus", "unit" and / or "module" used in the present application is a method for distinguishing different components, elements, parts, sections or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0065] As shown in the present application and claims, unless the context clearly indicates otherwise, the words "one", "an", "a", and / or "the" do not refer to the singular, but can also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements. The element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, product or device comprising the element.

[0066] In the description of embodiments of the present application, "a plurality of" means two or more than two. The following terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features.

[0067] In addition, flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or subsequent operations are not necessarily performed in sequence. On the contrary, each step can be processed in reverse order or simultaneously. At the same time, other operations can be added to these processes, or one or more steps of operation can be removed from these processes.

[0068] Reference Figure 1 The flowchart of an optional example of the semantic image transmission method based on lightweight network proposed by the present application can be applied to computer equipment. The semantic image transmission method proposed by the present embodiment can include but is not limited to the following steps:

[0069] Step S1, obtaining an input image;

[0070] Step S2, based on the Swin Transformer architecture, introducing quantization operation and nonlinear operation integer approximation, obtaining an image transmission network;

[0071] The image transmission network includes an encoder, a decoder and a channel modulation module;

[0072] Step S3, transmitting the input image based on the image transmission network.

[0073] In this embodiment, a lightweight semantic image transmission network (LSIT) is proposed to address the high computational complexity and high memory problem of full-precision Swin Transformer in feature extraction, especially for edge devices with limited computing resources and battery life. The network adopts an improved Swin Transformer architecture, which is divided into three parts: encoder, decoder and channel modulation module.

[0074] In some feasible embodiments, the step S3 specifically comprises:

[0075] S3.1, encoding processing the input image based on the encoder to obtain a feature map;

[0076] S3.2, modulating the feature map based on the channel modulation module to obtain a modulated feature map;

[0077] S3.3, transmitting the modulated feature map in a wireless channel;

[0078] S3.4, receiving the modulated feature map based on the decoder and performing demodulation processing and image recovery to obtain a final image.

[0079] In some feasible embodiments, the step S3.1 specifically comprises:

[0080] In the encoder part, LSIT first uses the quantized convolution layer (QConv2d) to perform image block division and linear embedding steps, converting the image into a format that deep learning can process. Next, we use multiple layers of improved STB (Swin Transformer Block) for feature extraction. In order to reduce the spatial dimension of the feature map and enhance the feature representation ability, the encoder inserts image block merging layers (patch merging) between the STBs, gradually reducing the spatial dimension of the feature map and enhancing the feature representation ability.

[0081] In some feasible embodiments, the step S3.2 specifically comprises:

[0082] After encoder processing, the output feature map is sent to the channel modulation module (channel Modnet). This module modulates the feature map according to the channel state to adapt to different wireless channel conditions.

[0083] In some feasible embodiments, it further comprises:

[0084] The modulated feature map is compressed by a quantized linear layer to reduce data volume, reduce transmission bandwidth and storage requirements, and is sent to the wireless channel after power normalization.

[0085] In some feasible embodiments, the step S3.4 specifically includes:

[0086] After the decoder receives the data, it first uses the quantized linear layer to decompress the feature map, and then demodulates the feature map through the channel modulation network. The demodulated feature map is processed through the Swin Transformer module for feature recovery, and then through the block splitting layer for upsampling. After multi-stage processing, the final recovered image is obtained.

[0087] In some feasible embodiments, the encoder in step S3.1 references a multi-layer improvement, SwinTransformer Block, which specifically includes:

[0088] The improved STB introduces quantization operations and integer approximation on the basis of the traditional structure. It uses the quantization matrix multiplication layer (Q_Matmul) and the quantized linear layer for linear operation quantization, and also includes integer-approximated LayerNorm, GELU, and Softmax.

[0089] These improvements enable the STB to efficiently extract or recover features on low-power integer computing units.

[0090] In some feasible embodiments, the channel modulation module specifically includes:

[0091] The channel modulation module (SNR Modulation) first receives the signal-to-noise ratio (SNR) of the current channel state as input. After three quantized linear layers and integer-approximated ReLU operations, it finally generates scaling coefficients of the same size as the input signal through integer-approximated Sigmoid operation. These scaling coefficients are used to adjust the signal to make it more suitable for communication under specific channel conditions, thereby optimizing the signal transmission performance. The channel modulation module takes the feature map extracted by the encoder or the feature map received after decompression from the wireless channel as input. It passes through 8 quantized linear layers in sequence, and the element product of the feature values between each two layers is multiplied by the scaling coefficients output by the SNR Modulation. After the signal passes through the last quantized linear layer, it is subjected to integer-approximated Sigmoid operation to generate a scaling coefficient matrix of the same size as the input feature map. The element product of this scaling coefficient matrix and the initial input feature map is obtained to obtain the final output feature map.

[0092] Through this embodiment, it is ensured that the signal is stably and efficiently transmitted in the wireless channel, or the original signal is as accurately as possible recovered from the actual channel.

[0093] Based on the above scheme, quantization technology is an important part of LSIT network, which converts model parameters and activation values from high precision (usually floating point numbers) to low precision integer representation. LSIT implements integer quantization of weights and activation values in all linear operations in the semantic communication scenario, including quantization of activation values (Q_Act), and quantization of weights parameters of linear layer, convolution layer and matrix multiplication layer. Q_Act is attached after all linear or nonlinear layers to quantize the output activation values from Int32 back to Int8 to ensure that the output of the subsequent process does not exceed the range. In order to solve the problem of non-differentiable quantization, LSIT uses a pass-through estimator for back propagation. In the implementation process of integer quantization, the first thing to do is to calculate the scaling factor of quantization. Since LSIT chooses symmetric uniform quantization, assuming the input is X, the quantization range is [-X max ,X max ]. Therefore, when the quantization bit number is k, the quantization factor S X is represented as:

[0094]

[0095] where S X represents the quantization factor, X represents the input, X max represents the maximum quantization range of the input, and k represents the quantization bit number.

[0096] In order to ensure that S X does not exceed the range after being quantized to an integer in the inference stage, we need to uniformly clip the data and divide it by the scaling factor, and use the rounding operation to approximate it to an integer. The quantized integer I X is:

[0097]

[0098] Since the scaling factor once determined in the training stage will remain unchanged, and LSIT considers inference on integer calculation units, the scaling factor needs to be converted to binary number (DN) format, so as to convert the floating point scaling factor to an integer calculation unit available format:

[0099]

[0100] where b and c are positive integers. Therefore, by performing integer multiplication with b and right shifting c bits, we can realize the integer calculation of the linear layer.

[0101] The quantization of linear layer, convolution layer and matrix multiplication layer is the same. Taking the linear layer Y = WX + B as an example, after quantizing the activation values of the previous layer, we get I X and S X , and quantize the weights of the linear layer to get the quantized integer value and scaling factor I Wand S W The bias scaling factor is by default the product of the weight and activation value scaling factors. Thus:

[0102] Y = S Y · I Y = WX + B = (S W · S X )((I W · I X + I B ))

[0103] Then we have:

[0104]

[0105] The implementation on the integer computing unit is:

[0106] I Y = [b'(I W · I X + I B )] » c'

[0107] where » c' denotes a right shift by c' bits.

[0108] The second part is the integer approximation of the nonlinear layer:

[0109] In LSIT, the nonlinear operation layers such as ReLU, GELU, Sigmoid, Softmax and LayerNorm are approximated by integer operations to solve the non-homogeneity problem of nonlinear operation layers. For the ReLU layer, since it is linear when the input is greater than 0, only the non-negative part needs to be preserved by quantization. GELU, Sigmoid and Softmax contain the nonlinear sub-function exp, which needs to be converted to base 2 operation because the bit shift operation on the integer computing unit is easier than the exponential operation:

[0110]

[0111] where log2e can be approximated as the binary number

[0112] log2e≈(1.0111)2=(1.1000)2-(0.0001)2

[0113] Thus we have:

[0114]

[0115] where

[0116] Because​ As the exponent term is not necessarily an integer, it can not be directly used for shift operation. Therefore, it can be divided into an integer part to find a non-negative integer r and q such that and S x ·(-r)∈(-1,0].

[0117] Since when x∈(-1,0], So:

[0118]

[0119] Can be converted to S x ·[-(r>>1)+I1], where

[0120] In summary, we can get:

[0121]

[0122] Since the Softmax molecule denominator contains S x , the integer implementation of the division operation can be expressed as The input of this integer division function is the dividend, divisor and quantization bit number respectively. Its calculation method is as follows:

[0123]

[0124] The specific steps are explained as follows: In order to reduce the loss of precision in the process of integer division, first multiply the divisor By a larger integer 2 M To expand the numerical range, then multiply the result by And right shift (M+1-k) to adapt to the target bit width. In addition, since this process is equivalent to amplifying 2 k-1 times, the scaling factor

[0125] For the Sigmoid function, it can be expressed as:

[0126]

[0127] Its form is similar to Softmax, so it can also use Intexp and Intdiv for approximation. According to the existing papers, the GELU function can be approximated as:

[0128] GELU(x) = x·σ(1.702x)

[0129] Where 1.702 can be converted to binary form, multiplied by the input x and used as the input of the Sigmoid function, and finally multiplied by x to get the final result.

[0130] In the inference phase, the integer arithmetic unit allows LayerNorm to directly calculate the mean and variance of the data, but does not support the square root of the standard deviation obtained by algorithm. Solving var(x) = 0 is equivalent to finding the root of F(I i ) = var(x) - I i , by Newton iteration method:

[0131]

[0132] Therefore, Layernorm can be represented as:

[0133] LayerNorm(S X ,I X ) = Intdiv[S X ·I X -mean(S X ·I X ), I i+1 , k]

[0134] Where i = 10 can achieve the desired effect.

[0135] Based on the above scheme, the linear layer and the nonlinear layer are quantized, which significantly reduces the computational complexity and energy consumption of the model, making it more suitable for deployment on edge devices with limited computing resources and energy consumption. This method can reduce the dependence on complex calculations and improve the running efficiency of the model on low-cost hardware. In the process of integer quantization of the linear layer, the model's weights and activation values are converted from floating-point numbers to low-precision integer representations, greatly reducing the computational complexity and storage requirements. In the processing of the nonlinear layer, the invention proposes an integer approximation method, which converts the nonlinear function operation into integer operation and shift operation, avoiding frequent integer and floating-point conversion in the inference process, thereby improving the operation efficiency. In addition, the application of the quantization perception training method ensures that the model can maintain high precision in the inference phase after integer quantization, ensuring the stable performance of the semantic communication system under different channel conditions. This series of technical means jointly support the efficient operation of the integer quantized model on hardware that only supports integer operation, significantly reducing the deployment cost and energy consumption, making the semantic communication system more feasible and widely applicable in practical applications.

[0136] Since the scheme only uses integer operations in the model inference process, the requirement for hardware is lower. This means that it can be run efficiently on hardware that only supports integer operations, reducing deployment costs and the demand for device space, and helping to promote it in a wider range of application scenarios. Although an integer quantization method is used, the scheme can still maintain almost equivalent transmission quality to the full-precision model under different image resolutions, channel conditions and compression ratios, and sometimes even performs better. This shows that the scheme can effectively guarantee the quality of semantic image transmission in a bandwidth-limited situation. The present embodiment uses the peak signal-to-noise ratio (PSNR) as an evaluation indicator to compare the transmission quality of the lightweight version LSIT, the corresponding full-precision model WITT without lightweight, and another semantic communication model JSCC. In order to reflect the robustness of the lightweight scheme, the present embodiment has carried out extensive experimental comparisons under different image resolutions, signal-to-noise ratios, channel types and compression rates.

[0137] In terms of image resolution, the present embodiment trains and tests the model on the CIFAR10 dataset to observe its effect on low-resolution images, and trains it on the DIV2K dataset and tests it on the KODAK dataset to measure its effect on high-resolution images. In addition, the present embodiment keeps the model architecture consistent with the full-precision model WITT without lightweight, and the number of stages varies with the resolution of the training image. For low-resolution images, the present embodiment uses 2 stages, set as [N1, N2] = [2, 4], [C1, C2] = [128, 256], and the window size is 2. For high-resolution images, we use 4 stages, set as [N1, N2, N3, N4] = [2, 2, 6, 2], [C1, C2, C3, C4] = [128, 192, 256, 320], and the window size is 8.

[0138] In terms of channel type, LSIT experiments on AWGN and Rayleigh channels under different signal-to-noise ratios, and simulations under different compression ratios (cbr).

[0139] The experimental results are shown in Figure 2 . Figure 2 (A) and Figure 2 (B) show that LSIT has good performance under different channel types and different signal-to-noise ratios. Figure 2 (A) and Figure 2 (C) show that LSIT maintains excellent performance under different image resolutions, Figure 2(D) shows the robust transmission performance of LSIT under different compression ratios. Experiments show that under different image resolutions, channel types and compression ratios, when the signal-to-noise ratio is low, the difference in PSNR can always be kept near 0.5% of the full-precision model, and sometimes even better. In addition, compared with the full-precision model, the pure integer quantization model not only maintains stable image transmission quality and ensures the function of semantic communication, but also has a significant cost advantage: lower cost in terms of computing unit area and energy consumption, and lower memory usage and computational complexity. Figure 2 In the full-precision model WITT before quantization and the LSIT after quantization, the bit number of weight parameters and activation values (W / A) is 32 and 8 respectively, and in the inference stage, the quantized model can also perform inference function with smaller storage occupation. In addition, Bit Operations (BOPs) is used to measure the computational complexity during the inference of deep learning model, and is positively correlated with the power consumption of the model. From the above data, it can be seen that the quantized LSIT has good improvement in memory occupation, computational complexity and energy consumption.

[0140] Table 1 Performance comparison of quantized model LSIT and full-precision model WITT on edge devices

[0141] Method W / A Memory footprint / MB BOPs / 10 12 ]] Best PSNR performance WITT 32 / 32 4033 202.8 29.65 LSIT 8 / 8 1008 12.7 29.47

[0142] The current research mainly optimizes the image source process of automatic driving target recognition, which can obtain the visual blind area in a more cost-friendly way for subsequent target recognition process. In addition, the current work considers the transmission of image data, and further considers the lightweight semantic communication of text data, realizing multi-modal process. Thus, multi-modal fusion automatic driving target recognition based on quantization optimization is realized.

[0143] A semantic image transmission system based on a lightweight network, comprising:

[0144] At least one sending end, comprising a processor and a memory, for storing and executing programs to realize semantic information extraction and compressed transmission of images;

[0145] At least one receiving end, comprising a processor and a memory, for storing and executing programs to decode and restore the received semantic information;

[0146] A communication channel for connecting the sending end and the receiving end to transmit the semantic information;

[0147] When the at least one program is executed by the at least one processor, the at least one processor implements a lightweight network-based semantic image transmission method as described above.

[0148] The content in the above method embodiments is applicable to the system embodiments, the system embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0149] A storage medium, wherein the storage medium stores processor-executable instructions, and the processor-executable instructions, when executed by a processor, are used to implement a lightweight network-based semantic image transmission method as described above.

[0150] The content in the above method embodiments is applicable to the storage medium embodiments, the storage medium embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0151] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above-mentioned embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A semantic image transmission method based on a lightweight network, characterized in that: The following steps are involved: Get the input image; Based on the Swin Transformer architecture, quantization operations and integer approximation of nonlinear operations are introduced to obtain an image transmission network; The image transmission network includes an encoder, a decoder and a channel modulation module; transmitting the input image based on the image transmission network; The step of transmitting the input image based on the image transmission network specifically includes: Performing encoding processing on the input image based on the encoder to obtain a feature map; Modulating the characteristic graph based on the channel modulation module to obtain a modulated characteristic graph; Transmitting the modulated characteristic image in a wireless channel; The decoder receives the modulated feature map, performs demodulation processing and image restoration, and obtains a final image; The step of encoding the input image based on the encoder to obtain a feature map specifically includes: Performing image block division and linear embedding on the input image based on a quantized convolutional layer to obtain format-converted data; Performing feature extraction on the data after the format conversion through a multi-layer improved Swin Transformer Block to obtain a feature map; The multi-layer improved Swin Transformer Block includes a quantized matrix multiplication layer, a quantized linear layer, an integer approximation LayerNorm, an integer approximation GELU and an integer approximation Softmax; The step of modulating the characteristic graph based on the channel modulation module to obtain a modulated characteristic graph specifically includes: Accept the signal-to-noise ratio of the current channel state; Based on the signal-to-noise ratio, a scaling factor is generated by sequentially performing a quantized linear layer, an integer-approximated ReLU operation, and an integer-approximated Sigmoid operation; The feature map is modulated according to the scaling factor to obtain a modulated feature map.

2. The method for transmitting semantic images based on a lightweight network according to claim 1, characterized in that: An image block merging layer is also included between the SwinTransformer Blocks.

3. The method for transmitting semantic images based on a lightweight network according to claim 2, characterized in that: Also includes: The modulated feature map is compressed through a quantized linear layer.

4. The method for transmitting semantic images based on a lightweight network according to claim 3, characterized in that: In the quantization operation, the quantization factor S x and the quantized integer representation I x As shown below: Among them, X represents the input, X max Indicates the maximum quantization range of the input, k indicates the number of quantization bits, Indicates rounding the value to the nearest integer, and clip means clipping X to the specified value range.

5. The method for transmitting semantic images based on a lightweight network according to claim 4, characterized in that: The integer approximation Softmax is expressed as follows: in, Indicates the quantized integer representation corresponding to the i-th index value, The value of d is the dimension of the hidden feature, express The quantized integer value obtained after integer operation approximation.

6. The method for transmitting semantic images based on a lightweight network according to claim 5, characterized in that: The integer approximation of the Sigmoid function is expressed as follows:

7. A semantic image transmission system based on a lightweight network, characterized in that: include: At least one transmitting end, comprising a processor and a memory, for storing and executing a program to extract semantic information from an image and to compress and transmit the image; at least one receiving end, comprising a processor and a memory, configured to store and execute a program to decode and restore received semantic information; a communication channel, for connecting the sending end and the receiving end to transmit the semantic information; When the at least one program is executed by the at least one processor, the at least one processor implements the semantic image transmission method based on a lightweight network as described in any one of claims 1 to 6.

Images