Text screen content image transmission method based on U-Net semantic segmentation

Abstract

The invention relates to a text screen content image transmission method based on U-Net semantic segmentation, and belongs to the technical field of wireless communication and multimedia transmission. The method comprises the following steps: acquiring an original data set, and constructing an image semantic knowledge base according to the original data set; segmenting an input text type screen content image into a text region and a background region by adopting U-Net; respectively carrying out feature coding on the divided areas through an encoder, and mapping coded feature information to different subcarriers of OFDM through a channel attention module according to channel state information; power distribution of key features of OFDM is optimized through a space attention feature module according to the space features; the information is transmitted through OFDM, and the information is received at a receiving end and subjected to semantic decoding, so that an image is reconstructed, and a corresponding semantic task is executed. According to the method, the transmission accuracy and robustness of the text type screen content image in a complex channel environment can be remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communication and multimedia transmission, and relates to a method for transmitting text-based screen content images based on U-Net semantic segmentation, and particularly to a task-driven OFDM wireless transmission method for text-based screen content images based on U-Net semantic segmentation. Background Art

[0002] In the context of the accelerating digitalization process today, wireless communication technology faces many challenges and opportunities. Although Shannon's separation theorem has laid a theoretical foundation for communication coding, with the booming development of emerging wireless applications such as the Internet of Things and edge intelligence, efficiently transmitting a large amount of data under strict delay constraints has become a key problem to be solved urgently, which has promoted joint source-channel coding (JSCC) to become a hot research field.

[0003] The rise of deep learning technology has injected new vitality into the design of JSCC solutions. Many studies are committed to using deep neural networks to optimize the wireless transmission of high-information content signals such as images and videos, and have achieved certain results in different aspects. For example, early architectures based on autoencoders have shown advantages on specific channels and have gradually been extended to aspects such as feedback channels and bandwidth adaptive transmission.

[0004] With the popularization of application scenarios such as telecommuting, online education, video conferencing, and cloud gaming, the functional requirements for screen sharing, document transmission, remote desktop control, etc. are increasing day by day. The efficient and accurate transmission of text-based screen content images has become an important topic in wireless communication technology.

[0005] Text-based screen content images are different from general natural images. They often contain a large amount of structured information such as text, charts, and symbols, and this information has extremely high value and significance for the receiver. However, traditional image transmission methods often do not optimize for the characteristics of text content, resulting in problems such as blurred text and distorted charts during transmission, seriously affecting the receiver's reading experience and information acquisition efficiency.

[0006] The emergence of joint source-channel coding (JSCC) technology provides a new idea for the efficient transmission of text-based screen content images. By introducing deep learning technology, a more intelligent and adaptive JSCC scheme can be designed, enabling the clarity and accuracy of text content to be retained as much as possible under limited bandwidth and strict delay constraints. For example, a deep neural network can be used to extract features and compress text images, and at the same time, channel coding technology can be combined to resist noise and interference during transmission, thereby achieving the efficient and reliable transmission of text content.

[0007] However, when traditional deep learning-based JSCC methods handle the wireless transmission task of text-based screen content images, they do not consider the semantic differences between text regions and background regions and adopt a homogenized coding strategy. Research shows that text region pixels only account for 12 - 25% of the screen content image, but their information entropy is 3.8 times that of the background region. Existing methods fail to establish a task-driven feature priority mechanism, resulting in waste of channel resources in low-value background regions.

[0008] In addition, JSCC still has significant deficiencies in adapting to complex and changing channel conditions. Existing OFDM resource allocation schemes (such as greedy subcarrier allocation) consider the dynamics of channel state information (CSI), but do not establish a joint optimization model with image spatial features. Existing power control algorithms (such as the improved water-filling algorithm) only aim to maximize channel capacity and do not consider the requirements of downstream semantic tasks. Experiments show that increasing the power of the text region by 3 dB can increase the OCR accuracy by 21%, while traditional methods only increase it by 7% under the same total power. Especially in low SNR scenarios (<5 dB), the SSIM value of text reconstruction of existing methods is 0.15 - 0.2 lower than that of the background region, significantly affecting readability. When traditional methods are trained, the processing methods for specific signal-to-noise ratios or a series of signal-to-noise ratios have drawbacks. Either a large amount of different network parameters need to be stored, or performance is sacrificed and channel state information (CSI) is not fully utilized. Although CSI has been applied to learning-aided design in some studies, in OFDM systems, its potential in feature mapping and power allocation has not been fully explored, especially in how to accurately map the key features of the input image to reliable subcarriers and reasonably allocate power, which still needs to be broken through. Summary of the Invention

[0009] In view of this, the purpose of the present invention is to provide a text-based screen content image transmission method based on U-Net semantic segmentation, aiming to improve the transmission efficiency and robustness of text-based screen content images, as well as the completion efficiency of downstream tasks.

[0010] To achieve the above purpose, the present invention provides the following technical solutions:

[0011] A text-based screen content image transmission method based on U-Net semantic segmentation, the method includes the following steps:

[0012] Obtain the original text-based screen content image dataset, preprocess the dataset, and construct an image semantic knowledge base according to the dataset;

[0013] Use U-Net to perform semantic segmentation on the input text-based screen content image, and divide the image into text regions and textless background regions;

[0014] The divided text regions and background regions are feature-encoded by an encoder, and the encoded feature information is mapped to different subcarriers of the OFDM transmission system according to the channel state information through a channel attention module;

[0015] Then, the spatial features of the image are captured by a spatial attention feature module in the encoder, and the power allocation of the key features of the OFDM transmission system is optimized according to the spatial features;

[0016] The information is transmitted through the OFDM transmission system, and the information is received and semantically decoded at the receiving end to reconstruct the image and perform corresponding semantic tasks.

[0017] Furthermore, the acquisition methods of the original dataset include: actively collecting through technical means, obtaining the screen image in real time and saving it as a dataset, including intercepting the application interface and dynamic operations; using the device camera or screen recording software to record the screen operations, and then generating an image dataset through video frame extraction; using existing open-source screen content datasets;

[0018] Then, preprocessing operations such as data cleaning, integration, transformation, and reduction are performed on the dataset, and the preprocessed dataset is divided into a training set and a test set;

[0019] At the same time, a semantic knowledge base in the image semantic communication system is constructed through the preprocessed dataset to support the efficient encoding and decoding of semantic information. The construction methods of the semantic knowledge base include three types: manual construction, automated construction, and hybrid construction; Manual construction relies on domain experts to define concepts, attributes, and relationships through formal languages; Automated construction is based on machine learning and natural language processing technologies to extract entities, relationships, and rules from text; Hybrid construction combines manual annotation and automated methods to improve quality through iterative optimization;

[0020] Ontology engineering realizes the hierarchical organization of knowledge through top-level framework design, while knowledge graphs emphasize multi-source data fusion and structured storage.

[0021] Furthermore, let the input text-based screen content image be p. Then, based on the advanced semantic segmentation technology of U-Net, it is divided into text regions and text-free background regions, which are respectively labeled as x t and x b :

[0022] [x t ,x b = f U-Net (p)

[0023] where p is the input text-based screen content image, and f U-Net (·) is the semantic segmentation network; The encoder of the U-Net network extracts features at different levels through multiple downsamplings.

[0024] Furthermore, the encoder adopts a dressing structure design, which includes a number of feature learning modules FL and a number of channel learning modules CL. Among them,

[0025] The feature learning module FL is used to learn the input intermediate representation, which includes a 2D convolution or deconvolution layer, a batch normalization layer, and a PReLU layer; the 2D convolution or deconvolution layer extracts and reconstructs the feature information of the image by performing specific convolution and deconvolution operations on the input data; the batch normalization layer normalizes the input of each layer during the training process to accelerate the convergence speed of the model; the PReLU layer, as an activation function, introduces non-linearity to enhance the expression ability of the model;

[0026] The channel learning module CL is used to learn the attention mask; the channel learning module CL is based on a dual attention mechanism, and internally includes a channel attention module and a spatial attention module. The channel attention module evaluates and analyzes the channel state information CSI of different subcarriers, and allocates and adjusts the mapping weights of features on subcarriers according to the channel quality; the spatial attention module considers the spatial structure and feature distribution of the image from the spatial dimension to optimize the mapping method of features on subcarriers.

[0027] Furthermore, in the channel attention module, first, the text region x t and the background region x b are encoded into the input feature codeword That is:

[0028] F in =[f t ,f b =f CNN [x t ,x b

[0029] In the formula, f CNN [·] represents the convolutional neural network feature extraction operation, f t ,f b are the text feature and background feature extracted by the convolutional neural network respectively; is the feature codeword composed of f t ,f b , among which, represents the real number tensor space of dimension c×h×w, where c, h, and w represent the number of channels, height, and width respectively;

[0030] Then, apply average pooling A in (F vec ) along the spatial direction to obtain the real number vector in where, Among them, ​

[0031]

[0032] Then, connect F ca with the estimated CSI vector to obtain the intermediate vector i c , that is:

[0033]

[0034] Among them, the CSI vector information includes the different subcarrier allocations and mapping characteristics of OFDM, c and L in f respectively represent the number of channels and the number of subcarrier groups; concatenate(·) represents the concatenation function;

[0035] Calculate the channel attention mask S c through several fully connected layers, that is:

[0036]

[0037] Among them, f c (·) represents the FC layer followed by the PreLu function. Finally, obtain the output F cout of the channel attention module, that is:

[0038]

[0039] In the formula, S c [i] represents the channel attention weight, the weight value of each channel i, and F in [i, h, w] represents the value of the input feature at the channel i, spatial positions h, and w. ⊙ represents the Hadamard product, that is, element-wise multiplication.

[0040] Furthermore, in the spatial attention module, first, apply the average pooling operation to F cout along the channel direction to obtain F sa , that is:

[0041]

[0042] In the formula, Ave s (·) is the average pooling operation;

[0043] Then, connect and F sa to obtain the intermediate feature i s , that is:

[0044]

[0045] i s Calculate the spatial mask S through several FC layerss , that is:

[0046]

[0047] In the formula, f s (·) represents the spatial mask calculation function, and h and w represent height and width;

[0048] Finally, calculate the output feature vector, that is:

[0049]

[0050] In the formula, S s [j, k] represents the spatial attention weight, reflecting the importance of the spatial position, and F cout [c, j, k] represents the eigenvalue of the channel c and spatial position (j, k) in the output of the channel attention module; j is the index of the spatial dimension.

[0051] Furthermore, in the OFDM transmission system, it includes an OFDM transmitter and an OFDM receiver. Among them, the OFDM transmitter and the OFDM receiver communicate through a multipath fading channel; the channel consists of an L t -path multipath fading channel, uses N s OFDM symbols to carry image information, and ensures accurate channel estimation for L p OFDM subcarriers by adding N f pilot signals;

[0052] The encoder E θ maps the input image to a complex matrix The size of the matrix is c×h×w, representing the color channel, height, and width of the image respectively. The generated channel input is:

[0053]

[0054] Among them, is the channel state information (CSI) estimation vector available at the transmitter, and E θ (·) is the encoding operation of the encoder;

[0055] The channel input is subject to an average power constraint, that is:

[0056]

[0057] Among them, represents taking the expectation, and the expectation depends on the input image. ||·|| is the Frobenius norm, and the subscript "F" indicates that the current norm is the Frobenius norm, which is used to measure the modulus of the matrix.

[0058] Furthermore, at the transmitter side, the encoded symbols are used to generate OFDM symbols through Inverse Discrete Fourier Transform (IFFT), and a cyclic prefix (CP) is appended before transmission through a multipath channel. The time-domain transfer function of the multipath fading channel is defined as:

[0059] y = h * x + w

[0060] where x and y represent the input and output signal vectors respectively, w is the additive white Gaussian noise (AWGN), * denotes the linear convolution operation, and h is the channel impulse response.

[0061] At the receiver side, the received symbols are demodulated by removing the cyclic prefix and applying the Fast Fourier Transform (FFT). The equivalent frequency-domain transfer function is:

[0062]

[0063] where the frequency-domain channel matrix is a diagonal matrix, and the k-th diagonal element is H[k], which represents the frequency-domain channel response of the k-th subcarrier. Let denote the output of the power normalization module at the transmitter, i.e., the input of the IFFT module. denote the i-th OFDM symbol at the k-th subcarrier. is the frequency-domain noise matrix, where and they are independent of each other.

[0064] Furthermore, at the receiver side, based on the FFT output of the pilot symbols, the Minimum Mean Square Error (MMSE) or Least Squares (LS) method is used to estimate the frequency-domain Channel State Information (CSI). Then, the estimated CSI vector is used for equalization processing to obtain the equalized data:

[0065]

[0066] where is the estimated channel gain of the i-th subcarrier; μ is the average Signal-to-Noise Ratio (SNR) of the transmit power P s and the noise power σ 2 .

[0067] The equalized data and the estimated CSI vector are input into the decoder D φ to reconstruct the transmitted image

[0068]

[0069] where D φ (·) represents the decoding operation of the decoder D φ , and the decoder is structurally consistent with the encoder.

[0070] Furthermore, the training process of this method is divided into two stages: semantic segmentation and semantic communication. Different loss functions are used for optimization respectively, and the loss is minimized through the gradient descent method until the stopping criterion is met;

[0071] The loss function in the semantic segmentation stage is calculated based on the true labels and weight maps of pixels;

[0072] The loss function in the semantic communication stage is calculated using the mean square error (MSE), channel estimation loss, and text loss function. The performance metric of image quality is defined using the peak signal-to-noise ratio (PSNR):

[0073]

[0074] where max x represents the maximum possible value x of the input signal, and the expectation is taken over all pixels;

[0075] The encoder and decoder are trained through end-to-end joint optimization, with the goal of minimizing the loss function:

[0076]

[0077] where the expectation of the loss function takes into account the randomness in the source distribution and the channel distribution.

[0078] The beneficial effects of the present invention are as follows:

[0079] The present invention first performs semantic segmentation on text-based screen content images through a U-Net network, accurately dividing the images into text regions and text-free background regions. This refined segmentation method enables subsequent feature encoding and transmission to more specifically process key information, avoiding interference from irrelevant information, thereby significantly improving the transmission efficiency.

[0080] The channel attention module of the present invention can intelligently allocate and adjust the mapping weights of features on OFDM subcarriers according to the estimated channel state information (CSI). This adaptive mechanism enables the system to better adapt to channel changes, ensuring that important features can be allocated to subcarriers with better channel quality for transmission, thereby enhancing the stability and reliability of transmission.

[0081] The introduction of the spatial attention module of the present invention effectively makes up for the spatial information ignored in the channel attention module. By capturing the spatial features of the image, the spatial attention module can optimize the power allocation of key features in the OFDM transmission system, enabling features related to key spatial regions of the image to obtain better transmission guarantee, and further improving the quality and reconstruction effect of image transmission.

[0082] In summary, the present invention can significantly improve the transmission accuracy and robustness of text-based screen content images in complex channel environments, effectively reduce the transmission error rate, and improve the image reconstruction quality and text recognition accuracy.

[0083] Other advantages, objectives, and features of the present invention will be described to some extent in the subsequent specification, and to some extent, will be obvious to those skilled in the art based on the study of the following text, or can be learned from the practice of the present invention. The objectives and other advantages of the present invention can be achieved and obtained through the following specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in detail preferably with reference to the accompanying drawings, where:

[0085] Figure 1 It is a schematic diagram of the overall framework of the task-driven OFDM wireless image transmission method based on U-Net semantic segmentation according to an embodiment of the present invention.

[0086] Figure 2 It is a schematic diagram of the structures of the encoder and decoder according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0087] The following uses specific specific examples to illustrate the embodiments of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present invention schematically. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0088] Among them, the accompanying drawings are only for illustrative purposes, showing only schematic diagrams, not physical diagrams, and should not be construed as a limitation to the present invention; in order to better illustrate the embodiments of the present invention, some components in the accompanying drawings will be omitted, enlarged, or reduced, and do not represent the dimensions of actual products; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the accompanying drawings may be omitted.

[0089] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "rear", etc. indicating the orientation or positional relationship, they are based on the orientation or positional relationship shown in the accompanying drawings. This is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the positional relationship in the accompanying drawings are only for illustrative purposes and cannot be understood as a limitation of the present invention. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0090] Please refer to Figure 1 , which is a method for transmitting text-based screen content images based on U-Net semantic segmentation.

[0091] Embodiment

[0092] This embodiment provides a detailed implementation process of a method for transmitting text-based screen content images based on U-Net semantic segmentation. As Figure 1 shown, it specifically includes the following steps:

[0093] S1. Obtain the original text-based screen content image dataset, perform preprocessing on the dataset, and construct an image semantic knowledge base according to the dataset;

[0094] S2. Use U-Net to perform semantic segmentation on the input text-based screen content image, and divide the image into a text region and a background region without text;

[0095] S3. Encode the features of the divided text region and background region through an encoder, and map the encoded feature information to different subcarriers of the OFDM transmission system according to the channel state information through a channel attention module;

[0096] S4. Then capture the spatial features of the image through a spatial attention feature module in the encoder, and optimize the power allocation of the key features of the OFDM transmission system according to the spatial features;

[0097] S5. Transmit the information through the OFDM transmission system, receive the information at the receiving end and perform semantic decoding to reconstruct the image and execute the corresponding semantic tasks.

[0098] In step S1 of this embodiment, the screen content dataset is actively collected by technical means or an existing open-source dataset is used. Then, preprocessing operations such as data cleaning, integration, conversion, and reduction are performed on the dataset to ensure data quality and model adaptability. Finally, the preprocessed dataset is divided into a training set and a test set.

[0099] Meanwhile, a semantic knowledge base in the image semantic communication system is constructed through the preprocessed dataset to support the efficient encoding and decoding of semantic information. There are three specific methods: manual construction, automated construction, and hybrid construction. Manual construction relies on domain experts to define concepts, attributes, and relationships through formal languages; automated construction is based on machine learning and natural language processing technologies to extract entities, relationships, and rules from texts; hybrid construction combines manual annotation and automated methods to improve quality through iterative optimization. In addition, ontology engineering realizes the hierarchical organization of knowledge through top-level framework design, while knowledge graphs emphasize multi-source data fusion and structured storage. Key issues such as entity disambiguation, relation extraction, and knowledge completion need to be addressed during the construction process to ensure the accuracy, integrity, and scalability of the knowledge base.

[0100] In step S2 of this embodiment, during the transmission of the source picture information, considering the diversity and complexity of the picture content, taking a specific picture p as an example, an advanced semantic segmentation technology based on U-Net is introduced to accurately divide it into a text region and a text-free background region, which are respectively labeled as x t and x b :

[0101] [x t ,x b = f U-Net (p)

[0102] where p is the input text-like screen content image, and f U-Net (·) is the semantic segmentation network. The encoder of the U-Net network extracts features at different levels through multiple downsamplings.

[0103] In step S3 of this embodiment, in order to adapt to different subcarriers in the OFDM system facing different channel gains and make full use of the estimated CSI, the JSCC model is optimized to adapt to channel changes. In the system architecture of this embodiment, as Figure 2 shown, both the encoder and the decoder adopt a unique comb-like structure design, which includes five feature learning (FL) modules and four channel learning (CL) modules. This delicate structure design endows the system with powerful capabilities, enabling it to finely modulate each feature at diverse scales, thereby better adapting to complex signal processing requirements.

[0104] Specifically, as a key part of the system for learning the intermediate representation of the input, the FL module is mainly composed of an organic combination of 2D convolutional / transposed convolutional layers, batch normalization layers, and PReLU layers. The 2D convolutional / transposed convolutional layers can effectively extract and reconstruct the feature information of the image through specific convolutional and transposed convolutional operations on the input data, laying a foundation for subsequent processing. The batch normalization layer normalizes the input of each layer during the training process, accelerating the convergence speed of the model and improving the stability and generalization ability of the model. As an activation function, the PReLU layer can introduce non-linearity, enhance the expressive ability of the model, and enable the system to better capture complex patterns and relationships in the input data. Through the collaborative work of this series of layers, the FL module can accurately learn and represent the intermediate representation of the input, preparing for subsequent feature mapping and transmission.

[0105] The CL module based on the dual attention mechanism shoulders a more crucial task, that is, to learn the attention mask, so as to accurately map the intermediate features processed by the FL module to the appropriate subcarriers according to the estimated channel state information (CSI) and input features. The CL module is further subdivided into a channel attention module and a spatial attention module. The channel attention module can evaluate and analyze the channel quality of different subcarriers based on the CSI information, so as to intelligently allocate and adjust the mapping weights of features on the subcarriers, ensuring that important features can be allocated to subcarriers with better channel quality for transmission, thereby improving the reliability and efficiency of transmission. The spatial attention module starts from the spatial dimension, fully considering the spatial structure and feature distribution of the image, and further optimizing the mapping method of features on the subcarriers, so that the features related to the key spatial regions of the image can obtain better transmission guarantee, thereby improving the transmission effect and reconstruction quality of the entire system for image information. Through the close cooperation of these two sub-modules, the CL module can achieve efficient mapping and optimal allocation of intermediate features, significantly improving the transmission performance of the system in complex channel environments.

[0106] Thus, in the channel attention module, for the features after semantic segmentation: the text region x t and the background region x b , first, with the help of the powerful feature extraction and encoding ability of the convolutional neural network, it is efficiently encoded into the input feature codewords thus laying a solid foundation for subsequent information transmission and ensuring that the picture information can maintain high accuracy and reliability under different channel conditions. That is:

[0107] F in =[f t , f b = f CNN [x t , x b ​

[0108] In the formula, f CNN [·] represents the convolutional neural network feature extraction operation, and f t , f b are respectively the text feature and the background feature extracted by the convolutional neural network; is the feature codeword composed of f t , f b . Among them, represents the real number tensor space of dimension c×h×w, where c, h, and w respectively represent the number of channels, height, and width.

[0109] Then, apply average pooling to the input feature F in along the spatial direction to obtain the vector . Among them,

[0110]

[0111] Then, connect F ca with the estimated CSI vector to obtain the intermediate vector i c , that is:

[0112]

[0113] Among them, the CSI vector information includes the different subcarrier allocations and mapping features of OFDM, c and L in f respectively represent the number of channels and the number of subcarrier groups; concatenate(·) represents the connection function.

[0114] Calculate the channel attention mask S c through several fully connected layers, that is:

[0115]

[0116] Among them, f c (·) represents the FC layer followed by the PreLu function. Finally, obtain the output F cout of the channel attention module, that is:

[0117]

[0118] In the formula, S c [i] represents the channel attention weight, the weight value of each channel i, and F in [i, h, w] represents the value of the input feature at the channel i, spatial position h, and w. ⊙ represents the Hadamard product, that is, element-wise multiplication.

[0119] The core function of the channel attention module is to skillfully map the input features to the corresponding subcarriers according to the estimated CSI. The channel attention module allocates and maps features to different subcarriers of OFDM based on the estimated CSI, enabling the system to better adapt to channel changes and transmit coded information in a semantic task-oriented and differentiated manner. This mechanism endows JSCC with the excellent ability to dynamically adapt to different channel signal-to-noise ratios, providing strong guarantees for the stability and reliability of the system.

[0120] However, in the process of calculating the channel attention mask, due to the use of average pooling operation, the spatial information of the image is ignored in this link.

[0121] In view of this, in step S4 of this embodiment, a spatial attention module is innovatively designed to effectively make up for this missing spatial information, thereby further improving the overall performance of the system and ensuring the integrity and accuracy of image transmission.

[0122] First, along the channel direction for F cout Apply average pooling operation to obtain F sa , that is:

[0123]

[0124] In the formula, Ave s (·) is the average pooling operation.

[0125] Then, connect and F sa to obtain the intermediate feature i s , that is:

[0126]

[0127] i s Calculate the spatial mask S s through several FC layers, that is:

[0128]

[0129] In the formula, f s (·) represents the spatial mask calculation function, and h, w represent the width and height.

[0130] Finally, calculate the output feature vector, that is:

[0131]

[0132] In the formula, S s [j,k] represents the spatial attention weight, reflecting the importance of the spatial position, F cout[c, j, k] represents the eigenvalue at channel c and spatial position (j, k) in the output of the channel attention module; j is the index of the spatial dimension.

[0133] The spatial attention module can learn to match the more critical spatial features of the image in the h and w dimensions with better channel conditions based on the estimated CSI. By leveraging spatial information, the PSNR performance is further improved, and it helps the JSCC encoder to perform more adaptive power allocation, enabling critical features to match better channels, thereby enhancing the adaptability of the entire system to channel conditions and transmission performance.

[0134] In step S5 of this embodiment, in the OFDM transmission system, it includes an OFDM transmitter and an OFDM receiver. Among them, the OFDM transmitter and the OFDM receiver communicate through a multipath fading channel. The channel consists of t a multipath fading channel with L s paths, and N f OFDM symbols are used to carry image information. At the same time, to ensure accurate channel estimation for L p OFDM subcarriers, N θ pilot signals are also attached. The encoder E maps the input image to a complex matrix. The size of the matrix is c×h×w, representing the color channel, height, and width of the image respectively. The generated channel input is:

[0135]

[0136] where, is the channel state information (CSI) estimation vector available at the transmitter, and E θ (·) is the encoding operation of the encoder.

[0137] The channel input is subject to an average power constraint, that is:

[0138]

[0139] where, denotes taking the expectation, and the expectation depends on the input image. ||·|| is the Frobenius norm, and the subscript "F" indicates that the current norm is the Frobenius norm, which is used to measure the modulus of the matrix. For simplicity of analysis, this embodiment assumes P s = 1.

[0140] At the transmitting end, the encoded symbols generate OFDM symbols through the inverse discrete Fourier transform (IFFT), and a cyclic prefix (CP) is added, and then transmitted through the multipath channel. The time-domain transfer function of the multipath fading channel is defined as:

[0141] y = h * x + w

[0142] where x and y represent the input and output signal vectors respectively, w is the additive white Gaussian noise (AWGN), * is the linear convolution operation, and h is the channel impulse response.

[0143] At the receiver, the received symbols are demodulated by removing the cyclic prefix and applying the fast Fourier transform (FFT). The equivalent frequency-domain transfer function is:

[0144]

[0145] where the frequency-domain channel matrix is a diagonal matrix, and the k-th diagonal element is H[k], which represents the frequency-domain channel response of the k-th subcarrier. Let represent the output of the power normalization module at the transmitter, i.e., the input to the IFFT module, where represents the i-th OFDM symbol at the k-th subcarrier. is the frequency-domain noise matrix, where and they are independent of each other.

[0146] At the receiver, based on the FFT output of the pilot symbols, the minimum mean square error (MMSE) or least squares (LS) method is used to estimate the frequency-domain CSI, and then the estimated CSI vector is used for equalization processing (such as the MMSE equalizer) to obtain the equalized data. There is,

[0147]

[0148] where is the estimated channel gain of the i-th subcarrier; μ is the average SNR of the transmit power P s and the noise power σ 2 , defined as

[0149] The equalized data and the estimated CSI vector are input to the decoder D φ to reconstruct the transmitted image

[0150]

[0151] where D φ (·) represents the decoding operation of the decoder D φ , and the decoder is consistent with the encoder in structure.

[0152] During the training process, it is divided into two stages: semantic segmentation and semantic communication. Different loss functions are used for optimization respectively, and the loss is minimized through the gradient descent method until the stopping criterion is met.

[0153] The loss function in the semantic segmentation stage is calculated based on the true labels and weight maps of pixels. The loss function in the semantic communication stage is calculated using the mean square error (MSE), channel estimation loss, and text loss function.

[0154] The performance metric of image quality is defined using the peak signal-to-noise ratio (PSNR):

[0155]

[0156] where max x represents the maximum possible value x of the input signal, and the expectation is taken over all pixels. The decoder fuses multi-level features for decoding to achieve the classification of image pixels and determine whether they belong to the text region and the background region without text.

[0157] The encoder and decoder are trained through end-to-end joint optimization, and the goal is to minimize the loss function:

[0158]

[0159] The expectation of the loss function takes into account the randomness in the source distribution and channel distribution. Through joint training, the system can adapt to the channel conditions and improve the image transmission quality.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the purpose and scope of the present technical solution, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A text-based screen content image transmission method based on U-Net semantic segmentation, characterized by: The method comprises the following steps: Obtaining a dataset of original text-based screen content images, preprocessing the dataset, and building an image semantic knowledge base based on the dataset; U-Net is used to perform semantic segmentation on the input text-type screen content image, dividing the image into a text area and a background area without text; The encoder encodes the features of the divided text area and background area, and the channel attention module maps the encoded feature information to different subcarriers of the OFDM transmission system according to the channel state information; Then, in the encoder, the spatial features of the image are captured through the spatial attention feature module, and the power allocation of the key features of the OFDM transmission system is optimized according to the spatial features; The information is transmitted through the OFDM transmission system, received at the receiving end and semantically decoded to reconstruct the image and perform corresponding semantic tasks.

2. According to claim 1, a text-based screen content image transmission method based on U-Net semantic segmentation is characterized in that: The original datasets can be obtained by: actively collecting data through technical means, obtaining screen images in real time and saving them as datasets, including capturing application interfaces and dynamic operations; using device cameras or screen recording software to record screen operations, and then generating image datasets by extracting frames from the video; using existing open source screen content datasets; Then the dataset is preprocessed by data cleaning, integration, transformation and reduction, and the preprocessed dataset is divided into training set and test set; At the same time, the semantic knowledge base in the image semantic communication system is constructed through the preprocessed data set to support the efficient encoding and decoding of semantic information. The construction methods of the semantic knowledge base include manual construction, automatic construction and hybrid construction. Manual construction relies on domain experts to define concepts, attributes and relationships through formal language. Automatic construction is based on machine learning and natural language processing technology to extract entities, relationships and rules from text. Hybrid construction combines manual annotation and automation methods to improve quality through iterative optimization. Ontology engineering realizes hierarchical organization of knowledge through top-level framework design, and knowledge graph emphasizes multi-source data fusion and structured storage.

3. According to the method for transmitting text-based screen content images based on U-Net semantic segmentation according to claim 1, it is characterized in that: Assume that the input text-type screen content image is p, then the advanced semantic segmentation technology based on U-Net divides it into a text area and a background area without text, marked as x t and x b : [x t ,x b ]=f U-Net (p) Where p is the input text-type screen content image, f U-Net (·) is a semantic segmentation network; the encoder of the U-Net network extracts features at different levels through multiple downsampling.

4. The method for transmitting text-based screen content images based on U-Net semantic segmentation according to claim 3, characterized in that: The encoder adopts a comb structure design, which includes several feature learning modules FL and several channel learning modules CL, among which, The feature learning module FL is used to learn the intermediate representation of the input, which includes a 2D convolution or deconvolution layer, a batch normalization layer, and a PReLU layer; the 2D convolution or deconvolution layer extracts and reconstructs the feature information of the image by performing specific convolution and deconvolution operations on the input data; the batch normalization layer normalizes the input of each layer during the training process to accelerate the convergence of the model; the PReLU layer, as an activation function, introduces nonlinear factors to enhance the expressiveness of the model; The channel learning module CL is used to learn the attention mask; the channel learning module CL is based on the dual attention mechanism, which includes a channel attention module and a spatial attention module. The channel attention module channel state information CSI evaluates and analyzes the channel quality of different subcarriers, and allocates and adjusts the mapping weights of features on subcarriers according to the channel quality; the spatial attention module considers the spatial structure and feature distribution of the image from the spatial dimension to optimize the mapping method of features on subcarriers.

5. The method for transmitting text-based screen content images based on U-Net semantic segmentation according to claim 1, characterized in that: In the channel attention module, the text region x is firstly transformed into t and background area x b Encoded as input feature codeword Right now: F in =[f t ,f b ]=f CNN [x t ,x b ] In the formula, f CNN [·] represents the convolutional neural network feature extraction operation, f t ,f b They are the text features and background features extracted by convolutional neural network respectively; f t ,f b The characteristic codewords are composed of Represents a real tensor space of dimension c×h×w, where c, h, and w represent the number of channels, height, and width respectively; Then, the input feature F is transformed along the spatial direction in Apply average pooling A vec (F in ) to obtain a real vector in, Then, F ca and the estimated CSI vector Connect them to get the intermediate vector i c ,Right now: Among them, the CSI vector information contains the different subcarrier allocation and mapping characteristics of OFDM. c, L f denote the number of channels and the number of subcarrier groups respectively; concatenate(·) denotes the concatenation function; The channel attention mask S is calculated through several fully connected layers c ,Right now: Among them, f c (·) indicates the FC layer followed by the PreLu function, and finally the output F of the channel attention module is obtained. cout ,Right now: In the formula, S c [i] represents the channel attention weight, the weight value of each channel i, F in [i,h,w] represents the value of the input feature in channel i, spatial position h, w, and ⊙ represents the Hadamard product, that is, the multiplication of corresponding elements.

6. The method for transmitting text-based screen content images based on U-Net semantic segmentation according to claim 5, characterized in that: In the spatial attention module, first, F cout Apply the average pooling operation to get F sa ,Right now: In the formula, Ave s (·) is the average pooling operation; Then, and F sa Connect them to get the intermediate feature i s ,Right now: i s The spatial mask S is calculated through several FC layers s ,Right now: In the formula, f s (·) represents the spatial mask calculation function, h and w represent the height and width; Finally, the output feature vector is calculated, namely: In the formula, S s [j, k] represents the spatial attention weight, reflecting the importance of the spatial position, F cout [c,j,k] represents the eigenvalue of channel c and spatial position (j,k) in the output of the channel attention module; j is the index of the spatial dimension.

7. The method for transmitting text-based screen content images based on U-Net semantic segmentation according to claim 1, characterized in that: In the OFDM transmission system, there are an OFDM transmitter and an OFDM receiver, wherein the OFDM transmitter and the OFDM receiver communicate through a multipath fading channel; the channel is composed of L t The multipath fading channel consists of N paths. s OFDM symbols are used to carry image information, and by adding N p A pilot signal is used to ensure that L f OFDM subcarriers for accurate channel estimation; Encoder E θ The input image Mapping to a complex matrix The size of the matrix is ​​c×h×w, which represents the color channel, height, and width of the image respectively. The generated channel input is: in, is the channel state information estimate vector available at the transmitter, E θ (·) is the encoding operation of the encoder; The channel input is subject to an average power constraint, namely: in, It means to seek the expectation, which depends on the input image, ||·|| is the Frobenius norm, and the subscript "F" indicates that the current norm is the Frobenius norm, which is used to measure the modulus of the matrix.

8. The method for transmitting text-based screen content images based on U-Net semantic segmentation according to claim 7, characterized in that: At the transmitter, the coded symbols are transformed into OFDM symbols through the inverse discrete Fourier transform IFFT, and the cyclic prefix CP is added, and then transmitted through the multipath channel; the time domain transfer function of the multipath fading channel is defined as: y=h*x+w Where x and y represent the input and output signal vectors respectively, w is additive white Gaussian noise, * is the linear convolution operation, and h is the channel impulse response; The receiver demodulates the received symbols by removing the cyclic prefix and applying a fast Fourier transform (FFT); the equivalent frequency domain transfer function is: Among them, the frequency domain channel matrix is a diagonal matrix, the kth diagonal element is H[k], which represents the frequency domain channel response of the kth subcarrier; let It represents the output of the power normalization module at the transmitter, which is the input of the IFFT module; represents the i-th OFDM symbol at the k-th subcarrier; is the frequency domain noise matrix, where And independent of each other.

9. The method for transmitting text-based screen content images based on U-Net semantic segmentation according to claim 8, characterized in that: At the receiving end, based on the FFT output of the pilot symbol, the minimum mean square error (MMSE) or least squares (LS) method is used to estimate the frequency domain CSI, and then the estimated CSI vector is used for equalization to obtain the equalized data: in, is the estimated channel gain of the ith subcarrier; μ is the transmit power P s and noise power σ 2 The average SNR of Balanced data and the estimated CSI vector is input to decoder D φ , reconstruct the transmission image Among them, D φ (·) represents the decoder D φ The decoding operation of the decoder is consistent with the encoder in structure.

10. A method for transmitting text-based screen content images based on U-Net semantic segmentation according to any one of claims 1 to 9, characterized in that: The training process of the method is divided into two stages: semantic segmentation and semantic communication. Different loss functions are used for optimization respectively, and the loss is minimized by gradient descent until the stopping criterion is met. The loss function in the semantic segmentation stage is calculated based on the true label of the pixel and the weight map; The loss function in the semantic communication stage is calculated using mean square error (MSE), channel estimation loss, and text loss function. The performance indicator of image quality is defined using peak signal-to-noise ratio (PSNR): Where max x represents the maximum possible value of the input signal And the expectation is made for all pixels; The encoder and decoder are trained through end-to-end joint optimization to minimize the loss function: Among them, the expectation of the loss function takes into account the randomness in the source distribution and channel distribution.

Images