Screen content image transmission method based on semantic communication

Abstract

The invention relates to a screen content image transmission method based on semantic communication, and belongs to the technical field of wireless communication and multimedia transmission. The method comprises the following steps: carrying out information source coding on an input text type screen content image through a coder; the method comprises the following steps of: carrying out weight distribution on code words after source coding through a weight distribution network, so that important semantic features are preferentially transmitted on a high-SNR (Signal to Noise Ratio) subcarrier; channel coding is carried out on the code words after weight distribution, and more transmission power is distributed for high SNR subcarriers in an OFDM (Orthogonal Frequency Division Multiplexing) transmitter; after the OFDM receiver receives the image, the text in the image is detected and recognized through the OCR and the scene text detection model, and the text level confidence coefficient is calculated; and designing a loss function based on the text level confidence, and protecting key text semantic features. According to the invention, the transmission quality of the image is improved, the transmission bandwidth is saved, and the problem that the traditional screen content image coding method is limited by the Shannon limit is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communication and multimedia transmission, and relates to application scenarios that require efficient image transmission, including remote conferencing, cloud gaming, online education, and real-time remote collaboration, and in particular to a screen content image transmission method based on semantic communication. Background Art

[0002] According to Google's research, 90% of online media interactions are screen-based, and screen content has become an important type of Internet information. Generally speaking, screen content is a general term for data generated or rendered by a computer, which has the characteristics of noise-free, sharp edges, repeated patterns, and high contrast relative to natural scene images. Text-based screen content images and videos refer to image and video data of screen display content dominated by text, which are static images that capture the text content displayed on the screen. They usually contain a large amount of text and may also include a small number of graphics and charts. This type of data usually contains documents, web pages, presentations, code editors, and other screen content dominated by text information. At present, China is vigorously promoting economic development led by the information industry. With the rapid development of intelligent communication systems based on the Internet of Things, including multimedia and cloud technologies, online media interactions have been widely used. Application scenarios include cloud video conferencing, remote screen sharing, online games, online education, etc. These processes provide convenience while also bringing huge amounts of data. The explosive growth of screen content data has brought considerable challenges to image and video coding technology. Therefore, in the face of the above-mentioned multiple application scenarios, it is imperative to study efficient screen content coding and transmission schemes.

[0003] Traditional image compression uses standards such as JPEG, JPEG2000, H.265 / HEVC and H.266 / VVC, and the recent Learned Image Compression (LIC) algorithm has surpassed H.266 / VVC. However, these methods are mainly designed for natural scene images, which limits their applicability to the special properties of screen content and poses challenges to the coding and transmission technology of screen content images. Many traditional coding standards have integrated screen content coding (SCC), making some progress, but there is still a lack of a comprehensive solution for the coding and transmission of screen content images. Shannon's separation theorem points out that in the case of infinite length codewords, it is optimal to design source coding and channel coding separately. Separate modular design has achieved great success in actual communication systems. From the development of the first generation to the fifth generation of mobile communication systems, traditional communication system design adopts a modular approach, that is, the source coding, channel coding and modulation modules of the encoder are designed separately. However, in various wireless environments, such as emerging wireless applications such as autonomous driving, smart manufacturing, and telemedicine, the implementation of communication systems is not realistic due to latency considerations, and the separate modular design approach is insufficient in the face of current complex communication requirements. Therefore, it is desirable to jointly optimize the source channel coding relative to shorter information, and this method is called Joint Source-Channel Coding (JSCC). The transmission rate of the screen content coding method of the traditional communication system is close to the Shannon limit, which poses a severe challenge to the existing requirements of low latency and high transmission rate. Due to the explosive growth of data volume, the existing source channel coding method has great problems in communication systems with limited bandwidth resources and low latency requirements. Moreover, traditional communication systems do not pay attention to the semantic aspects of information, but only pursue the precise transmission of information symbols, which exposes its limitations in the era of artificial intelligence. The above problems have prompted researchers to explore the paradigm of the next generation of communication systems.

[0004] As an emerging communication technology, semantic communication can greatly improve communication efficiency by extracting, encoding and transmitting information semantics. Compared with traditional communication, semantic communication does not require accurate transmission of data or communication symbols, but focuses on the matching between the semantic information input by the sender and the semantic information recovered by the receiver. By filtering out redundant information, extracting the meaning of effective information, and transmitting truly useful information, it can achieve the goal of greatly reducing communication resource consumption. Moreover, unlike traditional communication that uses bit error rate (BER) and symbol error rate (SER) as communication evaluation results, semantic communication systems restore the source information at the decoder end in a way that minimizes the semantic loss between input information and reconstructed information. In addition, with the rapid development of artificial intelligence, its ability to master human knowledge in limited scenarios is gradually improving. As one of the most important technologies in artificial intelligence, deep learning has achieved great success in understanding sources such as language, audio, pictures, and videos. Since deep learning can efficiently extract and transmit the semantic information contained in the source, and has the advantages of good feature extraction and learning capabilities, it has been widely used in semantic communication models. Semantic communication technology based on deep learning can not only optimize the traditional source, channel coding and modulation and demodulation modules, but also effectively address the bottleneck problems in traditional communication systems by establishing end-to-end (E2E) JSCC. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a screen content image transmission method based on semantic communication, so as to improve the transmission efficiency and robustness of text-type screen content images and the completion efficiency of downstream tasks.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for transmitting screen content images based on semantic communication, the method comprising:

[0008] For the input text-like screen content image, the encoder performs source encoding on it to reduce the redundancy in the source;

[0009] For the codewords after source encoding, a weight distribution network is used to assign weights to them, so that important semantic features are transmitted preferentially on high SNR subcarriers;

[0010] Channel coding is performed on the weighted codewords to allocate more transmission power to high SNR subcarriers in the OFDM transmitter;

[0011] After receiving the image, the OFDM receiver detects and recognizes the text in the image through OCR and scene text detection models, and calculates the text level confidence;

[0012] A loss function is designed based on text-level confidence to protect key text semantic features.

[0013] Further, weighting the codewords after source coding by a weighting distribution network includes: for a set of codewords obtained after source coding The weights are assigned through the weight assignment network. Specifically, the weight matrix generated by the weight assignment network is multiplied element by element with the codeword Y to achieve weight assignment; wherein the weight matrix is ​​expressed as W = [w1, w2, ..., w n ], The weight matrix is ​​multiplied element by element with the codeword Y to obtain the symbol sequence Symbol Sequence This is the OFDM symbol that will eventually be sent to the channel. The OFDM transmitter will Mapped to multiple subcarriers, parallel transmission is used to improve bandwidth utilization and anti-interference capabilities.

[0014] Furthermore, for the weight distribution network, its training process includes: taking the codeword Y after source encoding and the channel frequency response H as the input of the weight distribution network, and generating a weight matrix W for weight classification after training, wherein the weight matrix W has the same shape as Y, that is, each codeword of Y corresponds to a weight.

[0015] Furthermore, the weight distribution network includes two convolutional layers, two batch normalization layers and one ReLU layer.

[0016] In this network, the input data is extracted through the first convolutional layer to produce a set of feature maps:

[0017] Y1=W1*X+b1

[0018] In the formula, W1 represents the convolution kernel, “*” represents the convolution operation, X represents the input data, and b1 represents the bias;

[0019] These feature maps go through the first batch of normalization layers to standardize the feature map data:

[0020]

[0021] In the formula, μ and σ represent the mean and variance of the mini-batch data, and γ and β represent the learnable parameters;

[0022] The normalized feature map is transformed nonlinearly through the ReLU layer:

[0023] Y3=max(0,Y2)

[0024] The feature map after nonlinear transformation is further extracted through the second convolutional layer:

[0025] Y4=W2*Y3+b2

[0026] In the formula, W2 represents the convolution kernel, b2 represents the bias;

[0027] These feature maps are again normalized by a second batch of normalization layers:

[0028]

[0029] Where μ' and σ' represent the mean and variance of the mini-batch data, and γ' and β' represent learnable parameters.

[0030] Furthermore, the text in the image is detected and recognized by OCR and scene text detection models, and the text level confidence is calculated, including: preprocessing the received image, and then detecting the text region in the image by the DB detection model; wherein the text region of the image is detected by the TextRegion(·) function, and a list of bounding boxes of all individual text regions in the image is returned:

[0031] B={B1,B2,…,B n}=TextRegion(x)

[0032] In the formula, B i represents the detected text box, and x represents the input image;

[0033] For each B i Define the coordinates and crop the input image x according to the coordinates to get the text area:

[0034] b i =B i (x)

[0035] Where b i represents the i-th cropped text region of x;

[0036] The cropped text area is input into the CRNN model, and each character in the text area is recognized by the Recognize(·) function:

[0037] c j =Recognize(b i )

[0038] In the formula, c j Represents the confidence of each character; the confidence of all characters is averaged as the text level confidence: n represents the total number of characters in the recognition result.

[0039] Construct the text loss based on the calculated text-level confidence:

[0040]

[0041] In the formula, Represents the reconstructed image; the total loss function is constructed by combining text loss, reconstruction loss and channel estimation loss:

[0042]

[0043] Where, L rec Represents the loss function between the original image and the reconstructed image, L ce represents the channel estimation loss at the receiver end, and λ represents the weight parameter that controls the importance of the text loss function.

[0044] The beneficial effects of the present invention are:

[0045] (1) The present invention effectively improves the transmission quality of images and saves transmission bandwidth through end-to-end semantic coding and decoding and OFDM transmission framework, and solves the problem that traditional screen content image coding methods are limited by the Shannon limit.

[0046] (2) The present invention utilizes a dynamic encoding strategy based on channel state information (CSI) feedback and an additional loss function to protect key text semantic features, thereby achieving differentiated transmission of transmission parts of different importance. Compared with traditional screen content encoding and transmission methods, the present invention not only improves transmission efficiency, but also enhances the robustness and reliability of the system in a loaded wireless channel environment. Compared with traditional screen content image transmission schemes, the present invention is particularly superior under low SNR and low bit rate conditions.

[0047] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0049] Figure 1 A structural diagram of the method described in this statement;

[0050] Figure 2 Source channel coding with CSI feedback;

[0051] Figure 3 Schematic diagram of weight allocation. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0053] An embodiment of the present invention provides a method for transmitting screen content images based on semantic communication, the method comprising:

[0054] S1. Semantic feature extraction and encoding

[0055] The semantic features of the input text-based screen content image are extracted, and the text image is converted into a semantic feature representation through an end-to-end semantic framework based on a deep learning model. The semantic encoder analyzes the image content, extracts important text information from the image, and compresses or discards irrelevant data. By reducing the amount of unnecessary transmitted data, efficient encoding of key semantic features is achieved.

[0056] The details are as follows:

[0057] Assume that there is a screen content image x. After converting it into a semantic feature representation, the encoder first performs source encoding S(·) on it to reduce the redundancy in the source. Given an input image of size H×W, the output size of the encoder is Where d is the downsampling factor and C is the output channels.

[0058] At this point, the encoder output is real-valued, and is converted to complex by mapping half to the real part and the other half to the imaginary part. It is then reshaped into Where L fft represents the number of subcarriers in the OFDM symbol, and Y represents the frequency domain complex valued symbol to be fed to the OFDM encoder. It is known to both the encoder and the decoder, and assumes that N p All pilot symbols are the same.

[0059] Subsequently, an inverse discrete Fourier transform (IDFT) is performed on each OFDM symbol to transform the frequency domain signal into a time domain signal for transmission in the time domain, and a cyclic prefix CP is added. Secondly, OFDM power allocation is performed. In an orthogonal frequency division multiplexing system, the transmission performance of frequency domain symbols is directly affected by the channel frequency response. In an OFDM system, the channel state of each subcarrier can be different, and changes in SNR will affect the transmission performance. Resources are allocated based on the SNR of each subcarrier to ensure that important semantic features are transmitted on subcarriers with high SNR to reduce the bit error rate.

[0060] Each source image x is passed through a p pilot symbols and N s The pilot symbols are used to transmit information symbols. The pilot symbols are used for channel estimation and synchronization. They are known symbols that can help the decoder understand the state of the channel so as to better receive and decode the information symbols. The information symbols are the actual transmitted data. For a general OFDM system, the encoded OFDM information symbol y will be modulated onto n subcarriers, that is, y = [s1, s2, ..., s n ].

[0061] S2. Optimize coding parameters through real-time channel state information (CSI) feedback. In OFDM systems, channel states often change dynamically over time. Therefore, the present invention designs a dynamic coding algorithm that adjusts coding power allocation based on the CSI feedback mechanism so that important semantic feature data can be transmitted preferentially on high SNR subcarriers, thereby ensuring the quality and stability of image transmission under low SNR or complex channel conditions. The algorithm combines the instantaneous state of the channel with feedback information so that subcarriers with good channel conditions give priority to transmitting key semantic features, thereby improving data transmission quality. The algorithm is specifically shown in Table 1 below:

[0062] Table 1

[0063]

[0064] like Figure 2 and 3 As shown in the figure, the source text screen content image x passes through the source encoder to obtain a set of codewords Y = [y1, y2, ..., y n ], These codewords Y are the coded representation of the image information and the vectorized representation of the original image data for subsequent transmission. The weight matrix W = [w1,w2,…,w n ], It is generated by a weight assignment network, which learns an optimization strategy during training, that is, to assign weights to the encoded codewords Y according to the optimization goals of the downstream text recognition task of the channel characteristics and semantic communication, giving higher priority to certain data that are conducive to the completion of downstream tasks, to influence the various parts of the transmitted image information. The codeword Y and the weight W are multiplied element by element, that is, s i =y i ·w i , forming a new symbol sequence Through the modulation of weight W, the encoded codeword Y is adjusted to form the OFDM symbol to be sent to the channel. The OFDM transmitter converts OFDM symbols Mapped to multiple subcarriers, parallel transmission is used to improve bandwidth utilization and anti-interference capabilities.

[0065] After the system performs channel estimation, the receiver feeds back the estimated CSI of each subcarrier to the transmitter in real time, including parameters such as SNR and channel gain. The entire process of CSI feedback can be described by the following probability model:

[0066]

[0067] Where v represents the output of the compression layer, p φ (v|h'), p(z|v), They are defined for compression, quantization, feedback channel and decoder respectively.

[0068] The transmitter allocates power based on this feedback information. During the channel coding process, more transmission power is allocated to high SNR subcarriers in the OFDM system to transmit images at the maximum transmission rate and minimize the bit error rate. In addition, in order to ensure that important text semantic features are transmitted first, the system can allocate high SNR subcarriers to these features first based on the importance of semantic features.

[0069] The structure of the proposed weight encoding network includes two convolutional layers, two batch normalization layers, and an activation function ReLU layer. It extracts and normalizes features from the input data. The convolutional layers extract features, the batch normalization layers stabilize the training process, and the ReLU introduces nonlinearity. The whole process helps the network to better learn the complex patterns of the input data. First, the input data passes through the first convolutional layer to extract features and produce a set of feature maps. This process can be expressed by the following formula:

[0070] Y1=W1*X+b1

[0071] Among them, W1 is the convolution kernel, "*" indicates the convolution operation, X is the input data, and b1 is the bias. Then, these feature maps pass through the batch normalization layer to standardize the feature map data:

[0072]

[0073] Among them, μ and σ are the mean and variance of the mini-batch data, and γ and β are learnable parameters. Then, the normalized feature map is activated by the ReLU function and a nonlinear transformation is applied:

[0074] Y3=max(0,Y2)

[0075] Next, the feature map after nonlinear transformation passes through the second convolutional layer to further extract features:

[0076] Y4=W2*Y3+b2

[0077] Among them, W2 is the second convolution kernel and b2 is the second bias. Finally, these feature maps are again passed through the batch normalization layer for standardization:

[0078]

[0079] Among them, μ' and σ' are the mean and variance of the new mini-batch data, and γ' and β' are new learnable parameters. This subnetwork extracts and normalizes features from the input data, extracts features through convolutional layers, stabilizes the training process through batch normalization layers, and introduces nonlinearity through ReLU. The whole process helps the network better learn the complex patterns of the input data.

[0080] S3. At the receiving end, the transmitted data is decoded and reconstructed through an end-to-end semantic decoder. By combining CSI feedback and text protection mechanisms, the receiving end can effectively restore the semantic content of the image, especially text information, to ensure the reconstruction effect of the image at the receiving end. Through the precise semantic decoding process, the image restored by the receiving end can be highly consistent with the original image visually, thereby ensuring the semantic accuracy of the image content while ensuring transmission efficiency.

[0081] During the image transmission process, the main semantic information of the transmitted data is determined, and it is screened and optimized according to the needs of downstream tasks. How to identify the key objects and features in the image and transmit this information first so that the receiver can complete the task quickly and accurately is a problem that needs to be considered during the transmission of screen content images. In addition, images containing a large amount of text, especially screen content images, often suffer from text distortion during encoding and transmission. In order to address the quality of text reconstruction and the problem of semantic communication downstream tasks during image encoding and transmission, this method proposes a loss function for text recovery, which aims to measure the difference in text fidelity between the original image and the reconstructed image.

[0082] Specifically, we use Optical Character Recognition (OCR) and scene text detection models to detect and recognize text in images, and build the proposed loss function on this basis as an addition to the total loss. By minimizing this loss, we can enhance the perceptual similarity of text between two images. The total loss function consists of several parts, including the loss L that measures the difference between the original image and the reconstructed image. rec , and the channel estimation loss L at the decoder ce In addition, we introduce a text loss function L t As part of the overall loss function, it is used to characterize the difference in text fidelity.

[0083] Among them, the text loss L t Based on the reconstructed image The text confidence list is calculated by identifying, cropping and identifying the text region of the image. The recognition of text information includes four steps: image preprocessing, text detection, text region cropping and text recognition. First, image preprocessing performs grayscale and binarization preprocessing operations on the input image to improve the effect of text recognition. Then, text detection uses the pre-trained DB model to detect the text region in the image. The text region of the image is detected by the function TextRegion(·) and a list of bounding boxes of all individual text regions in the image is returned.

[0084] B={B1,B2,…,B n}=TextRegion(x)

[0085] Next, the text area in the image is cropped according to the detected text box. i , can be defined by the coordinates of the upper left corner and the lower right corner. i As a function, the input image is cropped according to the coordinates to obtain the text area,

[0086] b i =B i (x)

[0087] in represents the i-th cropped text region of x, whose height is h i , with a width of w i Finally, the cropped text region is input into the pre-trained model CRNN, and the function Recognize(·) can recognize each character in the given text region.

[0088] c i =Recognize(b i )

[0089] where c i Indicates the confidence of each character. Its value is usually between 0 and 1, indicating the degree of trust the model has in the recognition result. The higher the confidence, the more confident the model is in the recognition result.

[0090] The confidence of all characters is averaged as the confidence of the entire text recognition result. Assume that the recognition result contains n characters and the character-level confidence is {c1,c2,...,c n}, then the text-level confidence is calculated as:

[0091]

[0092] After obtaining the text recognition confidence, it can be applied to the design of the loss function, and the text loss is defined as:

[0093]

[0094] The text loss function After adding the weights, the total loss function is:

[0095]

[0096] Among them, L rec is the loss function between the original image and the reconstructed image, L ce is the channel estimation loss at the receiver end, λ is the weight parameter that controls the importance of the text loss function, and λ can be set to 0.8.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for transmitting screen content images based on semantic communication, characterized in that: The method includes: For the input text-like screen content image, the encoder performs source encoding on it to reduce the redundancy in the source; For the codewords after source encoding, a weight distribution network is used to assign weights to them, so that important semantic features are transmitted preferentially on high SNR subcarriers; Channel coding is performed on the weighted codewords to allocate more transmission power to high SNR subcarriers in the OFDM transmitter; After receiving the image, the OFDM receiver detects and recognizes the text in the image through OCR and scene text detection models, and calculates the text level confidence; A loss function is designed based on text-level confidence to protect key text semantic features.

2. The method according to claim 1, characterized in that: The weight distribution of the codewords after source coding by a weight distribution network includes: The weight distribution is performed through the weight distribution network, wherein the weight matrix generated by the weight distribution network is multiplied element by element with the codeword Y to achieve weight distribution; the weight matrix is ​​expressed as The weight matrix is ​​multiplied element by element with the codeword Y to obtain the symbol sequence 3. The method according to claim 1 or 2, characterized in that: The codeword Y after source encoding and the channel frequency response H are used as the input of the weight allocation network, and a weight matrix W for weight classification is generated after training, wherein the weight matrix W has the same shape as Y, that is, each codeword of Y corresponds to a weight.

4. The method according to claim 3, characterized in that: The weight distribution network includes two convolutional layers, two batch normalization layers and an activated ReLU layer; The input data is passed through the first convolutional layer for feature extraction, generating a set of feature maps: Y1=W1*X+b1 In the formula, W1 represents the convolution kernel, "*" represents the convolution operation, X represents the input data, and b1 represents the bias; These feature maps go through the first batch of normalization layers to standardize the feature map data: In the formula, μ and σ represent the mean and variance of the mini-batch data, and γ and β represent the learnable parameters; The normalized feature map is transformed nonlinearly through the ReLU layer: Y3=max(0,Y2) The feature map after nonlinear transformation is further extracted through the second convolutional layer: Y4=W2*Y3+b2 In the formula, W2 represents the convolution kernel, b2 represents the bias; These feature maps are again normalized by a second batch of normalization layers: Where μ' and σ' represent the mean and variance of the mini-batch data, and γ' and β' represent learnable parameters.

5. The method according to claim 1, characterized in that: The text in the image is detected and recognized by OCR and scene text detection models, and the text level confidence is calculated by preprocessing the received image and then detecting the text region in the image by DB detection model; wherein the text region of the image is detected by TextRegion(·) function, and a list of bounding boxes of all individual text regions in the image is returned: B={B1,B2,…,B n }=TextRegion(x) In the formula, B i represents the detected text box, and x represents the input image; For each B i Define the coordinates and crop the input image x according to the coordinates to get the text area: b i =B i (x) Where b i represents the i-th cropped text region of x; The cropped text area is input into the CRNN model, and each character in the text area is recognized by the Recognize(·) function: c j =Recognize(b i ) In the formula, c j Represents the confidence of each character; the confidence of all characters is averaged as the text level confidence: n represents the total number of characters in the recognition result.

6. The method according to claim 1 or 5, characterized in that: Construct the text loss based on the calculated text-level confidence: In the formula, represents the reconstructed image; The total loss function is constructed by combining text loss, reconstruction loss and channel estimation loss: Where, L rec Represents the loss function between the original image and the reconstructed image, L ce represents the channel estimation loss at the receiver end, and λ represents the weight parameter that controls the importance of the text loss function.

Images