Semantic communication method based on image content adaptive coding

Abstract

The application discloses a semantic communication method based on image content adaptive coding, comprising the following steps: clustering image blocks by initial connection probability capable of representing the correlation between the image blocks to obtain a parsing tree of the image to be transmitted; determining the coding length of the image blocks according to the connection probability of the image blocks in the parsing tree, and coding the image blocks according to the coding length. According to the method provided by the application, the image blocks are clustered by the initial connection probability capable of representing the correlation between the image blocks to obtain the parsing tree of the image to be transmitted, and since the structural complexity of the parsing tree is proportional to the complexity of the semantic information in the image, the coding length of the image with more complex semantic information is longer according to the method provided by the application, and the coding length of the image with simple semantic information is shorter, so that the image is adaptively coded according to the semantic characteristics of the image, and the utilization rate of the communication resource is improved.

Description

Technical Field

[0001] This invention belongs to the field of semantic communication technology, specifically relating to a semantic communication method based on adaptive encoding of image content. Background Technology

[0002] Semantics is a highly abstract knowledge carrier, typically represented by image features understood by machines or image content understood by humans. It is used to describe the structure, internal features, or mechanisms of signals such as text, images, or videos. Semantics is widely distributed in various modalities of signals, including text, images, and videos, which possess inherent meaning. The analysis and understanding of these special meaning representations are crucial for knowledge extraction and efficient communication, and are also important components of machine intelligence and advanced artificial intelligence. Encoding and decoding semantic features is fundamental to semantic communication. However, existing encoding and decoding methods mainly use fixed-length features or adaptive encoding and decoding based on the statistical characteristics of the source signal and channel conditions, failing to provide adaptive encoding for each specific communication instance and its image features. The main reasons are that the diverse semantic features of specific communication instances are difficult to obtain: firstly, the semantic information of image features is complex, with significant differences in various semantic information types within the source communication samples, making unified representation and processing difficult; secondly, image semantic feature representation cannot effectively guide each communication sample for adaptive encoding based on semantic complexity, meaning it cannot effectively utilize the semantic complexity of images to guide the generation and transmission of image semantics. Current adaptive coding methods do not perform adaptive coding for each communication transmission instance, thus preventing semantic communication coding from optimizing each transmission instance and hindering further improvements in source coding and transmission efficiency in semantic communication.

[0003] Therefore, current semantic communication methods that use fixed-length encoding waste communication resources. Another semantic communication method, which encodes images using statistical features from multiple images in a dataset, cannot adaptively encode each image based on its semantic features. Summary of the Invention

[0004] This invention provides a semantic communication method based on adaptive encoding of image content, which can solve the problem that current semantic communication methods have low utilization of communication resources and cannot achieve adaptive encoding based on the semantic features of each image.

[0005] In a first aspect, embodiments of the present invention provide a semantic communication method based on image content adaptive coding, the method comprising:

[0006] Based on the semantic feature information of the image to be transmitted, the initial connection probability of each image block in the image to be transmitted is determined, where an image block is a part of the image in the transmitted image, and all image blocks constitute the transmitted image.

[0007] The image patches are clustered according to the initial connection probability to obtain the parse tree of the image to be transmitted. The parse tree includes L levels, and the structural complexity of the parse tree is proportional to the complexity of the semantic information contained in the image to be transmitted.

[0008] Based on the connection probability of image blocks in the parse tree, the encoding length of the image block is determined, and the image block is encoded according to the encoding length to obtain the encoded semantic feature vector group of the image to be transmitted.

[0009] The connection probability is determined based on the initial connection probability, and the encoding length of the image patch is proportional to the complexity of the semantic information contained in the image patch.

[0010] In one possible implementation of the first aspect, semantic feature information of the image to be transmitted can be extracted to obtain the position feature vector of each image block in the image to be transmitted, and then the initial connection probability of the image block can be determined based on the position feature vector.

[0011] In one possible implementation of the first aspect, the image to be transmitted can be segmented to obtain multiple image blocks; semantic feature information of each image block can be extracted to obtain a feature vector of each image block; the feature vector can be positionally encoded to embed the position information of the image block into the feature vector to obtain a position feature vector of each image block.

[0012] In one possible implementation of the first aspect, the location feature vector can be input into the source encoder group to obtain X initial connection probabilities for each image block.

[0013] For example, the source encoder group includes X source encoders connected in series.

[0014] For example, the xth source encoder is used to determine the xth initial connection probability of the image patch based on the location feature vector.

[0015] For example, the xth initial connection probability of an image patch is used to determine the connection probability of the image patch at layer l in the parse tree.

[0016] For example, x = l, x = 0, 1, 2...X, l = 1, 2...L, X≥L.

[0017] In one possible implementation of the first aspect, a multi-head self-attention operation can be performed on the image block based on the positional feature vector of the image block to obtain the x-th multi-head self-attention matrix of the image to be transmitted; based on the positional feature vector of the image block, a cross-self-attention operation can be performed on the image block in each branch of the multi-head self-attention operation to obtain the x-th cross-connection probability matrix of the image to be transmitted; the x-th cross-connection probability matrix and the x-th multi-head self-attention matrix can be fused to obtain the x-th probability map of the image to be transmitted.

[0018] For example, the x-th probability map is a probability matrix consisting of the x-th initial connection probability of each image patch.

[0019] In one possible implementation of the first aspect, the connection probability of the (l+1)th layer of the image block can be determined based on the connection probability of the l-th layer of the image block and the initial connection probability of the (x+1)th layer of the image block; starting from the image block cluster of the 0th layer, the image block cluster of the l-th layer is clustered based on the connection probability of the l-th layer of the image block and the first segmentation threshold to obtain the parse tree of the image to be transmitted.

[0020] For example, x = l, the connection probability of the image patch at layer 0 is the initial connection probability of the image patch.

[0021] For example, the image block cluster in layer 0 consists of only one image block.

[0022] For example, an image patch cluster includes at least one image patch arranged sequentially according to the positional information of the image patches.

[0023] In one possible implementation of the first aspect, starting from the image block clusters of layer 0, it can be determined whether the connection probability of the edge image blocks of each image block cluster of layer l is greater than or equal to a first segmentation threshold; if the connection probability of the edge image blocks is greater than or equal to the first segmentation threshold, the image block cluster of layer l to which the edge image block belongs and another image block cluster of layer l adjacent to the edge image block are clustered to determine the image block cluster of layer l+1.

[0024] For example, an edge image block is an image block located at both ends of an image block cluster.

[0025] In one possible implementation of the first aspect, the vector dimension of the image block can be determined based on the connection probability of the Lth layer of the image block in the parse tree; and the encoding length of the image block can be determined based on the vector dimension and the connection probability of the image block.

[0026] In one possible implementation of the first aspect, the quotient of the normalized connection probability and the maximum connection probability of the image block can be determined as the first coding length of the image block; the coding length of the image block is obtained by performing a Kronecker product operation on the first coding length and the vector dimension.

[0027] For example, the normalized connection probability is obtained by normalizing the connection probability.

[0028] For example, the maximum connection probability is the maximum value obtained after normalizing the connection probability of the Lth layer of the image patch.

[0029] In one possible implementation of the first aspect, the parse tree of the image to be transmitted and the encoded semantic feature vector group of the image to be transmitted can also be sent to the destination, so that the destination can decode the encoded semantic feature vector group of the image to be transmitted based on the parse tree of the image to be transmitted to obtain the image to be transmitted.

[0030] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: According to the method provided by the present invention, image blocks are clustered by initial connection probabilities that can characterize the correlation between image blocks to obtain a parse tree of the image to be transmitted. Since the structural complexity of the parse tree is proportional to the complexity of the semantic information in the image, the image to be transmitted is encoded according to the encoding length determined by the connection probabilities of the image blocks in the parse tree. This allows images with more complex semantic information to have a longer encoding length and images with simpler semantic information to have a shorter encoding length. This enables adaptive encoding of images based on their semantic features and improves the utilization rate of communication resources. Attached Figure Description

[0031] Figure 1 A schematic flowchart illustrating a semantic communication method based on image content adaptive coding provided in an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram illustrating a scenario for determining the initial connection probability of a source encoder, as provided in an embodiment of the present invention.

[0033] Figure 3 A schematic diagram of an image to be transmitted, provided as an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of a scenario for determining a parse tree, provided by an embodiment of the present invention;

[0035] Figure 5 This is a schematic flowchart illustrating a semantic communication method based on image content adaptive coding, provided in an embodiment of the present invention. Detailed Implementation

[0036] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0037] Figure 1 This diagram illustrates a semantic communication method based on image content adaptive coding according to an embodiment of the present invention. As an example and not a limitation, method 100 can be applied to the aforementioned electronic device serving as the information source. Method 100 may include steps S101-S103, which are described below.

[0038] S101, determine the initial connection probability of each image block in the image to be transmitted based on the semantic feature information of the image to be transmitted.

[0039] For example, an image block is a portion of the transmitted image, and all image blocks constitute the complete transmitted image.

[0040] In some embodiments, semantic feature information can be extracted from the image to be transmitted to obtain the positional feature vector of each image block in the image to be transmitted. Then, the initial connection probability of each image block in the image to be transmitted and the probability map of the image to be transmitted are determined based on the positional feature vectors.

[0041] In one possible implementation, due to the sparsity of the features of the image to be transmitted and the lack of complex background information, the graphic and semantic feature information of the image to be transmitted can be extracted by a deconvolution-based variational graphic encoder to obtain the positional feature vector of the image to be transmitted.

[0042] For example, a deconvolution-based variational graph encoder can obtain the cross-connection probability of each image block in the image to be transmitted through steps S1011-S1013.

[0043] S1011, The image to be transmitted is segmented to obtain multiple image blocks.

[0044] For example, a deconvolution-based variational graphics encoder can perform superpixel segmentation on the image to be transmitted, resulting in multiple image blocks.

[0045] In one example, taking a w×h×3 image to be transmitted (where w represents width, h represents height, and 3 represents the number of color channels), the variational image encoder can convert the image to the RGB color space, then introduce XY coordinates to obtain a w×h×3 dimensional image to be transmitted. Next, the length and width of the image to be transmitted are scaled to make them consistent, resulting in a w'×h'×3 dimensional image to be transmitted (w'=h'). Then, the image to be transmitted is divided along both the X and Y coordinates to obtain multiple image blocks, each with a size of P×P.

[0046] For example, the image to be transmitted can be converted to the RGB color space, and then XY coordinates can be introduced to obtain a w×h×3 dimensional image to be transmitted. Then, the image to be transmitted is normalized in size, and a region interpolation algorithm is used to obtain a w'×h'×3 dimensional image to be transmitted (e.g., 256×256×3). Finally, it is segmented to obtain multiple image blocks. When P is 16, 256 = w'×h' / P can be obtained. 2 A segmented image block.

[0047] S1012, extract the semantic feature information of each image block to obtain the feature vector of each image block.

[0048] In one example, semantic feature information of the same size can be extracted from each image patch using the ROL (Region of Limiters) Allgn algorithm and convolution operations. Since ROL can only operate on rectangular regions, the superpixels of the image to be transmitted can be divided into m×m rectangular image patches in step S1011. Extracting semantic feature information from image patches using the ROL algorithm and convolution operations avoids rounding operations, preserves the spatial distribution of the original ROL, and effectively avoids the generation of errors.

[0049] For example, semantic feature information of image patches can be extracted first using the ROL algorithm to obtain a high-precision feature map corresponding to the image patch. Then, bilinear interpolation and 2D convolution operations are used to process the high-precision feature map corresponding to each image patch to obtain a feature vector of length D for that image patch. Thus, a w×h image to be transmitted is represented as m 2 The eigenvectors are ×D (e.g., 256×D).

[0050] S1013, perform position encoding on the feature vector to embed the position information of the image block into the feature vector, and obtain the position feature vector of each image block.

[0051] In one example, since a self-attention mechanism is subsequently needed to perform a self-attention operation on the positional feature vectors to extract feature relationships in the image patch sequence, the positional information of the image patches can be embedded in their feature vectors to obtain the positional feature vector of each image patch. This explicit representation that incorporates positional information has a better advantage for structured data modeling of image patch features.

[0052] For example, the feature vector can be encoded by absolute position encoding or relative position encoding to obtain the position feature vector of the image patch.

[0053] Alternatively, when encoding feature vectors using absolute position encoding, a single value can be used to represent the absolute position of each image patch. Furthermore, the positional information of the image patches can be represented in a learnable manner, specifically using linear transformations of the sin and cos functions.

[0054] For example, the positional encoding of an image patch can satisfy the following formula:

[0055]

[0056] Where, p k,2i The position encoding of the image block in the k-th row and 2l-th column is given, where d is the dimension of the feature vector of the image block.

[0057] By embedding the above positional encoding into the feature vector D of the image block, the positional feature vector of the image can be obtained.

[0058] In one possible implementation, since the positional feature vector of the image block only performs a weak feature transformation on the features of the image block, its anti-interference ability is poor, its semantics are weak, and its data volume is large. In order to obtain more robust and semantically stronger features, and at the same time further encode and compress the features of the image block, the positional feature vector of the image block can be input into a multi-layer source encoder. The source encoder performs further feature processing and compression on the positional feature vector of the image block, thereby obtaining X initial connection probabilities of the image block and X probability maps of the image to be transmitted.

[0059] For example, a multi-layer source encoder group can consist of X source encoder modules connected in series. See also Figure 2 The internal workings of each source encoder module shown herein can perform multi-head self-attention operations, cross-self-attention operations, and the feature fusion process of multi-head attention and cross-self-attention as described in steps S1014-S1016 below. After the x-th source encoder performs steps S1014-S1016 below, the x-th initial connection probability of the image block can be obtained, which is the initial connection probability of the image block at layer l in the parse tree, x = l.

[0060] For example, x = 0, 1, 2...X, l = 1, 2...L, X≥L.

[0061] For example, a parse tree may include L levels, and the structural complexity of the parse tree is proportional to the complexity of the semantic information contained in the image to be transmitted.

[0062] For example, the xth source encoder can obtain the xth initial connection probability of the image block and the xth probability map of the image to be transmitted through steps S1014-S1016.

[0063] S1014, Perform multi-head self-attention operation on the image block according to the position feature vector of the image block to obtain the x-th multi-head self-attention matrix of the image to be transmitted.

[0064] In one example, multi-head self-attention operation can be implemented using multiple self-attention modules in the source encoder. The positional feature vectors of the image patch are used as the query vector Q, key vector K, and value vector V for the self-attention operation. First, the self-attention operation is performed in one branch (i.e., in one self-attention module) to obtain the output matrix Z of that branch. Then, the output matrices of multiple branches (e.g., 8) are concatenated together and input into the fully connected layer to obtain the multi-head self-attention matrix.

[0065] For example, self-attention can be performed in one branch using the following formula:

[0066]

[0067] Where, d k Let be the dimension of the location feature vector of the image patch. The formula calculates the dot product of each row vector of matrices Q and K. To prevent the dot product from becoming too large, it is divided by d. k The square root of Q. Multiplying Q by the transpose of K results in a matrix with n rows and n columns, where n is the number of image patches. This matrix represents the attention strength between all image patches. Thus, QK is obtained. T Next, Softmax is used to calculate the attention coefficients of each image patch to other image patches. In the formula, Softmax is applied to each row of the matrix, that is, the sum of each row becomes 1.

[0068] S1015, based on the positional feature vector of the image block, perform cross-self-attention operation on the image block in each branch of the multi-head self-attention operation to obtain the x-th cross-connection probability matrix of the image to be transmitted.

[0069] For example, since the semantic information of the image to be transmitted is often hierarchical and sparse, the attention obtained solely from the fused feature vectors in the multi-head self-attention matrix obtained by fully connected layers is often weak. To introduce sparse and hierarchical image prior information, a cross-attention module focusing on the correlation between image patches can be used to construct a semantic probability graph relationship model between different image patches. This model is then added to the previous attention model for attention enhancement. Specifically, cross-self-attention operations are performed on image patches in each branch of the multi-head self-attention operation, resulting in the x-th cross-connection probability for each image patch. The x-th cross-probability matrix is ​​constructed from the x-th cross-connection probability of each image patch.

[0070] In one example, the similarity between two image patches can be calculated using multi-scale dot product. This similarity can be used to assess the degree of similarity in the semantic information contained in the two image patches, i.e., to assess the probability that the two image patches belong to the same image patch cluster in the parse tree. However, determining this probability solely through similarity is not stable. Therefore, the correlation obtained by the Gaussian kernel function can be used instead of click similarity. The correlation obtained by the Gaussian kernel function does not require further normalization, which can improve the sparsity of the computation and overcome the problem that an image patch node is often connected to all image patches.

[0071] For example, the correlation obtained through the Gaussian kernel function can satisfy the following formula:

[0072]

[0073] Where, θ n The correlation obtained by applying a Gaussian kernel function to the nth image patch is often represented as a square matrix Q when performing multi-head self-attention operations. i,: Let K be the query vector in the i-th row of this matrix; similarly, the key-value vector K will also be represented as a square matrix, K j,: Let be the key value vector of the j-th column in this matrix. |||2 is the L2 norm, and d e Let be the dimension (length) of the query vector in the i-th row.

[0074] In one example, after obtaining the correlation of image patches, a cross-attention operation can be performed on the image patches to calculate the cross-connection probability of the image patches. The cross-connection probabilities of all image patches are then integrated into a matrix to obtain the cross-connection probability matrix, the size of which is the number of image patches.

[0075] For example, the cross-connection probability of an image patch satisfies the following formula:

[0076]

[0077]

[0078] Among them, C m,n The image patch in row m and column n For the cross-connection probability, see Figure 3 , These are the image patch and its two horizontally adjacent image patches. Correlation between them These are the image block and its two vertically adjacent image blocks. The correlation between them Represents image blocks It is an image block of the image S to be transmitted.

[0079] S1016, fuse the cross-connection probability and the multi-head self-attention matrix to obtain the x-th probability map of the image to be transmitted.

[0080] For example, the obtained cross-connection probability matrix can represent the sparse semantic probability connections between all image patches. Using this sparse probability representation matrix (i.e., the cross-connection probability matrix) and fusing it with the probability matrix (i.e., the multi-head self-attention matrix) obtained earlier through the multi-head self-attention module, a more robust self-attention representation (i.e., the initial connection probabilities and probability graphs of the image patches) can be obtained.

[0081] For example, the probability map of the image to be transmitted can satisfy the following formula:

[0082]

[0083] Where E′ is the probability map of the image to be transmitted, which is also a matrix. Each element in this matrix is ​​the initial connectivity probability of the image patch corresponding to that element, d is the dimension of the feature vector of the image patch, and C is the cross-connection probability matrix. The whole represents the multi-head self-attention matrix, and ⊙ represents the element-wise multiplication operation in the matrix.

[0084] S102, cluster the image blocks according to the initial connection probability to obtain the parse tree of the image to be transmitted.

[0085] For example, the implicit probabilistic model representation (probabilistic graph) corresponding to multiple levels of the parse tree obtained in an unsupervised manner can represent the sparse semantic probabilistic relationships of image patches in multiple levels. By learning and processing the probabilistic graph, an explicit parse tree representing the intrinsic semantics of the image can be obtained. The leaf node features of the tree structure of the parse tree can be directly used for image feature representation. Therefore, the structural complexity of the parse tree is proportional to the complexity of the semantic information contained in the image to be transmitted.

[0086] In some embodiments, the connection probability matrix of the (x+1)th layer of the parse tree can be determined using the (x+1)th probability map and the connection probability matrix of the (l)th layer of the parse tree, where x = l. The connection probability matrix of the (l+1)th layer of the parse tree is composed of the connection probabilities of the image patches in that layer. That is, the connection probability of the (l+1)th layer of the image patch is determined using the connection probability of the (l)th layer of the image patch and the (x+1)th initial connection probability of the image patch. Starting from the image patch clusters in layer 0, the image patch clusters in each layer can be clustered according to the connection probability of the image patch in layer l and a first segmentation threshold to obtain the parse tree.

[0087] For example, the parsing tree of the image to be transmitted can be obtained through steps S1021-S1022.

[0088] S1021, determine the connection probability of the (l+1)th layer of the image block based on the connection probability of the l-th layer of the image block.

[0089] For example, the connection probability of the image patch at layer 0 is the initial connection probability of the image patch.

[0090] For example, the connection probability of the (l+1)th layer of an image patch satisfies the following formula:

[0091]

[0092] in, The connection probability of the (l+1)th layer of the nth image patch; Let l be the connection probability of the nth image patch in the lth layer; Let x be the initial connection probability of the (x+1)th image patch, where x = l, i.e., the (l+1)th initial connection probability of the nth image patch.

[0093] S1022, starting from the image patch clusters of layer 0, the image patch chains of layer l are clustered according to the connection probability of the image patch in layer l and the first segmentation threshold. Subsequently, the image patch clusters of each layer are clustered in turn. Finally, the multi-layer clustered image patch clusters are organized into a parse tree according to the belonging relationship between each layer.

[0094] For example, the image block cluster in layer 0 consists of only one image block.

[0095] For example, an image block cluster includes at least one image block arranged sequentially according to the position information of the image blocks.

[0096] For example, the image to be transmitted is divided into 4×4=16 image blocks, and a matrix is ​​used. This indicates its location information.

[0097] The image blocks can be arranged in a row according to their positions. The arrangement order can be: A(1,1), A(1,2), A(1,3), A(1,4), A(2,1), A(2,2)... Each image block in the parse tree of level 0 is a separate image block cluster.

[0098] In one example, starting from the image patch chain at level 0, the connection probabilities of edge image patches in each layer's image patch cluster are retrieved sequentially from bottom to top, beginning with the connection probability of the image patch at level 0. If the connection probability of an edge image patch is greater than or equal to a first segmentation threshold, the image patch cluster containing the edge image patch and another image patch cluster adjacent to the edge image patch in that layer are clustered together to determine the image patch cluster of the next layer in the parse tree. This clustering process is repeated for each layer's image patch clusters until a certain layer of the parse tree contains only one image patch cluster, thus obtaining the parse tree.

[0099] For example, an edge image patch is an image patch located at both ends of an image patch cluster. Figure 4 401, 402, and 403 in the example. The edge image block of an image block cluster consisting of only one image block is that image block itself.

[0100] For example, the first segmentation threshold can be 0.85.

[0101] For example, the connection probabilities of edge image patches in the image patch clusters can be retrieved sequentially from bottom to top. If the connection probability is greater than or equal to a first segmentation threshold, it is determined that the image patch cluster to which the edge image patch belongs is in the same image patch cluster in the previous layer as its neighboring image patch cluster. If the connection probability is less than the threshold, the image patch cluster is separated from the neighboring image patch cluster. After the lower layer of image patch clusters is clustered, the image patch chains in the previous layer can be further segmented based on the connection probabilities between the image patch clusters in the l-th layer and the image patches in the previous layer. This process is repeated forward to generate a parse tree.

[0102] For example, see Figure 4 The first segmentation threshold is set to 0.85. Based on this threshold, image patch clusters at layer 0 of the parse tree are clustered to obtain layer 1. Clustering continues upwards until only one image patch cluster remains at layer 4. At this point, clustering stops, resulting in the parse tree 400 of the image to be transmitted. When clustering at this layer, the basic elements are the already clustered image patch clusters at the next layer. Through clustering, the number of image patch clusters decreases continuously, eventually converging into a single cluster at the top layer. Different potential functions can be used to measure the clustering effect at each layer. These potential functions are typically closely related to the image features. For a given potential function, the location where the potential function value is minimized (i.e., the connection probability of edge image patches is greater than or equal to the first segmentation threshold) can be found for clustering the image patch clusters. The same operation is recursively performed on each image patch cluster in this layer.

[0103] For example, the explicit parse tree of the obtained hierarchical structure represents semantic feature information of different granularities at different layers, and also represents the decomposition relationship between feature clusters of different granularities between layers. This dynamic parsing for each image reflects the unique semantic structure of each image to be transmitted, and reflects that the role and structure of different semantic nodes in the whole image are different.

[0104] S103, determine the encoding length of the image block based on the connection probability of the image block in the parse tree.

[0105] In some embodiments, the vector dimension of an image patch can be determined based on the obtained connection probability of the image patch, and then the encoding length of the image patch can be determined based on the connection probability and the maximum connection probability of the image patch.

[0106] In one possible implementation, the connection probability T of the Lth layer of the image patch can be initialized based on the obtained L-layer complete parse tree. l To obtain the initialized vector dimension That is, the vector dimension of the image patch. Then, based on the vector dimension and the connection probability of the image patch, the encoding length of the image patch is determined.

[0107] In one example, the connection probability matrix of the Lth layer (i.e., the top layer) can be extracted and normalized as follows: Simultaneously, the highest normalized connection probability φ was obtained. max The maximum normalized connection probability is obtained. Then, the connection probabilities of all image patches in any layer from layer 1 to layer (L-1) are normalized to obtain the normalized connection probability φ of the image patch. n The first coding length of an image block is determined by dividing its normalized connection probability by the maximum normalized connection probability. Then, the Kronecker product is performed on the first coding length and the vector dimension of the image block to obtain its dynamic coding length.

[0108] For example, the connection probability of the (L-1)th layer of the image patch can be used for normalization to obtain the normalized connection probability φ of the image patch. n .

[0109] For example, since the connection probability of image blocks reflects the degree of semantic connection between them, if the semantic information of the image to be transmitted is simpler, fewer clustering operations are needed to obtain the parse tree of the image to be transmitted, resulting in a smaller connection probability at the top level of the image blocks and a shorter encoding length. Conversely, if the semantic information of the image is more complex, more clustering operations are needed to obtain the parse tree of the image to be transmitted, resulting in a larger connection probability at the top level and a longer encoding length.

[0110] For example, if the image to be transmitted contains only one apple, the parse tree of the image may only require one or two clustering operations, resulting in a shorter encoding length. However, if the image to be transmitted contains multiple objects such as chairs, sofas, and blankets, then the image blocks belonging to chairs need to be clustered together, the image blocks belonging to sofas need to be clustered together, and the image blocks belonging to blankets need to be clustered together, and then these need to be clustered into a single image block cluster. This type of image requires more clustering operations, resulting in a more complex parse tree structure and a longer encoding length for each image block.

[0111] S104, Encode the image block according to the encoding length to obtain the encoded semantic feature vector group of the image to be transmitted.

[0112] For example, image blocks can be encoded according to the encoding length of each image block in the image to be transmitted to obtain the semantic feature vectors of the encoded image blocks. The encoded semantic feature vectors constitute the semantic feature vector group of the image to be transmitted.

[0113] According to the method provided by the present invention, image blocks are clustered by initial connection probabilities that can characterize the correlation between image blocks to obtain a parse tree of the image to be transmitted. Since the structural complexity of the parse tree is proportional to the complexity of the semantic information in the image, the image to be transmitted is encoded according to the encoding length determined by the connection probabilities of the image blocks in the parse tree. This allows images with more complex semantic information to have a longer encoding length and images with simpler semantic information to have a shorter encoding length. This enables adaptive encoding of the image based on its semantic features, thereby improving the utilization rate of communication resources.

[0114] Figure 5 This illustration shows a semantic communication method based on image content adaptive coding provided by an embodiment of the present invention. Method 500 is a possible specific implementation of method 100. As an example and not a limitation, method 500 may include steps S501-S506. Each step is described below.

[0115] S501, the source determines the initial connection probability of each image block in the image to be transmitted based on the semantic feature information of the image to be transmitted.

[0116] S502, the source end clusters the image blocks according to the initial connection probability to obtain the parse tree of the image to be transmitted.

[0117] S503, the source determines the encoding length of the image block based on the connection probability of the image block in the parse tree.

[0118] S504, the source end encodes the image block according to the coding length to obtain the encoded semantic feature vector group of the image to be transmitted.

[0119] For example, steps S501-S504 in method 500 are the same as steps S101-S104 in method 100. Please refer to the relevant description in method 100 for details, which will not be repeated here.

[0120] S505, the source sends the parse tree of the image to be transmitted and the encoded semantic feature vector group of the image to be transmitted to the destination.

[0121] Optionally, the source end can perform power normalization processing on the semantic feature vector group of the image to be transmitted before sending it to the destination end, which can improve communication efficiency and reliability.

[0122] Accordingly, the receiving end receives the parse tree of the image to be transmitted and the encoded semantic feature vector group of the image to be transmitted.

[0123] S506, the receiving end decodes the encoded semantic feature vector group of the image to be transmitted based on the parse tree of the image to be transmitted, and obtains the image to be transmitted.

[0124] In one example, a wireless channel is the transmission medium for mobile communication, through which all information is transmitted. The quality of the channel directly determines the accuracy and timeliness of information transmission, and thus the user's communication quality.

[0125] For example, the signal containing the parse tree of the transmitted image and the encoded semantic feature vector group of the image to be transmitted, after being transmitted through a wireless channel, can satisfy the following formula:

[0126] y′=ω(y;h,n)=y(t)⊙h(t)+n(t)

[0127] Where y(t) is the input parse tree of the transmitted image and the encoded semantic feature vector group of the image to be transmitted, y′ is the output signal containing the parse tree of the transmitted image and the encoded semantic feature vector group of the image to be transmitted after transmission through the wireless channel, h(t) is the channel characteristics of the wireless channel, and n(t)~N(0,N_0I) is the channel noise that follows a Gaussian distribution.

[0128] In one example, due to practical considerations in real-world communication systems, such as limited energy and interference, the transmitter's output may be required to meet certain power constraints, such as peak and / or average power constraints. The output signal z is then transmitted over the channel, where its quality degrades due to noise, attenuation, interference, or other channel impairments. This damaged output of the communication channel is fed to the destination. The destination receives an approximate reconstructed image that approximates the image to be transmitted. The destination can then denoise the image to be transmitted, decode and reconstruct the semantic feature vector set of the image to be transmitted using a parse tree, and obtain the final image to be transmitted.

[0129] Specifically, because the source end uses hierarchical parse trees to flexibly encode features, the feature dimensions of each image patch are not uniform, making direct decoding impossible. Therefore, the features of all image patches are dimensionally corrected, and for incomplete features, the average features are supplemented. Losses such as thumbnail data streams and other metadata in the transmission system are considered in the bandwidth consumption. Then, through the decoder in the transformer, and through the concatenation of multiple decoders, all features are decoded and reconstructed. The reconstructed image features are rearranged according to the order of feature arrangement at the source end, and then reconstructed into a single image.

[0130] According to the method provided by this invention, image blocks are clustered using initial connection probabilities that characterize the correlation between image blocks to obtain a parse tree of the image to be transmitted. Since the structural complexity of the parse tree is proportional to the complexity of the semantic information in the image, encoding the image to be transmitted based on the encoding length determined by the connection probabilities of the image blocks in the parse tree allows for longer encoding lengths for image blocks with more complex semantic information and shorter encoding lengths for image blocks with simpler semantic information. This achieves adaptive encoding of the image based on its semantic features, improves the utilization of communication resources, maximizes encoding efficiency, and achieves the best balance between image quality and transmission efficiency. Adaptive encoding of the image based on the connection probabilities between elements in the image makes the method provided by this invention more flexible. Furthermore, by encoding the image using an explicit parse tree that pays more attention to the connections between semantic image blocks, the leaf node features of the tree structure can be directly used for image feature representation. This cross-attention encoding strategy has advantages in convergence speed and exponential index.

[0131] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0132] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0133] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0134] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, any modifications made without departing from the inventive concept should be considered within the scope of protection of the present invention.

Claims

1. A method of semantic communication based on image content adaptive coding, characterized in that, The method is applied at the source end, and the method includes: Based on the semantic feature information of the image to be transmitted, the initial connection probability of each image block is determined, wherein the image block is a part of the image to be transmitted, and all the image blocks constitute the image to be transmitted; According to the initial connection probability, the image blocks are clustered to obtain a parse tree of the image to be transmitted, wherein the parse tree comprises L levels, and a structural complexity of the parse tree is proportional to a complexity of semantic information contained in the image to be transmitted. Based on the connection probability of the image block in the parse tree, the encoding length of the image block is determined, and the image block is encoded according to the encoding length to obtain the encoded semantic feature vector group of the image to be transmitted; The connection probability is determined based on the initial connection probability, and the encoding length of the image block is proportional to the complexity of the semantic information contained in the image block. The step of determining the initial connection probability of each image block based on the semantic feature information of the image to be transmitted includes: Extract semantic feature information from the image to be transmitted to obtain the position feature vector of each image block in the image to be transmitted; The initial connection probability of the image patch is determined based on the location feature vector.

2. The method of claim 1, wherein, The step of extracting semantic feature information from the image to be transmitted, to obtain the position feature vector of each image block in the image to be transmitted, includes: The image to be transmitted is segmented to obtain multiple image blocks; Extract the semantic feature information of each image block to obtain the feature vector of each image block; The feature vector is positionally encoded to embed the position information of the image block into the feature vector, thereby obtaining the position feature vector of each image block.

3. The method of claim 2, wherein, Determining the initial connectivity probability of the image patch based on the location feature vector includes: inputting the position feature vector into a source encoder group to obtain an initial connection probability of each of the image blocks X ​ The source encoder group comprises X a plurality of serially connected source encoders, a first x of the source encoders being configured to determine a first x initial connection probability of the image block according to the position feature vector, a second x of the initial connection probabilities of the image block being configured to determine the connection probability of the first l layer of the image block in the parsing tree, wherein ，x=l , x =0,1,2…… X , l =1,2…… L , X≥L .

4. The method of claim 3, wherein, x The source encoder is particularly configured to:​ performing multi-head self-attention operation on the image block according to the position feature vector of the image block, to obtain a first multi-head self-attention matrix of the image to be transmitted x ; According to the position feature vector of the image block, a cross self-attention operation is performed on the image block in each branch of the multi-head self-attention operation, to obtain a first cross connection probability matrix of the image to be transmitted x . Fusion x The cross-connection probability matrix and the first x The multi-head self-attention matrix is ​​used to obtain the first image to be transmitted. x The probability graph, where the th probability graph is... x The probability map is formed by the first probability map of each image block. x The probability matrix consisting of the initial connection probabilities.

5. The method of claim 1, wherein, The step of clustering the image blocks according to the initial connection probability to obtain the parse tree of the image to be transmitted includes: According to the image block first l layer connection probability and the image block first x +1 layer connection probability, determine the image block first l +1 layer connection probability, wherein ，x=l , the image block first 0 layer connection probability is the initial connection probability of the image block; Starting from the image block cluster at layer 0, according to the image block... l The connection probability of the layer and the first segmentation threshold, for the th l The image block clusters of the layers are clustered to obtain the parse tree of the image to be transmitted. The image block cluster of the 0th layer includes only one image block, and the image block cluster includes at least one image block arranged in order according to the position information of the image block.

6. The method of claim 5, wherein, According to the image block first l layer connection probability and the first segmentation threshold, the image block cluster of the first l layer is clustered to obtain the analysis tree of the image to be transmitted, comprising: Starting from the image patch clusters of layer 0, determine the... l Whether the connection probability of the edge image block of each image block cluster in the layer is greater than or equal to the first segmentation threshold, wherein the edge image block is the image block located at both ends of the image block cluster; If the connection probability of the edge image patch is greater than or equal to the first segmentation threshold, then the edge image patch belongs to the first segmentation threshold. l The image block clusters of the layer and the first l Clustering is performed on another image patch cluster adjacent to the edge image patch in the layer, and it is determined to be the first... l The image block clusters in layer +1; Repeat the clustering of the image patch clusters until the... l The parse tree of the image to be transmitted is obtained when there is only one image block cluster in the layer.

7. The method of claim 1, wherein, Determining the encoding length of an image block based on the connection probability of the image block in the parse tree includes: According to the image block number in the parse tree L The connection probability of the layer determines the vector dimension of the image patch; The encoding length of the image block is determined based on the vector dimension of the image block and the connection probability of the image block.

8. The method of claim 7, wherein, Determining the encoding length of an image block based on its vector dimension and connection probability includes: The quotient of the normalized connection probability of the image block and the maximum connection probability is determined as the first encoding length of the image block, wherein the normalized connection probability is obtained by normalizing the connection probability, and the maximum connection probability is the maximum value obtained by normalizing the connection probability of the image block L layer. The encoding length of the image block is obtained by performing a Kronecker product operation on the first encoding length and the vector dimension.

9. The method of claim 1, wherein, The method further includes: The receiver sends the parse tree of the image to be transmitted and the encoded semantic feature vector group of the image to be transmitted to the receiver, so that the receiver can decode the encoded semantic feature vector group of the image to be transmitted based on the parse tree of the image to be transmitted to obtain the image to be transmitted.

Images