Sending-side image semantic communication method based on dual-route attention

Abstract

The present invention discloses a transmitter-side image semantic communication method based on dual-route attention, comprising the steps of obtaining an image to be classified and inputting it into a transmitter; performing cropping or scaling of the image to be classified by a convolutional layer at the transmitter to obtain an image to be classified of uniform size; inputting the image to be classified of uniform size into a feature extraction network for feature extraction to obtain a feature map; performing semantic extraction on the feature map; compressing the extracted semantics; dividing the compressed semantics into two branches, R2R and T2T, and extracting features related to the semantics from each branch; fusing the features extracted from the R2R and T2T branches to generate a final feature representation containing global and local semantic information; classifying the image according to the final feature representation and outputting a classification result; and performing semantic communication transmission based on the classification result and the needs of the receiver. The present invention solves the problem of low information transmission accuracy in the prior art by using a transmitter-side semantic communication method based on dual-route attention.

Description

Technical Field

[0001] The present invention belongs to the field of Internet of Things communication technology, and in particular relates to a sending-end image semantic communication method based on dual-route attention. Background Art

[0002] With the advent of a new economy led by the information industry, communications and networks have been fully integrated into social production and daily life, becoming the core engine driving the rapid development of my country's information industry. However, traditional communications technologies focus primarily on accurate data transmission and precise restoration of signal waveforms, neglecting the semantic information carried by the data or carrier waveform. While this approach has made progress in ensuring data integrity, it has exposed some significant shortcomings when faced with increasingly complex scenario-based communication needs.

[0003] First, traditional communication methods cannot effectively handle information transmission at the semantic level, resulting in low information transmission accuracy. As the concept of the Internet of Everything continues to deepen, future communications will not only focus on transmission speed and signal accuracy, but also require the ability to effectively deliver meaningful content based on the needs of the recipient. Therefore, optimizing communication accuracy and improving the ability to transmit semantic information in resource-limited IoT environments has become a major challenge in current communication technologies. Summary of the Invention

[0004] The purpose of the present invention is to provide a sending-end image semantic communication method based on dual-route attention, which solves the problem of low information transmission accuracy in the prior art.

[0005] The technical solution adopted by the present invention is a sending-end image semantic communication method based on dual-route attention, which specifically includes the following steps:

[0006] S1: Get the image to be classified and input it into the sending end;

[0007] S2: The convolutional layer in the sending end crops or scales the image to be classified to obtain an image of uniform size.

[0008] S3: Input the image to be classified with uniform size into the feature extraction network for feature extraction to obtain a feature map;

[0009] S4: perform semantic extraction on feature maps;

[0010] S5: compress the extracted semantics;

[0011] S6: Divide the compressed semantics into two branches, R2R and T2T, and extract semantic-related features respectively;

[0012] S7: The semantically relevant features extracted from the R2R and T2T branches are fused to generate the final feature representation containing global and local semantic information;

[0013] S8: Classify the image according to the final feature representation and output the classification result; perform semantic communication transmission based on the classification result and the needs of the receiving end.

[0014] The present invention is also characterized in that:

[0015] S3 is represented as follows:

[0016] (1)

[0017] Where, For images, is the feature map, is the feature extraction network, The network parameters of the feature extraction network.

[0018] S4 specifically includes the following steps:

[0019] S4.1: Calculate the gradient of semantic concept activation relative to the feature map to obtain the importance weight of the semantic concept , the calculation formula is as follows:

[0020] (2)

[0021] Where, Represents the neuron activation value before the last fully connected layer of the feature extraction network passes through Softmax. Represents semantic concepts, w and h Indicates the The width and height of the feature map, Representation feature map No. i Rank j The activation value at the column;

[0022] S4.2: Computational Semantic Concepts The feature map weight vector is as follows:

[0023] (3)

[0024] Where, Representing semantic concepts The feature map weight vector, N represents the total number of feature maps, Representing semantic concepts The importance weight of each feature map to the semantic concept calculated by the feature extraction network;

[0025] S4.3: Based on the feature map weight vector of each semantic concept, a feature map weight matrix is ​​formed , which is expressed as follows:

[0026] (4)

[0027] Where, Semantic Concepts The importance weight vector of ; L represents the total number of semantic concepts;

[0028] S4.4: For each semantic concept, use the weight value to sort the feature map, and obtain the corresponding relationship between any semantic concept and the importance ranking of the feature map, that is, the semantic relationship, which is expressed as follows:

[0029] (5)

[0030] Where, represents the total feature map index matrix, Respectively represent different semantic concepts The feature map ranking results.

[0031] S5 specifically includes the following steps:

[0032] S5.1: Select the compression threshold based on the semantic relationship, as shown below:

[0033] (6)

[0034] Where, Represents semantic concepts The compression threshold, n Indicates that for each semantic concept in the compression process Only the most important n feature maps will be retained and transmitted, Indicates the feature map to semantic concept The importance weight of the feature map is as follows: Weight Less than the compression threshold , then the feature map will not be transmitted, that is ; If the feature map Weight Greater than or equal to the compression threshold , then the feature map will be transmitted, that is ;

[0035] The specific steps are as follows:

[0036] S5.1.1: Calculate the number of transmission characteristics n :

[0037] (7)

[0038] Where, The compression ratio corresponds to the degree of semantic compression at the sender, 0 means no compression, 1 means maximum compression, and the actual transmission feature map ratio is , represents the floor function;

[0039] S5.1.2: Calculate the compression threshold weight based on the number of transmitted features :

[0040] (8)

[0041] Where, , represents the results sorted in descending order according to weight value, Indicates the N The threshold weight of the item;

[0042] S5.2: Targeting Semantic Concepts , in the actual transmission characteristics, the former n The index of the most important feature map is represented as:

[0043] (9)

[0044] Where, It is for semantic concepts The feature map index set of T represents transmission, Indicates the reserved n The weight of the feature map, that is, the weight ranked first among all weights n The weight of the bit;

[0045] S5.3: Calculate the feature map index vector actually transmitted through the channel , expressed as:

[0046] (10).

[0047] S6 specifically includes the following steps:

[0048] S6.1: The R2R branch applies an attention mechanism at the region level to identify and process key regions in the image and extract features related to semantic concepts. The specific steps are as follows:

[0049] S6.1.1: Using the feature map index vector, select a corresponding key feature region from the image, where the key feature region includes a plurality of pixel blocks;

[0050] S6.1.2: Use a linear mapping to embed the location of each key feature region into a low-dimensional space, generating the query matrix Q, key matrix K, and value matrix V, which are expressed as follows:

[0051] (11)

[0052] Where, , are the mapping weights of Q, K, and V respectively, Represents the square root of the number of regions; H, W, and C represent the height, width, and number of channels of the image, respectively; represents the index matrix of routing areas; r represents the area;

[0053] S6.1.3: Calculate the element-by-element average of the query matrix Q and key matrix K of each region in each channel dimension to obtain the region-level query matrix and bond matrix , ;

[0054] S6.1.4: By and Perform matrix multiplication to generate the adjacency matrix between regions , represents the semantic correlation between regions; Represents the bond matrix The transposed matrix of

[0055] S6.1.5: Using the adjacency matrix Generate routing index matrix , which is the feature extracted by the R2R branch, is expressed as follows:

[0056] (12)

[0057] Where, Indicates row-first top-k selection operation;

[0058] S6.2: The T2T branch further applies a fine-grained attention mechanism to the region selected in S6.1, calculates the similarity of each token with other tokens, performs weighted average, and generates a new token sequence containing contextual information. The specific steps are as follows:

[0059] S6.2.1: For each region r i The query token in , follow all The key-value pair of the index position; Indicates area r i The most relevant m The index of the region;

[0060] S6.2.2: Collect the key vector and value vector, expressed as follows:

[0061] (13)

[0062] in, Represents the collected key vector and value vector; Indicates that from the key matrix K and the value matrix V according to the index Extract relevant key-value pairs;

[0063] S6.2.3: Apply attention mechanism on the collected key-value pairs and combine it with local context enhancement to obtain context-enhanced token sequence , which is the feature extracted by the T2T branch, is expressed as follows:

[0064] (14)

[0065] in, It represents the enhancement of the value matrix V by using depthwise convolution.

[0066] S7 specifically includes the following steps:

[0067] S7.1: Extract features from R2R branches and features extracted by the T2T branch Perform splicing to obtain the spliced ​​feature vector :

[0068] (15);

[0069] S7.2: Concatenated feature vectors Use weighted average to fuse and generate the fused feature vector .

[0070] The beneficial effects of the present invention are:

[0071] This method, based on a dual-route attention-based sender-side image semantic communication method, improves the accuracy of information transmission by establishing associations between semantic concepts and image features. Through semantic compression at the sender, this method ensures that key semantic information is preserved during transmission, reducing data loss and misunderstandings. Ultimately, it improves the receiver's understanding of semantic information, thereby enhancing the reliability and accuracy of communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 1 is a diagram of the semantic communication transmission process of the sending-end image semantic communication method based on dual-route attention of the present invention;

[0073] Figure 2It is a flow chart of the sending-end image semantic communication method based on dual-route attention of the present invention;

[0074] Figure 3 1 is a dual-route structure diagram of the sending-end image semantic communication method based on dual-route attention of the present invention;

[0075] Figure 4 3 is a data graph of training accuracy of Example 3 of the sending-end image semantic communication method based on dual-route attention of the present invention and the baseline method;

[0076] Figure 5 This is a data graph of verification accuracy of Example 3 of the sending-end image semantic communication method based on dual-route attention of the present invention and the baseline method. DETAILED DESCRIPTION

[0077] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0078] like Figure 1 As shown in the figure, the semantic communication transmission process can be summarized as follows: first, the original data such as text, voice, image, and video generated by the source is preliminarily processed and then semantic extraction is performed. Semantic extraction is an in-depth analysis of the preliminarily processed data to extract the semantic information therein. Taking image data as an example, semantic extraction includes feature extraction, object recognition, etc. The semantic encoding module encodes the extracted semantic information to facilitate transmission during the communication process. Semantic encoding not only compresses the amount of transmitted data but also retains key information. The channel coding module further encodes the semantically encoded data to enhance its anti-interference ability during transmission. Channel coding technologies such as error correction codes can effectively improve the reliability of data transmission. The present invention uses a single-layer neural network to model the channel. The channel-coded data is transmitted to the receiving end through the transmission channel, where it is decoded and restored, and the processed result is output. The output data can be used in various applications such as speech recognition, image classification, and text comprehension.

[0079] The present invention is based on the sending end image semantic communication method of dual-route attention, such as Figure 2 As shown, the specific steps include:

[0080] S1: Get the image to be classified and input it into the sending end;

[0081] The experiments used in this paper use the STL-10 dataset, which consists of images representing 10 different semantic concepts, each of which is associated with a specific image category.

[0082] S2: The convolutional layer at the sending end preprocesses the image to be classified, including cropping or scaling, and adjusting the image to a fixed size, generally 224×224 pixels.

[0083] S3: Input the image to be classified with uniform size into the feature extraction network for feature extraction to obtain a feature map;

[0084] Use feature extraction network to resize the image to a fixed size I For feature extraction, the process can be expressed as:

[0085] (1)

[0086] Where, For images, is the feature map, is the feature extraction network, The network parameters of the feature extraction network.

[0087] like Figure 3 Figure 2 shows a semantic communication convolutional neural network for image classification tasks, which can achieve efficient semantic transmission and maximize the accuracy of classification tasks.

[0088] The sender first resizes the image to a fixed size and uses a feature extraction network, such as ResNet18, to extract features from the input image. The extracted feature maps are then semantically compressed based on the extracted semantic relationships, further reducing the amount of data actually transmitted. The feature maps are then passed through a classifier and a Softmax layer, and the classification results are output.

[0089] S4: Extract semantics from the feature map, which includes the following steps:

[0090] S4.1: Calculate the gradient of semantic concept activation relative to the feature map to obtain the importance weight of the semantic concept , the calculation formula is as follows:

[0091] (2)

[0092] Where, Represents the neuron activation value before the last fully connected layer of the feature extraction network passes through Softmax. Represents semantic concepts, w and h Indicates the The width and height of the feature map, Representation feature map No. i Rank j The activation value at the column;

[0093] The activation value is the output value of a neuron after the neural network calculates the value. For example, in a classification problem, the number of neurons in the last layer is N, and each neuron corresponds to a category, or semantic concept. When an image to be classified is input and propagated through the neural network's forward computation, each neuron in the last layer outputs an activation value.

[0094] S4.2: Computational Semantic Concepts The feature map weight vector is as follows:

[0095] (3)

[0096] Where, Representing semantic concepts The feature map weight vector, N represents the total number of feature maps, Representing semantic concepts The importance weight of each feature map to the semantic concept calculated by the feature extraction network;

[0097] S4.3: Based on the feature map weight vector of each semantic concept, a feature map weight matrix is ​​formed , which is expressed as follows:

[0098] (4)

[0099] Where, Semantic Concepts The importance weight vector of ; if there are multiple semantic concepts involved in the image, then L is the total number of semantic concepts.

[0100] S4.4: For each semantic concept, use the weight value to sort the feature map, and obtain the corresponding relationship between any semantic concept and the importance ranking of the feature map, that is, the semantic relationship, which is expressed as follows:

[0101] (5)

[0102] Where, Represents the total feature map index matrix, which contains the importance ranking of feature maps for different semantic concepts. Respectively represent different semantic concepts The feature map ranking results.

[0103] S5: compress the extracted semantics;

[0104] Based on the actual communication and computing resources, combined with the feature map weights and semantic relationships, the extracted semantics are compressed to remove redundant information. Specifically, the following steps are performed:

[0105] S5.1: Select the compression threshold based on the semantic relationship, as shown below:

[0106] (6)

[0107] Where, Represents semantic concepts The compression threshold, n Indicates that for each semantic concept in the compression process Only the most important n feature maps will be retained and transmitted, Indicates the feature map to semantic concept The importance weight of the feature map is as follows: Weight Less than the compression threshold , then the feature map will not be transmitted, that is ; If the feature map Weight Greater than or equal to the compression threshold , then the feature map will be transmitted, that is ;

[0108] The specific steps are as follows:

[0109] S5.1.1: Calculate the number of transmission characteristics n :

[0110] (7)

[0111] Where, The compression ratio corresponds to the degree of semantic compression at the sender, 0 means no compression, 1 means maximum compression, and the actual transmission feature map ratio is , represents the floor function;

[0112] S5.1.2: Calculate the compression threshold weight based on the number of transmitted features :

[0113] (8)

[0114] Where, , represents the results sorted in descending order according to weight value, Indicates the N The threshold weight of the item;

[0115] S5.2: Targeting Semantic Concepts , in the actual transmission characteristics, the former n The index of the most important feature map is expressed as:

[0116] (9)

[0117] Where, It is for semantic concepts The feature map index set represents the feature map index that is finally retained for transmission after compression; T represents transmission, Indicates the reserved n The weight of the feature map, that is, the weight ranked first among all weights n The weight of the bit;

[0118] S5.3: Calculate the feature map index vector actually transmitted through the channel , expressed as:

[0119] (10).

[0120] S6: Divide the compressed semantics into two branches, R2R and T2T, and extract semantic-related features respectively;

[0121] The compressed feature map is processed with dual-route attention, which is divided into two branches: Region-to-Region (R2R) and Token-to-Token (T2T). Features are extracted from the two branches respectively. The specific process is as follows:

[0122] S6.1: The R2R branch applies an attention mechanism at the region level to identify and process key regions in the image and extract features related to semantic concepts. The specific steps are as follows:

[0123] S6.1.1: Using the feature map index vector, select a corresponding key feature region from the image, where the key feature region includes a plurality of pixel blocks;

[0124] S6.1.2: Use a linear mapping to embed the location of each key feature region into a low-dimensional space, generating the query matrix Q, key matrix K, and value matrix V, which are expressed as follows:

[0125] (11)

[0126] Where, , are the mapping weights of Q, K, and V respectively, Represents the square root of the number of regions; H, W, and C represent the height, width, and number of channels of the image, respectively; represents the index matrix of routing areas; r represents the area;

[0127] S6.1.3: Calculate the element-by-element average of the query matrix Q and key matrix K of each region in each channel dimension to obtain the region-level query matrix and bond matrix , ;

[0128] S6.1.4: By and Perform matrix multiplication to generate the adjacency matrix between regions , represents the semantic correlation between regions; Represents the bond matrix The transposed matrix of

[0129] S6.1.5: Using the adjacency matrix Generate routing index matrix , which is the feature extracted by the R2R branch, is expressed as follows:

[0130] (12)

[0131] Where, Indicates row-first top-k selection operation;

[0132] S6.2: The T2T branch further applies a fine-grained attention mechanism to the region selected in S6.1, calculates the similarity of each token with other tokens, performs weighted average, and generates a new token sequence containing contextual information. The specific steps are as follows:

[0133] S6.2.1: For each region r i The query token in , follow all The key-value pair of the index position; Indicates area r i The most relevant m The index of the region;

[0134] S6.2.2: Collect the key vector and value vector, expressed as follows:

[0135] (13)

[0136] in, Represents the collected key vector and value vector; Indicates that from the key matrix K and the value matrix V according to the index Extract relevant key-value pairs;

[0137] S6.2.3: Apply attention mechanism on the collected key-value pairs and combine it with local context enhancement to obtain context-enhanced token sequence , which is the feature extracted by the T2T branch, is expressed as follows:

[0138] (14)

[0139] in, It represents the enhancement of the value matrix V by using depthwise convolution.

[0140] S7: Fuse the features extracted from the R2R and T2T branches to generate the final feature representation containing global and local semantic information. The specific process is as follows:

[0141] S7.1: Extract features from R2R branches and features extracted by the T2T branch Perform splicing to obtain the spliced ​​feature vector :

[0142] (15);

[0143] S7.2: Concatenated feature vectors Use weighted average to fuse and generate the fused feature vector .

[0144] S8: Classify the image according to the final feature representation and output the classification result; perform semantic communication transmission based on the classification result and the needs of the receiving end.

[0145] Example 1

[0146] This embodiment provides a sending-end image semantic communication method based on dual-route attention, such as Figure 2 As shown,

[0147] S1: Get the image to be classified and input it into the sending end;

[0148] S2: The convolutional layer in the sending end crops or scales the image to be classified to obtain an image of uniform size.

[0149] S3: Input the image to be classified with uniform size into the feature extraction network for feature extraction to obtain a feature map;

[0150] S4: Extract semantics from feature maps;

[0151] S5: compress the extracted semantics;

[0152] S6: Divide the compressed semantics into two branches, R2R and T2T, and extract semantic-related features respectively;

[0153] S7: The semantically relevant features extracted from the R2R and T2T branches are fused to generate the final feature representation containing global and local semantic information;

[0154] S8: Classify the image according to the final feature representation and output the classification result; perform semantic communication transmission based on the classification result and the needs of the receiving end.

[0155] Example 2

[0156] This embodiment provides a sending-end image semantic communication method based on dual-route attention, such as Figure 2 As shown,

[0157] S1: Get the image to be classified and input it into the sending end;

[0158] S2: The convolutional layer at the sending end crops or scales the image to be classified to obtain an image of uniform size.

[0159] S3: Input the image to be classified with uniform size into the feature extraction network for feature extraction to obtain a feature map, specifically:

[0160] (1)

[0161] Where, For images, is the feature map, is the feature extraction network, The network parameters of the feature extraction network.

[0162] S4: Extract semantics from feature maps;

[0163] S4.1: Calculate the gradient of semantic concept activation relative to the feature map to obtain the importance weight of the semantic concept , the calculation formula is as follows:

[0164] (2)

[0165] Where, Represents the neuron activation value before the last fully connected layer of the feature extraction network passes through Softmax. Represents semantic concepts, w and h Indicates the The width and height of the feature map, Representation feature map No. i Rank j The activation value at the column;

[0166] S4.2: Computational Semantic Concepts The feature map weight vector is as follows:

[0167] (3)

[0168] Where, Representing semantic concepts The feature map weight vector, N represents the total number of feature maps, Representing semantic concepts The importance weight of each feature map to the semantic concept calculated by the feature extraction network;

[0169] S4.3: Based on the feature map weight vector of each semantic concept, a feature map weight matrix is ​​formed , which is expressed as follows:

[0170] (4)

[0171] Where, Semantic Concepts The importance weight vector of ; L represents the total number of semantic concepts;

[0172] S4.4: For each semantic concept, use the weight value to sort the feature map, and obtain the corresponding relationship between any semantic concept and the importance ranking of the feature map, that is, the semantic relationship, which is expressed as follows:

[0173] (5)

[0174] Where, represents the total feature map index matrix, Respectively represent different semantic concepts The feature map ranking results.

[0175] S5: compress the extracted semantics;

[0176] S5.1: Select the compression threshold based on the semantic relationship, as shown below:

[0177] (6)

[0178] Where, Represents semantic concepts The compression threshold, n Indicates that for each semantic concept in the compression process Only the most important n feature maps will be retained and transmitted, Indicates the feature map to semantic concept The importance weight of the feature map is as follows: Weight Less than the compression threshold , then the feature map will not be transmitted, that is ; If the feature map Weight Greater than or equal to the compression threshold , then the feature map will be transmitted, that is ;

[0179] The specific steps are as follows:

[0180] S5.1.1: Calculate the number of transmission characteristics n :

[0181] (7)

[0182] Where, The compression ratio corresponds to the degree of semantic compression at the sender, 0 means no compression, 1 means maximum compression, and the actual transmission feature map ratio is , represents the floor function;

[0183] S5.1.2: Calculate the compression threshold weight based on the number of transmitted features :

[0184] (8)

[0185] Where, , represents the results sorted in descending order according to weight value, Indicates the N The threshold weight of the item;

[0186] S5.2: Targeting Semantic Concepts , in the actual transmission characteristics, the former n The index of the most important feature map is expressed as:

[0187] (9)

[0188] Where, It is for semantic concepts The feature map index set of T represents transmission, Indicates the reserved n The weight of the feature map, that is, the weight ranked first among all weights n The weight of the bit;

[0189] S5.3: Calculate the feature map index vector actually transmitted through the channel , expressed as:

[0190] (10).

[0191] S6: Divide the compressed semantics into two branches, R2R and T2T, and extract semantic-related features respectively;

[0192] S6.1: The R2R branch applies an attention mechanism at the region level to identify and process key regions in the image and extract features related to semantic concepts. The specific steps are as follows:

[0193] S6.1.1: Using the feature map index vector, select a corresponding key feature region from the image, where the key feature region includes a plurality of pixel blocks;

[0194] S6.1.2: Use a linear mapping to embed the location of each key feature region into a low-dimensional space, generating the query matrix Q, key matrix K, and value matrix V, which are expressed as follows:

[0195] (11)

[0196] Where, , are the mapping weights of Q, K, and V respectively, Represents the square root of the number of regions; H, W, and C represent the height, width, and number of channels of the image, respectively; represents the index matrix of routing areas; r represents the area;

[0197] S6.1.3: Calculate the element-by-element average of the query matrix Q and key matrix K of each region in each channel dimension to obtain the region-level query matrix and bond matrix , ;

[0198] S6.1.4: By and Perform matrix multiplication to generate the adjacency matrix between regions , represents the semantic correlation between regions; Represents the bond matrix The transposed matrix of

[0199] S6.1.5: Using the adjacency matrix Generate routing index matrix , which is the feature extracted by the R2R branch, is expressed as follows:

[0200] (12)

[0201] Where, Indicates row-first top-k selection operation;

[0202] S6.2: The T2T branch further applies a fine-grained attention mechanism to the region selected in S6.1, calculates the similarity of each token with other tokens, performs weighted average, and generates a new token sequence containing contextual information. The specific steps are as follows:

[0203] S6.2.1: For each region r i The query token in , follow all The key-value pair of the index position; Indicates area r i The most relevant m The index of the region;

[0204] S6.2.2: Collect the key vector and value vector, expressed as follows:

[0205] (13)

[0206] in, Represents the collected key vector and value vector; Indicates that from the key matrix K and the value matrix V according to the index Extract relevant key-value pairs;

[0207] S6.2.3: Apply attention mechanism on the collected key-value pairs and combine it with local context enhancement to obtain context-enhanced token sequence , which is the feature extracted by the T2T branch, is expressed as follows:

[0208] (14)

[0209] in, It represents the enhancement of the value matrix V by using depthwise convolution.

[0210] S7: The semantically relevant features extracted from the R2R and T2T branches are fused to generate the final feature representation containing global and local semantic information;

[0211] S7.1: Extract features from R2R branches and features extracted by the T2T branch Perform splicing to obtain the spliced ​​feature vector :

[0212] (15);

[0213] S7.2: Concatenated feature vectors Use weighted average to fuse and generate the fused feature vector .

[0214] S8: Classify the image according to the final feature representation and output the classification result; perform semantic communication transmission based on the classification result and the needs of the receiving end.

[0215] Example 3

[0216] This embodiment provides a sending-end image semantic communication method based on dual-route attention, such as Figure 2 As shown,

[0217] S1: Use the STL-10 dataset and input the sender;

[0218] S2: The sending convolutional layer crops or scales the image to be classified and adjusts the image to a fixed size of 224×224 pixels;

[0219] S3: Input the fixed-size image into the ResNet18 feature extraction network for feature extraction to obtain the feature map, which is represented as:

[0220] (1)

[0221] Where, For images, is the feature map, is the feature extraction network, The network parameters of the feature extraction network.

[0222] S4: Extract semantics from feature maps;

[0223] S4.1: Calculate the gradient of semantic concept activation relative to the feature map to obtain the importance weight of the semantic concept , the calculation formula is as follows:

[0224] (2)

[0225] Where, Represents the neuron activation value before the last fully connected layer of the feature extraction network passes through Softmax. Represents semantic concepts, w and h Indicates the The width and height of the feature map, Representation feature map No. i Rank j The activation value at the column;

[0226] S4.2: Computational Semantic Concepts The feature map weight vector is as follows:

[0227] (3)

[0228] Where, Representing semantic concepts The feature map weight vector, N represents the total number of feature maps, Representing semantic concepts The importance weight of each feature map to the semantic concept calculated by the feature extraction network;

[0229] S4.3: Based on the feature map weight vector of each semantic concept, a feature map weight matrix is ​​formed , which is expressed as follows:

[0230] (4)

[0231] Where, Semantic Concepts The importance weight vector of ; L represents the total number of semantic concepts;

[0232] S4.4: For each semantic concept, use the weight value to sort the feature map, and obtain the corresponding relationship between any semantic concept and the importance ranking of the feature map, that is, the semantic relationship, which is expressed as follows:

[0233] (5)

[0234] Where, represents the total feature map index matrix, Respectively represent different semantic concepts The feature map ranking results.

[0235] S5: Set the compression rate to 0.2 to compress the extracted semantics;

[0236] S5.1: Select the compression threshold based on the semantic relationship, as shown below:

[0237] (6)

[0238] Where, Represents semantic concepts The compression threshold, n Indicates that for each semantic concept in the compression process Only the most important n feature maps will be retained and transmitted, Indicates the feature map to semantic concept The importance weight of the feature map is as follows: Weight Less than the compression threshold , then the feature map will not be transmitted, that is ; If the feature map Weight Greater than or equal to the compression threshold , then the feature map will be transmitted, that is ;

[0239] The specific steps are as follows:

[0240] S5.1.1: Calculate the number of transmission characteristicsn :

[0241] (7)

[0242] Where, The compression ratio corresponds to the degree of semantic compression at the sender, 0 means no compression, 1 means maximum compression, and the actual transmission feature map ratio is , represents the floor function;

[0243] S5.1.2: Calculate the compression threshold weight based on the number of transmitted features :

[0244] (8)

[0245] Where, , represents the results sorted in descending order according to weight value, Indicates the N The threshold weight of the item;

[0246] S5.2: Targeting Semantic Concepts , in the actual transmission characteristics, the former n The index of the most important feature map is expressed as:

[0247] (9)

[0248] Where, It is for semantic concepts The feature map index set of T represents transmission, Indicates the reserved n The weight of the feature map, that is, the weight ranked first among all weights n The weight of the bit;

[0249] S5.3: Calculate the feature map index vector actually transmitted through the channel , expressed as:

[0250] (10).

[0251] S6: Divide the compressed semantics into two branches, R2R and T2T, and extract semantic-related features respectively;

[0252] S6.1: The R2R branch applies an attention mechanism at the region level to identify and process key regions in the image and extract features related to semantic concepts. The specific steps are as follows:

[0253] S6.1.1: Using the feature map index vector, select a corresponding key feature region from the image, where the key feature region includes a plurality of pixel blocks;

[0254] S6.1.2: Use a linear mapping to embed the location of each key feature region into a low-dimensional space, generating the query matrix Q, key matrix K, and value matrix V, which are expressed as follows:

[0255] (11)

[0256] Where, , are the mapping weights of Q, K, and V respectively, Represents the square root of the number of regions; H, W, and C represent the height, width, and number of channels of the image, respectively; represents the index matrix of routing areas; r represents the area;

[0257] S6.1.3: Calculate the element-by-element average of the query matrix Q and key matrix K of each region in each channel dimension to obtain the region-level query matrix and bond matrix , ;

[0258] S6.1.4: By and Perform matrix multiplication to generate the adjacency matrix between regions , represents the semantic correlation between regions; Represents the bond matrix The transposed matrix of

[0259] S6.1.5: Using the adjacency matrix Generate routing index matrix , which is the feature extracted by the R2R branch, is expressed as follows:

[0260] (12)

[0261] Where, Indicates row-first top-k selection operation;

[0262] S6.2: The T2T branch further applies a fine-grained attention mechanism to the region selected in S6.1, calculates the similarity of each token with other tokens, performs weighted average, and generates a new token sequence containing contextual information. The specific steps are as follows:

[0263] S6.2.1: For each region r i The query token in , follow all The key-value pair of the index position; Indicates area r i The most relevant mThe index of the region;

[0264] S6.2.2: Collect the key vector and value vector, expressed as follows:

[0265] (13)

[0266] in, Represents the collected key vector and value vector; Indicates that from the key matrix K and the value matrix V according to the index Extract relevant key-value pairs;

[0267] S6.2.3: Apply attention mechanism on the collected key-value pairs and combine it with local context enhancement to obtain context-enhanced token sequence , which is the feature extracted by the T2T branch, is expressed as follows:

[0268] (14)

[0269] in, It represents the enhancement of the value matrix V by using depthwise convolution.

[0270] S7: The semantically relevant features extracted from the R2R and T2T branches are fused to generate the final feature representation containing global and local semantic information;

[0271] S7.1: Extract features from R2R branches and features extracted by the T2T branch Perform splicing to obtain the spliced ​​feature vector :

[0272] (15);

[0273] S7.2: Concatenated feature vectors Use weighted average to fuse and generate the fused feature vector .

[0274] S8: Classify the image according to the final feature representation and output the classification result; perform semantic communication transmission based on the classification result and the needs of the receiving end.

[0275] Simulation experiment:

[0276] The experimental platform consists of an i7-12700F processor and an NVIDIA GeForce RTX 3060 (12G) GPU, using CUDA 12.2 for computing. In addition, the experiment uses Python 3.8.17 as the programming language and a deep learning framework built on PyTorch 1.9.0.

[0277] The method in the paper "Semantic Communication Methods for Intelligent Tasks in the Artificial Intelligence Internet of Things" was used as the baseline method. The STL-10 dataset, consisting of images representing 10 different semantic concepts, each associated with a specific image category, was used. Within each category, there were 1,300 images at a resolution of 96 × 96 dpi, divided into 500 for training and 800 for testing. The experimental parameters were set as follows: 100 iterations, a batch size of 32, the Adam optimizer, a learning rate of 0.0001, 10 categories, and a compression ratio of 0.2. The experimental environment and parameter settings were the same as in Example 3.

[0278] The training accuracy and verification accuracy of the method proposed in this invention are compared with those of the limit method. Figure 4 and Figure 5 As shown in the figure, the image classification accuracy of the method proposed in the present invention is improved by about 5%, and it has higher training accuracy and verification accuracy.

Claims

1. A sender-side image semantic communication method based on dual-route attention, characterized in that: The specific steps include: S1: Get the image to be classified and input it into the sending end; S2: The convolutional layer in the transmitting end crops or scales the image to be classified to obtain an image to be classified with a uniform size; S3: Inputting the image to be classified with a uniform size into a feature extraction network to extract features and obtain a feature map; S4: performing semantic extraction on the feature map; S4.1: Calculate the gradient of semantic concept activation relative to the feature map to obtain the importance weight of the semantic concept; S4.2: Calculate the feature map weight vector of the semantic concept; S4.3: Based on the obtained feature map weight vector of each semantic concept, a feature map weight matrix is ​​formed; S4.4: For each semantic concept, use the weight value to sort the feature map, and obtain the corresponding relationship between any semantic concept and the importance ranking of the feature map, that is, the semantic relationship; S5: compress the extracted semantics; S5.1: Select compression threshold based on semantic relationship; S5.2: For semantic concepts, in the features actually transmitted, the calculation n The index of the most important feature maps; S5.3: Calculate the feature map index vector actually transmitted through the channel; S6: Divide the compressed semantics into two branches, R2R and T2T, and extract semantic-related features respectively; S7: The semantically relevant features extracted from the R2R and T2T branches are fused to generate the final feature representation containing global and local semantic information; S8: Classify the image according to the final feature representation and output the classification result; perform semantic communication transmission based on the classification result and the needs of the receiving end.

2. The sending-end image semantic communication method based on dual-route attention according to claim 1 is characterized in that S3 is represented as follows: （1） Where, For images, is the feature map, is the feature extraction network, The network parameters of the feature extraction network.

3. The sending-end image semantic communication method based on dual-route attention according to claim 2 is characterized in that: S4 specifically includes the following steps: S4.1: Calculate the gradient of semantic concept activation relative to the feature map to obtain the importance weight of the semantic concept , the calculation formula is as follows: （2） Where, Represents the neuron activation value before the last fully connected layer of the feature extraction network passes through Softmax. Represents semantic concepts, w and h Indicates the The width and height of the feature map, Representation feature map No. i Rank j The activation value at the column; S4.2: Computational Semantic Concepts The feature map weight vector is as follows: （3） Where, Representing semantic concepts The feature map weight vector, N represents the total number of feature maps, Representing semantic concepts The importance weight of each feature map to the semantic concept calculated by the feature extraction network; S4.3: Based on the feature map weight vector of each semantic concept, a feature map weight matrix is ​​formed , which is expressed as follows: （4） Where, Representing semantic concepts The importance weight vector of ; L represents the total number of semantic concepts; S4.4: For each semantic concept, use the weight value to sort the feature map, and obtain the corresponding relationship between any semantic concept and the importance ranking of the feature map, that is, the semantic relationship, which is expressed as follows: （5） Where, represents the total feature map index matrix, Respectively represent different semantic concepts The feature map ranking results.

4. The sending-end image semantic communication method based on dual-route attention according to claim 3 is characterized in that S5 specifically includes the following steps: S5.1: Select the compression threshold based on the semantic relationship, as shown below: （6） Where, Represents semantic concepts The compression threshold, n Indicates that for each semantic concept in the compression process Only the most important n feature maps will be retained and transmitted, Indicates the feature map to semantic concept The importance weight of the feature map is as follows: Weight Less than the compression threshold , then the feature map will not be transmitted, that is ; If the feature map Weight Greater than or equal to the compression threshold , then the feature map will be transmitted, that is ; The specific steps are as follows: S5.1.1: Calculate the number of transmission characteristics n : （7） Where, The compression ratio corresponds to the degree of semantic compression at the sender, 0 means no compression, 1 means maximum compression, and the actual transmission feature map ratio is , represents the floor function; S5.1.2: Calculate the compression threshold weight based on the number of transmitted features : （8） Where, , represents the results sorted in descending order according to weight value, Indicates the N The threshold weight of the item; S5.2: Targeting Semantic Concepts , in the actual transmission characteristics, the former n The index of the most important feature map is represented as: （9） Where, It is for semantic concepts The feature map index set of T represents transmission, Indicates the reserved n The weight of the feature map, that is, the weight ranked first among all weights n The weight of the bit; S5.3: Calculate the feature map index vector actually transmitted through the channel , expressed as: （10）。 5. The sending-end image semantic communication method based on dual-route attention according to claim 1 is characterized in that: S6 specifically includes the following steps: S6.1: The R2R branch applies an attention mechanism at the region level to identify and process key regions in the image and extract features related to semantic concepts. The specific steps are as follows: S6.1.1: Using the feature map index vector, select a corresponding key feature region from the image, where the key feature region includes a plurality of pixel blocks; S6.1.2: Use a linear mapping to embed the location of each key feature region into a low-dimensional space, generating the query matrix Q, key matrix K, and value matrix V, which are expressed as follows: （11） Where, , are the mapping weights of Q, K, and V respectively, Represents the square root of the number of regions; H, W, and C represent the height, width, and number of channels of the image, respectively; represents the index matrix of routing areas; r represents the area; S6.1.3: Calculate the element-by-element average of the query matrix Q and key matrix K of each region in each channel dimension to obtain the region-level query matrix and bond matrix , ; S6.1.4: By and Perform matrix multiplication to generate the adjacency matrix between regions , represents the semantic correlation between regions; Represents the bond matrix The transposed matrix of S6.1.5: Using the adjacency matrix Generate routing index matrix , which is the feature extracted by the R2R branch, is expressed as follows: （12） Where, Indicates row-first top-k selection operation; S6.2: The T2T branch further applies a fine-grained attention mechanism to the region selected in S6.1, calculates the similarity of each token with other tokens, performs weighted average, and generates a new token sequence containing contextual information. The specific steps are as follows: S6.2.1: For each region r i The query token in , follow all The key-value pair of the index position; Indicates area r i The most relevant m The index of the region; S6.2.2: Collect the key vector and value vector, expressed as follows: （13） in, Represents the collected key vector and value vector; Indicates that from the key matrix K and the value matrix V according to the index Extract relevant key-value pairs; S6.2.3: Apply attention mechanism on the collected key-value pairs and combine it with local context enhancement to obtain context-enhanced token sequence , which is the feature extracted by the T2T branch, is expressed as follows: （14） in, It represents the enhancement of the value matrix V by using depthwise convolution.

6. The sending-end image semantic communication method based on dual-route attention according to claim 1 is characterized in that S7 specifically includes the following steps: S7.1: Extract features from R2R branches and features extracted by the T2T branch Perform splicing to obtain the spliced ​​feature vector : （15）； S7.2: Concatenated feature vectors Use weighted average to fuse and generate the fused feature vector .

Images