Sending end image semantic communication method based on double-route attention

Abstract

The invention discloses a sending end image semantic communication method based on double-route attention. The method comprises the steps of obtaining a to-be-classified image and inputting the to-be-classified image into a sending end; the sending end convolution layer cuts or scales the to-be-classified images to obtain the to-be-classified images with the uniform size; inputting the to-be-classified images with the uniform size into a feature extraction network for feature extraction to obtain a feature map; performing semantic extraction on the feature map; compressing the extracted semantics; dividing the compressed semantics into an R2R branch and a T2T branch, and respectively extracting features related to the semantics; the features extracted from the R2R branch and the T2T branch are fused, and final feature representation containing global and local semantic information is generated; classifying the image according to the final feature representation, and outputting a classification result; and performing semantic communication transmission according to the classification result and the demand of the receiving end. According to the sending end semantic communication method based on the double-route attention, the problem that in the prior art, the information transmission accuracy is low is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of Internet of Things communications, and in particular relates to a sending end image semantic communication method based on dual-route attention. Background Art

[0002] With the advent of the new economy dominated by the information industry, communications and networks have been fully integrated into social production and daily life, becoming the core engine driving the rapid development of my country's information industry. However, traditional communication technology mainly focuses on the accurate transmission of data and the precise restoration of signal waveforms, ignoring the semantic information carried in data or carrier waveforms. Although this approach has made progress in ensuring data integrity, it has exposed some significant defects in the face of increasingly complex scenario-based communication needs.

[0003] First, traditional communication methods cannot effectively handle information transmission at the semantic level, resulting in low accuracy of information transmission. As the concept of the Internet of Everything continues to deepen, future communications are not only about transmission speed and signal accuracy, but also require the ability to effectively deliver meaningful content according to the needs of the receiver. Therefore, how to optimize communication accuracy and improve the ability to transmit semantic information in a resource-limited IoT environment has become an important challenge in current communication technology. Summary of the invention

[0004] The purpose of the present invention is to provide a sending-end image semantic communication method based on dual-route attention, which solves the problem of low information transmission accuracy in the prior art.

[0005] The technical solution adopted by the present invention is a sending-end image semantic communication method based on dual-route attention, which specifically includes the following steps: S1: Get the image to be classified and input it into the sending end; S2: The convolutional layer in the sending end crops or scales the image to be classified to obtain an image to be classified with a uniform size; S3: Input the image to be classified with uniform size into the feature extraction network for feature extraction to obtain a feature map; S4: Extract semantics from feature maps; S5: compress the extracted semantics; S6: Divide the compressed semantics into two branches, R2R and T2T, and extract semantically related features respectively; S7: The semantically relevant features extracted from the R2R and T2T branches are fused to generate the final feature representation containing global and local semantic information; S8: Classify the image according to the final feature representation and output the classification result; perform semantic communication transmission according to the classification result and the needs of the receiving end.

[0006] The present invention is also characterized in that: S3 is represented as follows: (1) In the formula, For images, is the feature map, is the feature extraction network, The network parameters of the feature extraction network.

[0007] S4 specifically includes the following steps: S4.1: Calculate the gradient of semantic concept activation relative to the feature map to obtain the importance weight of the semantic concept , the calculation formula is as follows: (2) In the formula, It represents the neuron activation value before Softmax in the last fully connected layer of the feature extraction network. Represents semantic concepts, w and h Indicates The width and height of the feature map, Representation feature map No. i Line j The activation value at the column; S4.2: Computational semantic concepts The feature map weight vector is as follows: (3) In the formula, Representing semantic concepts The feature map weight vector, N represents the total number of feature maps, Representing semantic concepts The importance weight of each feature map calculated by the feature extraction network for the semantic concept; S4.3: Based on the feature map weight vector of each semantic concept, a feature map weight matrix is ​​formed , which is expressed as follows: (4) In the formula, Semantic Concepts The importance weight vector of ; L represents the total number of semantic concepts; S4.4: For each semantic concept, the feature map is sorted using the weight value to obtain the corresponding relationship between any semantic concept and the importance ranking of the feature map, that is, the semantic relationship, which is expressed as follows: (5) In the formula, represents the total feature map index matrix, Respectively represent different semantic concepts The feature map sorting results.

[0008] S5 specifically includes the following steps: S5.1: According to the semantic relationship, the compression threshold is selected, which is expressed as follows: (6) In the formula, Represents semantic concepts The compression threshold, n Indicates that for each semantic concept in the compression process Only the most important n feature maps will be retained and transmitted, Indicates feature map to semantic concept The importance weight of the feature map is as follows: Weight Less than compression threshold , then the feature map will not be transmitted, that is ; If the feature map Weight Greater than or equal to the compression threshold , then the feature map will be transmitted, that is, ; The specific steps are as follows: S5.1.1: Calculate the number of transmission features n : (7) In the formula, The compression ratio corresponds to the degree of semantic compression at the sender. 0 means no compression and 1 means maximum compression. The actual transmission feature map ratio is , represents the floor function; S5.1.2: Calculate the compression threshold weight based on the number of transmitted features : (8) In the formula, , which indicates the results sorted in descending order according to the weight value. Indicates N The threshold weight of the item; S5.2: Targeting semantic concepts , in the actual transmission characteristics, the former n The index of the most important feature map is expressed as: (9) In the formula, For semantic concepts The feature map index set of T represents transmission. Indicates the reserved n The weight of the feature graph, that is, the weight ranked first among all weights n The weight of the bit; S5.3: Calculate the feature map index vector actually transmitted through the channel , expressed as: (10).

[0009] S6 specifically includes the following steps: S6.1: The R2R branch identifies and processes key regions in the image and extracts features related to semantic concepts by applying an attention mechanism at the region level of the image. The specific steps are as follows: S6.1.1: Using the feature map index vector, select a corresponding key feature region from the image, where the key feature region includes a plurality of pixel blocks; S6.1.2: Use linear mapping to embed the position of each key feature region into a low-dimensional space to generate the query matrix Q, key matrix K, and value matrix V, which are expressed as follows: (11) In the formula, , are the mapping weights of Q, K, and V respectively, Represents the square root of the number of regions; H, W, and C represent the height, width, and number of channels of the image, respectively; represents the index matrix of the routing area; r represents the area; S6.1.3: Calculate the element-by-element average of the query matrix Q and key matrix K of each region in each channel dimension to obtain the query matrix at the region level and key matrix , ; S6.1.4: By and Perform matrix multiplication to generate the adjacency matrix between regions , represents the semantic correlation between regions; Represents the key matrix The transposed matrix of S6.1.5: Using the adjacency matrix Generate routing index matrix , which is the feature extracted by the R2R branch, is expressed as follows: (12) In the formula, Indicates row-first top-k selection operation; S6.2: The T2T branch further applies a fine-grained attention mechanism to the region selected in S6.1, calculates the similarity of each token with other tokens, performs weighted average, and generates a new token sequence containing context information. The specific steps are as follows: S6.2.1: For each region r i The query token in Key-value pairs at index positions; Indicates area r i Most relevant m The index of the region; S6.2.2: Collect the key vector and value vector, expressed as follows: (13) in, Represents the collected key vector and value vector; Indicates that from the key matrix K and the value matrix V according to the index Extract relevant key-value pairs; S6.2.3: Apply the attention mechanism on the collected key-value pairs and combine it with local context enhancement to obtain context-enhanced token sequences , which is the feature extracted by the T2T branch, is expressed as follows: (14) in, It means that the value matrix V is enhanced by using depthwise convolution.

[0010] S7 specifically includes the following steps: S7.1: Extract features from the R2R branch And the features extracted by T2T branch Perform splicing to obtain the spliced ​​feature vector : (15); S7.2: Concatenated feature vector Use weighted average to fuse and generate the fused feature vector .

[0011] The beneficial effects of the present invention are: The present invention is based on a sending-end image semantic communication method with dual-route attention, which improves the accuracy of information transmission by establishing associations between semantic concepts and image features. The present invention method ensures that key semantic information is retained during transmission through semantic compression at the sending end, reduces data loss and misunderstanding, and ultimately improves the receiver's understanding accuracy of semantic information, thereby improving the reliability and accuracy of communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a semantic communication transmission process diagram of the sending end image semantic communication method based on dual-route attention of the present invention; Figure 2 It is a flow chart of the sending end image semantic communication method based on dual-route attention of the present invention; Figure 3 It is a dual-route structure diagram of the sending-end image semantic communication method based on dual-route attention of the present invention; Figure 4 It is a training accuracy data diagram of Example 3 of the sending-end image semantic communication method based on dual-route attention of the present invention and the baseline method; Figure 5 This is a verification accuracy data graph of Example 3 of the sending-end image semantic communication method based on dual-route attention of the present invention and the baseline method. DETAILED DESCRIPTION

[0013] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] like Figure 1 As shown in the figure, the semantic communication transmission process can be summarized as follows: first, the original data such as text, voice, image and video generated by the source are preliminarily processed and then semantic extraction is performed. Semantic extraction is to conduct in-depth analysis of the preliminarily processed data to extract the semantic information therein. Taking image data as an example, semantic extraction includes feature extraction, object recognition, etc. The semantic coding module encodes the extracted semantic information for transmission during the communication process. The semantic coding not only compresses the amount of transmission data, but also retains key information. The channel coding module further encodes the semantically encoded data to enhance its anti-interference ability during the transmission process. Channel coding technology such as error correction code can effectively improve the reliability of data transmission. The present invention uses a single-layer neural network to model the channel. The channel-coded data is transmitted to the receiving end through the transmission channel, and is decoded and restored to output the processed result. The output data can be used for various applications, such as speech recognition, image classification, text understanding, etc.

[0015] The present invention is based on the sending end image semantic communication method of dual-route attention, such as Figure 2 As shown, the specific steps include: S1: Get the image to be classified and input it into the sending end; The experiments used in the present invention use the STL-10 dataset, which consists of images representing 10 different semantic concepts, each of which is associated with a specific image category.

[0016] S2: The convolutional layer at the sending end preprocesses the image to be classified, including cropping or scaling, and adjusting the image to a fixed size, generally 224×224 pixels.

[0017] S3: Input the image to be classified with uniform size into the feature extraction network for feature extraction to obtain a feature map; Use the feature extraction network to resize the image to a fixed size I For feature extraction, the process can be expressed as: (1) In the formula, For images, is the feature map, is the feature extraction network, The network parameters of the feature extraction network.

[0018] like Figure 3 As shown, a semantic communication convolutional neural network is used for image classification tasks to achieve efficient semantic transmission and maximize the accuracy of classification tasks.

[0019] The sender first resizes the image to a fixed size, uses a feature extraction network, such as the ResNet18 network, to extract features from the input image, and then semantically compresses the extracted feature map based on the extracted semantic relationship to further reduce the amount of data actually transmitted. The feature map is passed through the classifier and the Softmax layer to output the classification result.

[0020] S4: Extract semantics from the feature map, which specifically includes the following steps: S4.1: Calculate the gradient of semantic concept activation relative to the feature map to obtain the importance weight of the semantic concept , the calculation formula is as follows: (2) In the formula, It represents the neuron activation value before Softmax in the last fully connected layer of the feature extraction network. Represents semantic concepts, w and h Indicates The width and height of the feature map, Representation feature map No. i Line j The activation value at the column; The activation value is the output value of the neuron after the neural network calculates; for example, the number of neurons in the last layer of the classification problem is N, and each neuron corresponds to a category, that is, a semantic concept. Input an image to be classified, and after the neural network forward calculation propagation, each neuron in the last layer will output an activation value.

[0021] S4.2: Computational semantic concepts The feature map weight vector is as follows: (3) In the formula, Representing semantic concepts The feature map weight vector, N represents the total number of feature maps, Representing semantic concepts The importance weight of each feature map calculated by the feature extraction network for the semantic concept; S4.3: Based on the feature map weight vector of each semantic concept, a feature map weight matrix is ​​formed , which is expressed as follows: (4) In the formula, Semantic Concepts The importance weight vector of ; if there are multiple semantic concepts involved in the image, then L is the total number of semantic concepts.

[0022] S4.4: For each semantic concept, the feature map is sorted using the weight value to obtain the corresponding relationship between any semantic concept and the importance ranking of the feature map, that is, the semantic relationship, which is expressed as follows: (5) In the formula, Represents the total feature map index matrix, which contains the importance ranking of feature maps for different semantic concepts. Respectively represent different semantic concepts The feature map sorting results.

[0023] S5: compress the extracted semantics; Based on the actual communication and computing resources, combined with the feature map weights and semantic relationships, the extracted semantics are compressed to remove redundant information, including: S5.1: According to the semantic relationship, the compression threshold is selected, which is expressed as follows: (6) In the formula, Represents semantic concepts The compression threshold, n Indicates that for each semantic concept in the compression process Only the most importantn feature maps will be retained and transmitted, Indicates feature map to semantic concept The importance weight of the feature map is as follows: Weight Less than compression threshold , then the feature map will not be transmitted, that is ; If the feature map Weight Greater than or equal to the compression threshold , then the feature map will be transmitted, that is, ; The specific steps are as follows: S5.1.1: Calculate the number of transmission features n : (7) In the formula, The compression ratio corresponds to the degree of semantic compression at the sender. 0 means no compression and 1 means maximum compression. The actual transmission feature map ratio is , represents the floor function; S5.1.2: Calculate the compression threshold weight based on the number of transmitted features : (8) In the formula, , which indicates the results sorted in descending order according to the weight value. Indicates N The threshold weight of the item; S5.2: Targeting semantic concepts , in the actual transmission characteristics, the former n The index of the most important feature map is expressed as: (9) In the formula, For semantic concepts The feature map index set represents the feature map index that is finally retained for transmission after compression; T represents transmission, Indicates the reserved n The weight of the feature graph, that is, the weight ranked first among all weights n The weight of the bit; S5.3: Calculate the feature map index vector actually transmitted through the channel , expressed as: (10).

[0024] S6: Divide the compressed semantics into two branches, R2R and T2T, and extract semantically related features respectively; The compressed feature map is processed with dual-route attention and divided into two branches: Region-to-Region (R2R) and Token-to-Token (T2T). Features are extracted from the two branches respectively. The specific process is as follows: S6.1: The R2R branch identifies and processes key regions in the image and extracts features related to semantic concepts by applying an attention mechanism at the region level of the image. The specific steps are as follows: S6.1.1: Using the feature map index vector, select a corresponding key feature region from the image, where the key feature region includes a plurality of pixel blocks; S6.1.2: Use linear mapping to embed the position of each key feature region into a low-dimensional space to generate the query matrix Q, key matrix K, and value matrix V, which are expressed as follows: (11) In the formula, , are the mapping weights of Q, K, and V respectively, Represents the square root of the number of regions; H, W, and C represent the height, width, and number of channels of the image, respectively; represents the index matrix of the routing area; r represents the area; S6.1.3: Calculate the element-by-element average of the query matrix Q and key matrix K of each region in each channel dimension to obtain the query matrix at the region level and key matrix , ; S6.1.4: By and Perform matrix multiplication to generate the adjacency matrix between regions , represents the semantic correlation between regions; Represents the key matrix The transposed matrix of S6.1.5: Using the adjacency matrix Generate routing index matrix , which is the feature extracted by the R2R branch, is expressed as follows: (12) In the formula, Indicates row-first top-k selection operation; S6.2: The T2T branch further applies a fine-grained attention mechanism to the region selected in S6.1, calculates the similarity of each token with other tokens, performs weighted average, and generates a new token sequence containing context information. The specific steps are as follows: S6.2.1: For each region r i The query token in Key-value pairs at index positions; Indicates area r i Most relevant m The index of the region; S6.2.2: Collect the key vector and value vector, expressed as follows: (13) in, Represents the collected key vector and value vector; Indicates that from the key matrix K and the value matrix V according to the index Extract relevant key-value pairs; S6.2.3: Apply the attention mechanism on the collected key-value pairs and combine it with local context enhancement to obtain context-enhanced token sequences , which is the feature extracted by the T2T branch, is expressed as follows: (14) in, It means that the value matrix V is enhanced by using depthwise convolution.

[0025] S7: The features extracted from the R2R and T2T branches are fused to generate the final feature representation containing global and local semantic information. The specific process is as follows: S7.1: Extract features from the R2R branch And the features extracted by T2T branch Perform splicing to obtain the spliced ​​feature vector : (15); S7.2: Concatenated feature vector Use weighted average to fuse and generate the fused feature vector .

[0026] S8: Classify the image according to the final feature representation and output the classification result; perform semantic communication transmission according to the classification result and the needs of the receiving end.

[0027] Example 1 This embodiment provides a sending end image semantic communication method based on dual-route attention, such as Figure 2 As shown, S1: Get the image to be classified and input it into the sending end; S2: The convolutional layer in the sending end crops or scales the image to be classified to obtain an image to be classified with a uniform size; S3: Input the image to be classified with uniform size into the feature extraction network for feature extraction to obtain a feature map; S4: Extract semantic information from feature maps; S5: compress the extracted semantics; S6: Divide the compressed semantics into two branches, R2R and T2T, and extract semantically related features respectively; S7: The semantically relevant features extracted from the R2R and T2T branches are fused to generate the final feature representation containing global and local semantic information; S8: Classify the image according to the final feature representation and output the classification result; perform semantic communication transmission according to the classification result and the needs of the receiving end.

[0028] Example 2 This embodiment provides a sending end image semantic communication method based on dual-route attention, such as Figure 2 As shown, S1: Get the image to be classified and input it into the sending end; S2: The convolutional layer at the sending end crops or scales the image to be classified to obtain an image to be classified with a uniform size; S3: Input the image to be classified with uniform size into the feature extraction network for feature extraction to obtain a feature map, specifically: (1) In the formula, For images, is the feature map, is the feature extraction network, The network parameters of the feature extraction network.

[0029] S4: Extract semantic information from feature maps; S4.1: Calculate the gradient of semantic concept activation relative to the feature map to obtain the importance weight of the semantic concept , the calculation formula is as follows: (2) In the formula, It represents the neuron activation value before Softmax in the last fully connected layer of the feature extraction network. Represents semantic concepts, w and h Indicates The width and height of the feature map, Representation feature map No. i Line j The activation value at the column; S4.2: Computational semantic concepts The feature map weight vector is as follows: (3) In the formula, Representing semantic concepts The feature map weight vector, N represents the total number of feature maps, Representing semantic concepts The importance weight of each feature map calculated by the feature extraction network for the semantic concept; S4.3: Based on the feature map weight vector of each semantic concept, a feature map weight matrix is ​​formed , which is expressed as follows: (4) In the formula, Semantic Concepts The importance weight vector of ; L represents the total number of semantic concepts; S4.4: For each semantic concept, the feature map is sorted using the weight value to obtain the corresponding relationship between any semantic concept and the importance ranking of the feature map, that is, the semantic relationship, which is expressed as follows: (5) In the formula, represents the total feature map index matrix, Respectively represent different semantic concepts The feature map sorting results.

[0030] S5: compress the extracted semantics; S5.1: According to the semantic relationship, the compression threshold is selected, which is expressed as follows: (6) In the formula, Represents semantic concepts The compression threshold, n Indicates that for each semantic concept in the compression process Only the most important n feature maps will be retained and transmitted, Indicates feature map to semantic concept The importance weight of the feature map is as follows: Weight Less than compression threshold , then the feature map will not be transmitted, that is ; If the feature map Weight Greater than or equal to the compression threshold , then the feature map will be transmitted, that is, ; The specific steps are as follows: S5.1.1: Calculate the number of transmission features n : (7) In the formula, The compression ratio corresponds to the degree of semantic compression at the sender. 0 means no compression and 1 means maximum compression. The actual transmission feature map ratio is , represents the floor function; S5.1.2: Calculate the compression threshold weight based on the number of transmitted features : (8) In the formula, , which indicates the results sorted in descending order according to the weight value. Indicates N The threshold weight of the item; S5.2: Targeting semantic concepts , in the actual transmission characteristics, the former n The index of the most important feature map is expressed as: (9) In the formula, For semantic concepts The feature map index set of T represents transmission. Indicates the reserved n The weight of the feature graph, that is, the weight ranked first among all weights n The weight of the bit; S5.3: Calculate the feature map index vector actually transmitted through the channel , expressed as: (10).

[0031] S6: Divide the compressed semantics into two branches, R2R and T2T, and extract semantically related features respectively; S6.1: The R2R branch identifies and processes key regions in the image and extracts features related to semantic concepts by applying an attention mechanism at the region level of the image. The specific steps are as follows: S6.1.1: Using the feature map index vector, select a corresponding key feature region from the image, where the key feature region includes a plurality of pixel blocks; S6.1.2: Use linear mapping to embed the position of each key feature region into a low-dimensional space to generate the query matrix Q, key matrix K, and value matrix V, which are expressed as follows: (11) In the formula, , are the mapping weights of Q, K, and V respectively, Represents the square root of the number of regions; H, W, and C represent the height, width, and number of channels of the image, respectively; represents the index matrix of the routing area; r represents the area; S6.1.3: Calculate the element-by-element average of the query matrix Q and key matrix K of each region in each channel dimension to obtain the query matrix at the region level and key matrix , ; S6.1.4: By and Perform matrix multiplication to generate the adjacency matrix between regions , represents the semantic correlation between regions; Represents the key matrix The transposed matrix of S6.1.5: Using the adjacency matrix Generate routing index matrix , which is the feature extracted by the R2R branch, is expressed as follows: (12) In the formula, Indicates row-first top-k selection operation; S6.2: The T2T branch further applies a fine-grained attention mechanism to the region selected in S6.1, calculates the similarity of each token with other tokens, performs weighted average, and generates a new token sequence containing context information. The specific steps are as follows: S6.2.1: For each region r i The query token in Key-value pairs at index positions; Indicates area r i Most relevant m The index of the region; S6.2.2: Collect the key vector and value vector, expressed as follows: (13) in, Represents the collected key vector and value vector; Indicates that from the key matrix K and the value matrix V according to the index Extract relevant key-value pairs; S6.2.3: Apply the attention mechanism on the collected key-value pairs and combine it with local context enhancement to obtain context-enhanced token sequences , which is the feature extracted by the T2T branch, is expressed as follows: (14) in, It means that the value matrix V is enhanced by using depthwise convolution.

[0032] S7: The semantically relevant features extracted from the R2R and T2T branches are fused to generate the final feature representation containing global and local semantic information; S7.1: Extract features from the R2R branch And the features extracted by T2T branch Perform splicing to obtain the spliced ​​feature vector : (15); S7.2: Concatenated feature vector Use weighted average to fuse and generate the fused feature vector .

[0033] S8: Classify the image according to the final feature representation and output the classification result; perform semantic communication transmission according to the classification result and the needs of the receiving end.

[0034] Example 3 This embodiment provides a sending end image semantic communication method based on dual-route attention, such as Figure 2 As shown, S1: Use the STL-10 data set and input the sender; S2: The sending convolutional layer crops or scales the image to be classified and adjusts the image to a fixed size of 224×224 pixels; S3: Input the fixed-size image into the ResNet18 feature extraction network for feature extraction to obtain a feature map, which is represented as: (1) In the formula, For images, is the feature map, is the feature extraction network, The network parameters of the feature extraction network.

[0035] S4: Extract semantic information from feature maps; S4.1: Calculate the gradient of semantic concept activation relative to the feature map to obtain the importance weight of the semantic concept , the calculation formula is as follows: (2) In the formula, It represents the neuron activation value before Softmax in the last fully connected layer of the feature extraction network. Represents semantic concepts, w and h Indicates The width and height of the feature map, Representation feature map No. i Line j The activation value at the column; S4.2: Computational semantic concepts The feature map weight vector is as follows: (3) In the formula, Representing semantic concepts The feature map weight vector, N represents the total number of feature maps, Representing semantic concepts The importance weight of each feature map calculated by the feature extraction network for the semantic concept; S4.3: Based on the feature map weight vector of each semantic concept, a feature map weight matrix is ​​formed , which is expressed as follows: (4) In the formula, Semantic Concepts The importance weight vector of ; L represents the total number of semantic concepts; S4.4: For each semantic concept, the feature map is sorted using the weight value to obtain the corresponding relationship between any semantic concept and the importance ranking of the feature map, that is, the semantic relationship, which is expressed as follows: (5) In the formula, represents the total feature map index matrix, Respectively represent different semantic concepts The feature map sorting results.

[0036] S5: Set the compression rate to 0.2 to compress the extracted semantics; S5.1: According to the semantic relationship, the compression threshold is selected, which is expressed as follows: (6) In the formula, Represents semantic concepts The compression threshold, n Indicates that for each semantic concept in the compression process Only the most important n feature maps will be retained and transmitted, Indicates feature map to semantic concept The importance weight of the feature map is as follows: Weight Less than compression threshold , then the feature map will not be transmitted, that is ; If the feature map Weight Greater than or equal to the compression threshold , then the feature map will be transmitted, that is, ; The specific steps are as follows: S5.1.1: Calculate the number of transmission features n : (7) In the formula, The compression ratio corresponds to the degree of semantic compression at the sender. 0 means no compression and 1 means maximum compression. The actual transmission feature map ratio is , represents the floor function; S5.1.2: Calculate the compression threshold weight based on the number of transmitted features : (8) In the formula, , which indicates the results sorted in descending order according to the weight value. Indicates N The threshold weight of the item; S5.2: Targeting semantic concepts , in the actual transmission characteristics, the former n The index of the most important feature map is expressed as: (9) In the formula, For semantic concepts The feature map index set of T represents transmission. Indicates the reserved n The weight of the feature graph, that is, the weight ranked first among all weights n The weight of the bit; S5.3: Calculate the feature map index vector actually transmitted through the channel , expressed as: (10).

[0037] S6: Divide the compressed semantics into two branches, R2R and T2T, and extract semantically related features respectively; S6.1: The R2R branch identifies and processes key regions in the image and extracts features related to semantic concepts by applying an attention mechanism at the region level of the image. The specific steps are as follows: S6.1.1: Using the feature map index vector, select a corresponding key feature region from the image, where the key feature region includes a plurality of pixel blocks; S6.1.2: Use linear mapping to embed the position of each key feature region into a low-dimensional space to generate the query matrix Q, key matrix K, and value matrix V, which are expressed as follows: (11) In the formula, , are the mapping weights of Q, K, and V respectively, Represents the square root of the number of regions; H, W, and C represent the height, width, and number of channels of the image, respectively; represents the index matrix of the routing area; r represents the area; S6.1.3: Calculate the element-by-element average of the query matrix Q and key matrix K of each region in each channel dimension to obtain the query matrix at the region level and key matrix , ; S6.1.4: By and Perform matrix multiplication to generate the adjacency matrix between regions , represents the semantic correlation between regions; Represents the key matrix The transposed matrix of S6.1.5: Using the adjacency matrix Generate routing index matrix , which is the feature extracted by the R2R branch, is expressed as follows: (12) In the formula, Indicates row-first top-k selection operation; S6.2: The T2T branch further applies a fine-grained attention mechanism to the region selected in S6.1, calculates the similarity of each token with other tokens, performs weighted average, and generates a new token sequence containing context information. The specific steps are as follows: S6.2.1: For each region r iThe query token in Key-value pairs at index positions; Indicates area r i Most relevant m The index of the region; S6.2.2: Collect the key vector and value vector, expressed as follows: (13) in, Represents the collected key vector and value vector; Indicates that from the key matrix K and the value matrix V according to the index Extract relevant key-value pairs; S6.2.3: Apply the attention mechanism on the collected key-value pairs and combine it with local context enhancement to obtain context-enhanced token sequences , which is the feature extracted by the T2T branch, is expressed as follows: (14) in, It means that the value matrix V is enhanced by using depthwise convolution.

[0038] S7: The semantically relevant features extracted from the R2R and T2T branches are fused to generate the final feature representation containing global and local semantic information; S7.1: Extract features from the R2R branch And the features extracted by T2T branch Perform splicing to obtain the spliced ​​feature vector : (15); S7.2: Concatenated feature vector Use weighted average to fuse and generate the fused feature vector .

[0039] S8: Classify the image according to the final feature representation and output the classification result; perform semantic communication transmission according to the classification result and the needs of the receiving end.

[0040] Simulation experiment: The experimental platform consists of an i7-12700F processor and an NVIDIA GeForce RTX 3060 (12G) GPU, and CUDA 12.2 is used for computing. In addition, the experiment uses Python 3.8.17 as the programming language and a deep learning framework built on PyTorch1.9.0.

[0041] The method of the paper "Semantic Communication Methods for Intelligent Tasks in Artificial Intelligence Internet of Things" is used as the baseline method, using the STL-10 dataset, which consists of images representing 10 different semantic concepts, each of which is associated with a specific image category. In each category, there are 1,300 images with a resolution of 96 dpi×96 dpi, divided into 500 for training and 800 for testing. The experimental parameters are set as follows: the number of iterations is 100, the batch size is 32, the optimizer is Adam, the learning rate is 0.0001, the number of categories is 10, and the compression rate is 0.2. The experimental environment and parameter settings are the same as those in Example 3.

[0042] The training accuracy and verification accuracy of the method proposed in the present invention are compared with those of the limit method. Figure 4 and Figure 5 As shown, the image classification accuracy of the method proposed in the present invention is improved by about 5%, and has higher training accuracy and verification accuracy.

Claims

1. A sender-side image semantic communication method based on dual-route attention, characterized in that: The specific steps include: S1: Get the image to be classified and input it into the sending end; S2: the convolutional layer in the transmitting end crops or scales the image to be classified to obtain an image to be classified with a uniform size; S3: inputting the image to be classified with a uniform size into a feature extraction network for feature extraction to obtain a feature map; S4: performing semantic extraction on the feature map; S5: compress the extracted semantics; S6: Divide the compressed semantics into two branches, R2R and T2T, and extract semantically related features respectively; S7: The semantically relevant features extracted from the R2R and T2T branches are fused to generate the final feature representation containing global and local semantic information; S8: Classify the image according to the final feature representation and output the classification result; perform semantic communication transmission according to the classification result and the needs of the receiving end.

2. The sending end image semantic communication method based on dual-route attention according to claim 1 is characterized in that: S3 is represented as follows: （1） In the formula, For images, is the feature map, is the feature extraction network, The network parameters of the feature extraction network.

3. The sending end image semantic communication method based on dual-route attention according to claim 2 is characterized in that: S4 specifically includes the following steps: S4.1: Calculate the gradient of semantic concept activation relative to the feature map to obtain the importance weight of the semantic concept , the calculation formula is as follows: （2） In the formula, It represents the neuron activation value before Softmax in the last fully connected layer of the feature extraction network. Represents semantic concepts, w and h Indicates The width and height of the feature map, Representation feature map No. i Line j The activation value at the column; S4.2: Computational semantic concepts The feature map weight vector is as follows: （3） In the formula, Representing semantic concepts The feature map weight vector, N represents the total number of feature maps, Representing semantic concepts The importance weight of each feature map calculated by the feature extraction network for the semantic concept; S4.3: Based on the feature map weight vector of each semantic concept, a feature map weight matrix is ​​formed , which is expressed as follows: （4） In the formula, Semantic Concepts The importance weight vector of ; L represents the total number of semantic concepts; S4.4: For each semantic concept, the feature map is sorted using the weight value to obtain the corresponding relationship between any semantic concept and the importance ranking of the feature map, that is, the semantic relationship, which is expressed as follows: （5） In the formula, represents the total feature map index matrix, Respectively represent different semantic concepts The feature map sorting results.

4. The sending-end image semantic communication method based on dual-route attention according to claim 1 is characterized in that: S5 specifically includes the following steps: S5.1: According to the semantic relationship, the compression threshold is selected, which is expressed as follows: （6） In the formula, Represents semantic concepts The compression threshold, n Indicates that for each semantic concept in the compression process Only the most important n feature maps will be retained and transmitted, Indicates feature map to semantic concept The importance weight of the feature map is as follows: Weight Less than compression threshold , then the feature map will not be transmitted, that is ; If the feature map Weight Greater than or equal to the compression threshold , then the feature map will be transmitted, that is, ; The specific steps are as follows: S5.1.1: Calculate the number of transmission features n : （7） In the formula, The compression ratio corresponds to the degree of semantic compression at the sender. 0 means no compression and 1 means maximum compression. The actual transmission feature map ratio is , represents the floor function; S5.1.2: Calculate the compression threshold weight based on the number of transmitted features : （8） In the formula, , which indicates the results sorted in descending order according to the weight value. Indicates N The threshold weight of the item; S5.2: Targeting semantic concepts , in the actual transmission characteristics, the former n The index of the most important feature map is expressed as: （9） In the formula, For semantic concepts The feature map index set of ; T represents transmission, Indicates the reserved n The weight of the feature graph, that is, the weight ranked first among all weights n The weight of the bit; S5.3: Calculate the feature map index vector actually transmitted through the channel , expressed as: （10）。 5. The sending end image semantic communication method based on dual-route attention according to claim 1 is characterized in that: S6 specifically includes the following steps: S6.1: The R2R branch identifies and processes key regions in the image and extracts features related to semantic concepts by applying an attention mechanism at the region level of the image. The specific steps are as follows: S6.1.1: Using the feature map index vector, select a corresponding key feature region from the image, where the key feature region includes a plurality of pixel blocks; S6.1.2: Use linear mapping to embed the position of each key feature region into a low-dimensional space to generate the query matrix Q, key matrix K, and value matrix V, which are expressed as follows: （11） In the formula, , are the mapping weights of Q, K, and V respectively, Represents the square root of the number of regions; H, W, and C represent the height, width, and number of channels of the image, respectively; represents the index matrix of the routing area; r represents the area; S6.1.3: Calculate the element-by-element average of the query matrix Q and key matrix K of each region in each channel dimension to obtain the query matrix at the region level and key matrix , ; S6.1.4: By and Perform matrix multiplication to generate the adjacency matrix between regions , represents the semantic correlation between regions; Represents the key matrix The transposed matrix of S6.1.5: Using the adjacency matrix Generate routing index matrix , which is the feature extracted by the R2R branch, is expressed as follows: （12） In the formula, Indicates row-first top-k selection operation; S6.2: The T2T branch further applies a fine-grained attention mechanism to the region selected in S6.1, calculates the similarity of each token with other tokens, performs weighted average, and generates a new token sequence containing context information. The specific steps are as follows: S6.2.1: For each region r i The query token in Key-value pairs at index positions; Indicates area r i Most relevant m The index of the region; S6.2.2: Collect the key vector and value vector, expressed as follows: （13） in, Represents the collected key vector and value vector; Indicates that from the key matrix K and the value matrix V according to the index Extract relevant key-value pairs; S6.2.3: Apply the attention mechanism on the collected key-value pairs and combine it with local context enhancement to obtain context-enhanced token sequences , which is the feature extracted by the T2T branch, is expressed as follows: （14） in, It means that the value matrix V is enhanced by using depthwise convolution.

6. The sending end image semantic communication method based on dual-route attention according to claim 1 is characterized in that: S7 specifically includes the following steps: S7.1: Extract features from the R2R branch And the features extracted by T2T branch Perform splicing to obtain the spliced ​​feature vector : （15）； S7.2: Concatenated feature vector Use weighted average to fuse and generate the fused feature vector .

Images