A generative artificial intelligence-based semantic communication method and system

Abstract

The application discloses a kind of based on generative artificial intelligence's semantic communication method and system, the method includes: based on bootstrap language-image pretraining model and uniform perception analysis network model, and introduce Grad-CAM method, the semantic features of original image are filtered, obtain refined semantic features;Based on the model driven PPO algorithm of importance, the semantic feature selection of refined semantic features is carried out, and the selected semantic feature is obtained;Through non-orthogonal multiple access transmission method, jointly solve receiving beam forming and NOMA decoding sequence, and the selected semantic feature is transmitted to base station by wireless;Based on generative artificial intelligence semantic recovery method, the selected semantic feature is recovered and handled, and the recovered image is output.The application can simultaneously improve semantic fidelity and transmission delay performance.The application can be widely applied to semantic communication transmission technical field as a kind of based on generative artificial intelligence's semantic communication method and system.

Description

Technical Field

[0001] This invention relates to the field of semantic communication transmission technology, and in particular to a semantic communication method and system based on generative artificial intelligence. Background Technology

[0002] Addressing the challenges of scarce spectrum resources and high transmission latency in large-scale Internet of Things (IoT) applications, non-orthogonal multiple access (NOMA) technology and semantic communication methods are considered promising solutions. In particular, thanks to the rapid development of generative artificial intelligence (AI), applying it to information extraction and retrieval has become a new path in semantic communication. NOMA, through non-orthogonal resource reuse and intelligent interference management, has become a key technology for increasing capacity and connection numbers in 5G / 6G. Its value in semantic communication lies in adapting to heterogeneous semantic feature requirements through dynamic resource allocation, providing physical layer support for the efficient transmission of high-dimensional AI data. Unlike traditional bit-level communication, semantic communication focuses on conveying the meaning of information, significantly reducing the amount of data transmitted and alleviating spectrum resource pressure. Furthermore, semantic communication exhibits strong robustness; even in scenarios with low signal-to-noise ratios or partial feature loss, the receiving end, based on the contextual understanding and knowledge completion capabilities of generative AI tools, can still reconstruct semantic information with high fidelity, overcoming the stringent dependence of traditional communication on channel quality. In addition, by extracting and transmitting key semantic information, it can reshape users' traffic needs, thereby effectively adapting to dynamic traffic scenarios.

[0003] Existing research has explored various techniques for achieving semantic communication. For example, knowledge graphs are used to characterize the inherent relationships between semantic entities. By filtering out less relevant entities, transmission can focus on more critical information, thus reducing data volume. Alternatively, a deep learning-based semantic communication system has been proposed, utilizing joint source-channel coding to achieve efficient end-to-end semantic transmission. Given the advantages of semantic communication, integrating it into existing wireless networks holds promise for overcoming their inherent limitations. For instance, a UAV-assisted semantic communication network has been proposed. In this network, UAVs can collect lightweight semantic information, enabling faster service to ground users and significantly reducing the system's information age. Another approach is to integrate semantic communication into non-orthogonal multiple access (NOMA) systems. With the help of semantic communication, co-channel interference between users is mitigated, resulting in higher overall transmission efficiency.

[0004] However, existing semantic communication technologies generally rely on static knowledge bases or deep learning models, making it difficult to adapt to dynamic network changes. Although model fine-tuning can provide some adaptability, it introduces significant latency and computational overhead, limiting the practicality of such methods in real-time and resource-constrained scenarios.

[0005] Generative Artificial Intelligence (GAI) models, by leveraging massive amounts of model parameters and extensive pre-training on large-scale datasets, demonstrate strong generalization capabilities across diverse wireless scenarios. By generating user-customized content through customized prompts, GAI avoids the need for repeated retraining. Recent research shows that GAI can play multiple roles in wireless networks, such as optimizers and semantic decoders. It has been demonstrated that GAI can achieve optimization without gradient information, making it suitable for problems with unknown or non-differentiable loss functions. Furthermore, GAI has been used as a semantic decoder in semantic communication, reconstructing road scenes and predicting future states based on high-level descriptions of road node locations.

[0006] Typically, the design of semantic communication can be formulated as a joint optimization problem of semantic control variables and transmission control variables. A direct approach is to decompose it into a series of subproblems and solve them iteratively. However, since semantic models often operate as black boxes, directly applying traditional model-based optimization methods is challenging. Deep reinforcement learning (DRL) has been considered an effective solution for handling complex decision-making tasks under conditions of incomplete information. Through interaction with the environment, DRL agents can continuously improve their decision-making capabilities based on accumulated experience. Therefore, DRL can serve as an efficient method to solve optimization problems in semantic communication. For example, a proposed UAV-assisted semantic communication network uses DRL to jointly optimize the scale of transmitted semantic symbols and the trajectory planning of the UAV. Although DRL can jointly optimize all control variables in semantic communication, it may ignore available model information, such as channel conditions and interference characteristics. Furthermore, as the dimensionality of control variables increases, the computational cost of the learning process increases significantly, which may hinder convergence to the optimal solution. Summary of the Invention

[0007] To address the aforementioned technical problems, the present invention aims to provide a semantic communication method and system based on generative artificial intelligence, which can simultaneously improve semantic fidelity and transmission latency performance.

[0008] The first technical solution adopted in this invention is: a semantic communication method based on generative artificial intelligence, comprising the following steps:

[0009] Based on the bootstrap language-image pre-trained model and the unified perception parsing network model, and by introducing the Grad-CAM method, the semantic features of the original image are filtered to obtain refined semantic features.

[0010] Based on the importance matching PPO algorithm, semantic feature selection is performed on the refined semantic features to obtain the selected semantic features;

[0011] By using a non-orthogonal multiple access transmission method, the receiving beamforming and NOMA decoding order are jointly solved, and the selected semantic features are wirelessly transmitted to the base station.

[0012] Based on the generative artificial intelligence semantic restoration method, the selected semantic features are restored and the restored image is output.

[0013] Furthermore, the step of filtering the semantic features of the original image based on the bootstrap language-image pre-trained model and the unified perception parsing network model, and introducing the Grad-CAM method to obtain refined semantic features, specifically includes:

[0014] The original image is input into the image encoder to extract high-level visual features and guide the decoder to generate corresponding semantic feature elements, which include text feature elements and visual feature elements.

[0015] Calculate the gradient of each semantic feature element relative to the high-level visual features, quantify the contribution of the semantic feature element to the corresponding image region, and generate a set of heatmaps.

[0016] Construct a correlation matrix and a contribution matrix, and combine them with a heatmap set to obtain the importance scores of semantic feature elements;

[0017] By introducing a predefined threshold, semantic feature elements are filtered based on their importance scores to obtain refined semantic features.

[0018] Furthermore, the step of selecting semantic features from the refined semantic features using the importance-matching PPO algorithm to obtain the selected semantic features specifically includes:

[0019] An importance-matching PPO algorithm is constructed, which includes a model-free PPO module and a model-based optimization module.

[0020] Based on the model-free PPO module, a state space, an action space, and a reward function are defined. The state space includes the channel conditions of all semantic user equipment and the non-orthogonal multiple access decoding order. The action space includes the semantic feature selection strategy, the receive beamforming strategy, and the non-orthogonal multiple access decoding order.

[0021] Given an action from the modelless PPO module, optimize it using the model-based optimization module, and output the optimized action.

[0022] Merge the actions of the model-free PPO module with the optimized actions, and execute the merged actions in the wireless network to obtain the next state and feedback reward;

[0023] The importance-matching PPO algorithm is executed cyclically based on the next state and feedback reward to select semantic features from the refined semantic features, thus obtaining the selected semantic features.

[0024] Furthermore, the specific expression of the reward function is as follows:

[0025]

[0026] In the above formula, Represents the penalty function. Represents the reward function, Indicates the actions after the merger. This refers to the perceived similarity metric used to measure user... The accuracy of semantic information recovery The weighting coefficients are used to balance semantic recovery performance and transmission latency performance. Indicates transmission delay. Indicates user index, This indicates the number of users.

[0027] Furthermore, the step of jointly solving the receive beamforming and NOMA decoding order using a non-orthogonal multiple access (NOMA) transmission method, and then wirelessly transmitting the selected semantic features to the base station, specifically includes:

[0028] A receive beamforming strategy is defined. Based on the non-orthogonal multiple access decoding order, continuous interference cancellation is used to decode the superimposed signal. The problem is modeled as a standard generalized eigenvalue problem. The optimal receive beamforming vector is obtained by solving the eigenvector corresponding to the maximum eigenvalue.

[0029] Based on the optimal received beamforming vector, auxiliary variables and a quadratic transformation are introduced to approximate SINR and transform it into a convex optimization problem. The discrete decoding order constraint is linearized using a first-order Taylor expansion and iteratively solved to determine the optimal NOMA decoding order to match the semantic traffic requirements of each user device.

[0030] Based on the optimal NOMA decoding order, the semantic traffic requirements of each user device are matched, and the selected semantic features are wirelessly transmitted to the base station.

[0031] Furthermore, the step of the generative artificial intelligence-based semantic restoration method, which involves restoring the selected semantic features and outputting the restored image, specifically includes:

[0032] Semantic features are used as conditions to guide the denoising process. The selected semantic features are restored through a multi-step denoising process to obtain a preliminary restored image.

[0033] Perceptual similarity is measured by comparing the differences between images in the deep feature representation space. The semantic restoration performance is evaluated by calculating the LPIPS value between the original image and the restored image, and the restored image is output.

[0034] Furthermore, the expression for restoring the selected semantic features is specifically shown below:

[0035]

[0036] In the above formula, Indicates control parameters, Representing feature information, This represents a denoising network. and Indicates user In the noise reduction process, the first and The noise situation of the step, Indicates the speed control coefficient. The index representing the number of denoising steps.

[0037] Furthermore, the specific expression for calculating the LPIPS value is as follows:

[0038]

[0039] In the above formula, Indicates the LPIPS value. and The feature extractor and the corresponding evaluation network are represented by the first... Layer weights, Represents the original image. Indicates image restoration. This indicates the total number of layers in the network being evaluated.

[0040] The second technical solution adopted in this invention is: a semantic communication system based on generative artificial intelligence, comprising:

[0041] The first module is used to filter the semantic features of the original image based on the bootstrap language-image pre-trained model and the unified perception parsing network model, and introduces the Grad-CAM method to obtain the refined semantic features.

[0042] The second module is used to perform semantic feature selection on the refined semantic features based on the importance matching PPO algorithm, and obtain the selected semantic features.

[0043] The third module is used to jointly solve the receiving beamforming and NOMA decoding order through a non-orthogonal multiple access transmission method, and then wirelessly transmit the selected semantic features to the base station.

[0044] The fourth module is used to perform recovery processing on the selected semantic features based on generative artificial intelligence semantic recovery methods, and output the recovered image.

[0045] The beneficial effects of the method and system of this invention are as follows: This invention proposes a generative artificial intelligence (GAI)-assisted semantic non-orthogonal multiple access (NOMA) transmission framework. Non-orthogonal multiple access (NOMA) allocates transmission priority to semantic user equipment (SU) based on the semantic feature scale. At the same time, the quality of semantic feature selection restricts the effect of semantic recovery. By jointly optimizing the semantic feature selection strategy, the non-orthogonal multiple access (NOMA) decoding order, and the reception beamforming strategy of the access point (AP), the semantic fidelity and transmission delay performance can be improved simultaneously. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the steps of a semantic communication method based on generative artificial intelligence according to the present invention.

[0047] Figure 2 This is a structural block diagram of a semantic communication system based on generative artificial intelligence according to the present invention;

[0048] Figure 3 This is a schematic diagram of the NOMA communication process assisted by generative artificial intelligence provided in a specific embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the semantic communication process provided in a specific embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of a feature clipping method based on importance cross-modal matching provided in a specific embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of the importance-based model-driven PPO algorithm framework provided in a specific embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram illustrating the convergence performance of different algorithms provided in specific embodiments of the present invention;

[0053] Figure 8 These are schematic diagrams illustrating system performance under different environments provided in specific embodiments of the present invention;

[0054] Figure 9 This is a schematic diagram illustrating the verification results of the visual feature selection function provided in a specific embodiment of the present invention. Detailed Implementation

[0055] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0056] First, it's important to note that the rapid development of the Internet of Things (IoT) in recent years has become a core driving force for digital transformation. Hundreds of millions of smart devices, from industrial sensors and wearables to smart home systems and connected vehicle terminals, are continuously connecting to networks, collecting, processing, and exchanging massive amounts of data in real time. This exponential growth in the number of connected devices and the explosive trend in data generation have significantly exacerbated the demand for network traffic. However, this surge in traffic poses a severe challenge to existing communication infrastructure. The core issues are manifested in two aspects: first, the increasing scarcity of spectrum resources—limited wireless spectrum is insufficient to support such a massive concurrent data transmission demand; second, the significant increase in transmission latency—in critical application scenarios requiring real-time response (such as industrial automation control, telemedicine, and autonomous driving), high latency can lead to serious consequences. These bottlenecks severely restrict the full realization of the IoT's potential and threaten the scalability and efficiency of future networks.

[0057] To overcome the bottlenecks encountered by traditional communication paradigms in addressing the challenges of the large-scale Internet of Things (IoT), semantic communication has emerged as a revolutionary communication concept, demonstrating enormous potential. It represents a fundamental paradigm shift: unlike traditional bit-level transmission—whose core objective is to ensure that the bit stream sent by the sender is accurately and error-freely recovered at the receiver—semantic communication focuses on conveying the sender's "intended meaning." This means that the effectiveness of communication no longer depends solely on the fidelity of bit transmission, but rather on the receiver's ability to correctly understand and reconstruct the core semantics or intent carried by the information.

[0058] Semantic communication systems achieve deep compression and refinement of raw data by intelligently extracting and identifying the most critical and valuable semantic information in the information stream and prioritizing or transmitting only this information. This approach fundamentally reshapes user traffic needs—significantly reducing the amount of data that needs to be transmitted. Therefore, semantic communication exhibits superior dynamic adaptability, enabling it to more effectively cope with complex and ever-changing traffic scenarios (such as bursty data reporting from IoT devices, network congestion, and fluctuating channel conditions). It provides a promising new path to address the spectrum pressure and latency challenges brought by large-scale IoT, laying the foundation for more intelligent, efficient, and future-ready communication networks.

[0059] Based on this, this invention proposes a semantic non-orthogonal multiple access (NOMA) wireless communication method based on generative artificial intelligence (GAI). In this wireless network scenario, multiple semantic users (SUs) extract and transmit cross-modal semantic features, which are recovered by the GAI model at the access point (AP). This method not only enhances high-level semantic understanding by extracting textual features but also utilizes visual features to preserve low-level details; both work together to improve semantic recovery performance. It is important to note that differences in semantic feature size can lead to different traffic demands from different semantic users. To address this, we employ non-orthogonal multiple access technology, which allows multiple users to simultaneously share the same time-frequency resources, achieving signal superposition transmission through power domain multiplexing. Specifically, in this invention, a matching decoding order is allocated according to each user's transmission requirements. Therefore, the semantic feature selection strategy simultaneously affects semantic fidelity and NOMA transmission delay. We maximize the weighted performance of semantic recovery accuracy and transmission delay by jointly optimizing the semantic feature selection strategy, NOMA decoding order, and AP receive beamforming. This optimization problem is a high-dimensional non-convex problem with coupled multivariate relationships, typically requiring approximate or highly complex iterative optimization. In contrast, this invention innovatively proposes an importance-matching PPO algorithm (IM-PPO). Based on the availability of model information, the control variables are decomposed into two parts: a model-based optimization method is used to address the transfer of control variables, while model-free proximal policy optimization (PPO) is used to optimize the semantic feature selection strategy. Furthermore, this invention designs a novel cross-modal matching method to quantify the importance of semantic features, thereby reducing the search space and accelerating the PPO learning process.

[0060] Reference Figure 1 This invention provides a semantic communication method based on generative artificial intelligence, which includes the following steps:

[0061] S100: Based on the bootstrap language-image pre-training model and the unified perception parsing network model, and by introducing the Grad-CAM method, the semantic features of the original image are filtered to obtain refined semantic features.

[0062] Specifically, the original image is input into an image encoder to extract high-level visual features and guide the decoder to generate corresponding semantic feature elements. The semantic feature elements include text feature elements and visual feature elements. The gradient of each semantic feature element relative to the high-level visual features is calculated, the contribution of the semantic feature element to the corresponding image region is quantified, and a heatmap set is generated. A correlation matrix and a contribution matrix are constructed, and combined with the heatmap set, the importance score of the semantic feature elements is obtained. A predefined threshold is introduced, and the semantic feature elements are filtered according to their importance scores to obtain refined semantic features.

[0063] In this embodiment, to achieve reliable semantic communication, the present invention employs both textual and visual features as semantic information. Specifically, a Unified Perceptual Parsing Network (UperNet) and a Bootstrap Language-Image Pre-trained Model (BLIP) are used to extract visual and textual features, respectively. Textual features provide human-understandable intents, such as image descriptions, which convey highly compressed meanings and reflect high-level objectives. In contrast, visual features capture fine-grained structural semantics that are difficult to describe using only text. We utilize semantic segmentation maps as visual features, which highlight object-level regions and preserve task-relevant semantic structures.

[0064] The initially extracted semantic information contains comprehensive semantic features. However, some of these features are redundant and cause additional transmission latency. To address this issue, our goal is to select and transmit only the semantically richest features to improve the transmission efficiency of Non-Orthogonal Multiple Access (NOMA) while maintaining semantic fidelity. A straightforward approach is to employ Deep Reinforcement Learning (DRL) to select the most informative semantic features from the original feature space. However, as the feature space increases, this leads to significant training overhead and increases the risk of the DRL algorithm converging to local optima. This is primarily because model-free DRL does not utilize information from the underlying model, and therefore its performance largely depends on the quality of experience gathered through interaction with the environment. To address this challenge, we propose a cross-model matching method that introduces model knowledge into the DRL framework. In this way, the DRL agent can reduce its reliance on trial-and-error exploration, thereby significantly improving learning efficiency.

[0065] We utilize a pre-trained BLIP model to generate text features. However, the BLIP model may generate redundant text features. To reduce redundancy, we use the Grad-CAM method to identify the attribution of each text feature element to the original image and select the text features with the most information content accordingly.

[0066] like Figure 5 As shown, the original image is first processed by an image encoder to extract high-level visual features. , ,in This represents a set of different image regions. These features capture the semantic information of each region in the original image and guide the text decoder to generate relevant text features. We compute each text feature element. Compared to The gradient of , thus quantizing the first The text feature element pair of the first Contribution of each image region As shown below:

[0067]

[0068] Therefore, we generate a set of heatmaps. This set highlights the important regions in the original image that contribute to the generation of the corresponding text features, as follows:

[0069]

[0070] Given a set of heatmaps The importance of textual feature elements can be assessed by exploring the correlations between them and their contribution to image regions. and This represents a combination of extracted text and visual features. The specific method is as follows:

[0071] 1) Relevance: Relevance refers to the degree of mutual influence between two elements. For example, ... Figure 5 As shown, the heatmaps for the text features [woman] and [jacket] significantly overlap, indicating a strong correlation between them. This is reasonable, as women wear jackets. However, the overlap between [jacket] and [hill] is small, indicating a weaker correlation. Therefore, we can design an association matrix. , This is used to represent this relationship.

[0072]

[0073] 2) Contribution: The contribution reflects the degree to which each semantic feature contributes to the original image. The activated regions in the heatmap reflect the spatial influence of the corresponding text features on the original image. Larger activated regions indicate a more significant contribution of the associated text features to the original image. Therefore, we define a contribution matrix. as follows:

[0074]

[0075] To assess the importance of text features, we define an importance score by comprehensively considering the impact of contribution and relevance, as follows:

[0076]

[0077] High The textual feature elements of the value are considered to have a stronger semantic influence during the generation process. Therefore, we introduce a predefined threshold. and only retain those that meet the requirements. The features are used to filter out those with low semantic importance. A similar method can be used to perform the same cropping operation on visual information.

[0078] S200: Based on the importance matching PPO algorithm, semantic feature selection is performed on the refined semantic features to obtain the selected semantic features;

[0079] Specifically, an importance-matching PPO algorithm is constructed, comprising a model-free PPO module and a model-based optimization module. Based on the model-free PPO module, a state space, an action space, and a reward function are defined. The state space includes the channel conditions and non-orthogonal multiple access (NMO) decoding order of all semantic user equipment. The action space includes semantic feature selection strategies, receive beamforming strategies, and NMO decoding order. Given actions from the model-free PPO module, the model-based optimization module optimizes the actions and outputs the optimized actions. The actions from the model-free PPO module and the optimized actions are merged and executed in the wireless network to obtain the next state and feedback reward. The importance-matching PPO algorithm is cyclically executed based on the next state and feedback reward to perform semantic feature selection on the refined semantic features, resulting in the selected semantic features.

[0080] In this embodiment, after importance-aware semantic feature refinement, the selected cross-modal features exhibit higher semantic information density. However, we do not need to transmit all of these features for the following two reasons:

[0081] 1) Uncertainty in transmission capacity: Due to the dynamic changes in the transmission capacity of semantic user equipment and the limitations of latency requirements, it may be impossible to transmit all filtered semantic features. Therefore, further selection of semantic features is needed to effectively reduce traffic demand and ensure communication reliability.

[0082] 2) Redundancy of cross-modal features: The semantic information conveyed by textual and visual features is usually complementary, but there may also be some overlap. For example, if a textual feature has fully expressed a specific semantic concept, the visual feature associated with it may only provide redundant information. Therefore, identifying and eliminating such cross-modal redundancy can further improve transmission efficiency.

[0083] Therefore, we adopt the Proximal Policy Optimization (PPO) method to learn semantic feature selection strategies, because its advantage estimation and pruning mechanisms can effectively stabilize training and improve learning efficiency.

[0084] like Figure 6 As shown, the Importance Matching PPO algorithm (IM-PPO) integrates a model-based optimization module and a model-free PPO module. This PPO framework consists of tuples (state space). Action space reward function Definition.

[0085] state : Contains channel conditions and non-orthogonal multiple access (NOMA) decoding order for all semantic user equipment (SU).

[0086] Action space decomposition: Based on the availability of information from the analytical model, all control variables are decomposed into two action spaces:

[0087] 1) : Represents semantic feature selection strategy .

[0088] 2) Includes receive beamforming strategy Non-orthogonal multiple access (NOMA) decoding order .

[0089] Before the PPO agent selects semantic features, we perform a cross-modal matching method to identify features with high semantic importance. This importance-aware feature pruning effectively reduces the action space. This adds a dimension and improves exploration efficiency.

[0090] Based on the transmission capabilities of the Semantic User Equipment (SU), the PPO selects semantic features that can carry more semantic information while meeting its traffic requirements. Model collaboration process:

[0091] 1) Given an action from the PPO module .

[0092] 2) Model-based optimization module optimizes actions This enables efficient non-orthogonal multiple access (NOMA) transmission.

[0093] 3) Merge actions from two modules .

[0094] 4) Merge action Execute on the wireless network.

[0095] 5) Obtain the next status and feedback reward .

[0096] 6) This reward is used to update the PPO strategy during the next round of learning.

[0097] To comprehensively consider both semantic fidelity and transmission efficiency, we define the reward function as follows:

[0098]

[0099] in The penalty function is defined as a sufficiently large overhead to penalize the agent for breaking constraints. We utilize the Learned Perceptual Image Patch Similarity (LPIPS) metric to evaluate semantic accuracy.

[0100] S300: By using a non-orthogonal multiple access transmission method, the receiving beamforming and NOMA decoding order are jointly solved, and the selected semantic features are wirelessly transmitted to the base station.

[0101] Specifically, a receive beamforming strategy is defined. Based on the non-orthogonal multiple access (NOMA) decoding order, continuous interference cancellation is used to decode the superimposed signals. The problem is modeled as a standard generalized eigenvalue problem. By solving for the eigenvector corresponding to the largest eigenvalue, the optimal receive beamforming vector is obtained. Based on the optimal receive beamforming vector, auxiliary variables and a quadratic transformation are introduced to approximate SINR, transforming it into a convex optimization problem. The discrete decoding order constraint is linearized using a first-order Taylor expansion, and iterative solutions are performed to determine the optimal NOMA decoding order, matching the semantic traffic requirements of each user equipment. According to the optimal NOMA decoding order, the semantic traffic requirements of each user equipment are matched, and the selected semantic features are wirelessly transmitted to the base station.

[0102] In this embodiment, the access point (AP) equipped with Z receiving antennas can adjust its receiving beamforming strategy to enhance the difference in received signal power among semantic user equipments (SUs), thereby improving the transmission efficiency of non-orthogonal multiple access (NOMA). The set of users is represented by the receive beamforming strategy defined as follows: ,in This represents the receive beamforming weight assigned to SU-k. NOMA allows multiple SUs to access the radio channel simultaneously. Based on the set NOMA decoding order, the receiver uses successive interference cancellation (SIC) to decode the superimposed signals. To characterize the NOMA decoding order, we introduce a binary variable. , This indicates that SU-k is decoded preferentially over SU-j (i.e., when decoding the signal of SU-k, the signal of SU-j is regarded as interference).

[0103] Therefore, each SU must satisfy the following NOMA decoding constraints:

[0104]

[0105]

[0106] in Indicates the transmission priority of SU and A sufficiently large constant is used to guarantee acyclic decoding order. This indicates the wireless channel from SU-k to AP. This represents the transmission power. Therefore, the transmission rate can be expressed as...

[0107]

[0108] in, Indicates bandwidth This represents the noise spectral density. To ensure the transmission of semantic information, each SU needs to satisfy the following condition. Therefore, all transmission delays It can be represented as:

[0109]

[0110] It is important to note that semantic feature selection strategies reshape traffic demands by controlling the size of semantic features, while the receive beamforming and NOMA decoding order at the access point affect transmission capacity by reconstructing channel and interference conditions. Therefore, our goal is to synergistically optimize these semantic transmission control mechanisms to improve semantic fidelity and transmission efficiency.

[0111] By employing a model-driven optimization method, the receiving beamforming and NOMA decoding order are jointly solved, providing an efficient transmission control basis for the semantic feature selection of PPO.

[0112] 1) Optimization of receiver beamforming:

[0113] Objective: To improve the signal-to-interference-plus-noise ratio (SINR) of each unit and enhance the flexibility of NOMA decoding.

[0114] Method: The problem is modeled as a standard generalized eigenvalue problem. By solving for the eigenvector corresponding to the largest eigenvalue, the optimal receiving beamforming vector is obtained.

[0115]

[0116] in and Let represent the covariance matrices of the signal and the interference plus noise, respectively. This represents the eigenvector corresponding to the largest eigenvalue.

[0117] Simplification: Adopt the "worst-case" decoding order to avoid frequent beamforming updates.

[0118] 2) NOMA decoding order optimization:

[0119] Objective: To minimize overall transmission latency while meeting the traffic requirements of each unit (SU). .

[0120] Method: Introduce auxiliary variables The problem is transformed into a convex optimization problem by using a quadratic transformation and approximating SINR. The discrete decoding sequence constraint is linearized using a first-order Taylor expansion, and the solution is obtained iteratively.

[0121] Result: Efficiently determined the optimal NOMA decoding order. Match the semantic traffic requirements of each SU.

[0122] S400, based on a generative artificial intelligence semantic restoration method, performs restoration processing on the selected semantic features and outputs the restored image.

[0123] Specifically, semantic features are used as conditions to guide the denoising process. The selected semantic features are restored through a multi-step denoising process to obtain a preliminary restored image. Perceptual similarity is measured by comparing the differences between the images in the deep feature representation space. The LPIPS value between the original image and the restored image is calculated to evaluate the semantic restoration performance, and the restored image is output.

[0124] In this embodiment, the access point (AP) uses a pre-trained Stable Diffusion Model (SDM) as generative AI. After receiving the semantic features after channel decoding, it gradually generates a high-quality reconstruction of the original image from pure Gaussian noise through a multi-step denoising process. Semantic features (visual + text) are used as conditional guidance in the denoising process. Representing feature information, the recovery process can be expressed as:

[0125]

[0126] in These are control parameters. They are used in feature filtering. Guided by the denoising network Predict the noise signal and minimize its deviation from the actual noise.

[0127] During the denoising process, feature selection The encoded high-dimensional features are gradually incorporated into the model, guiding the generation of output data. This highlights a key point: selecting features with richer semantic information can improve the quality of the reconstructed image. The semantic consistency with the original data is better. We utilize the Learned Perceptual Image Patch Similarity (LPIPS) metric to evaluate semantic accuracy. This metric measures perceptual similarity by comparing the differences between images in the deep feature representation space. We compute the semantic similarity of the original image... With image restoration The semantic recovery performance is evaluated using the LPIPS value between the two values, and its expression is:

[0128]

[0129] in and The feature extractor and the corresponding evaluation network are represented by the first... Layer weights. The smaller this metric, the higher the semantic accuracy of the recovered data.

[0130] In summary, such as Figure 4 As shown, the first part of this embodiment of the invention is the process of semantic information filtering by the user end based on feature extraction and reinforcement learning of neural networks, followed by wireless transmission; the second part is the process of the base station receiving the data and restoring the semantic information based on generative AI to obtain the transmitted data.

[0131] like Figure 3 As shown, this invention considers a generative artificial intelligence (GAI)-assisted semantic nonorthogonal multiple access (NOMA) wireless network. The network comprises K semantic users (SUs) holding image data that needs to be transmitted to an access point (AP). To improve transmission efficiency, the semantic users extract semantic information from the original image and transmit it using NOMA technology. The access point uses a pre-trained GAI model to reconstruct the original image from the received semantic information. While richer semantic information improves reconstruction accuracy, it also increases transmission overhead due to the increased data volume. Furthermore, different semantic users have different levels of semantic features, requiring an adapted NOMA decoding order to meet their respective traffic demands. Therefore, semantic control and transmission control are inherently coupled and need to be jointly adjusted to improve overall system performance.

[0132] This invention utilizes generative artificial intelligence (GAI) to assist a semantic NOMA network, achieving semantic information recovery without retraining and adapting to dynamic environments. Simultaneously, it designs an importance-matching PPO algorithm (IM-PPO), combining model-based methods with model-free PPO algorithms to balance efficiency and adaptability. Furthermore, this invention innovatively develops a cross-modal matching method that quantifies and prunes low-value semantic features, narrowing the search space, accelerating PPO learning, and ultimately jointly optimizing semantic recovery accuracy and transmission latency.

[0133] Therefore, the distinguishing features of the embodiments of the present invention compared to the prior art include:

[0134] 1) Generative Artificial Intelligence-Assisted Semantic Non-Orthogonal Multiple Access Transmission Method: This invention proposes a generative artificial intelligence (GAI)-assisted semantic non-orthogonal multiple access (NOMA) transmission framework. Under this framework, NOMA allocates transmission priorities to semantic user equipments (SUs) based on the scale of their semantic features; simultaneously, the quality of semantic feature selection constrains the effectiveness of semantic recovery. Therefore, we propose a Joint Transmission and Semantic Control (JTSC) scheme, which simultaneously improves semantic fidelity and transmission delay performance by jointly optimizing the semantic feature selection strategy, the NOMA decoding order, and the access point's (AP) receive beamforming strategy.

[0135] 2) Semantic Feature Pruning Technique Based on Cross-Modal Matching: To improve feature selection efficiency, this invention proposes a cross-modal matching method for pruning the candidate semantic feature space. Specifically, this method assesses the importance of text features by visualizing their attribution in the original image and quantifies the importance of visual features based on the correlation between visual and text features. By filtering out low-importance features, the dimensionality of the feature space is effectively reduced, thereby significantly improving the efficiency of the subsequent feature selection process.

[0136] 3) Importance-Aware and Model-Driven Proximal Policy Optimization Framework: This invention proposes an importance-matching PPO algorithm. Based on the availability of model information, this algorithm decouples the coupled control variables into two parts. The model-based optimization method adaptively adjusts the transmission control variables; simultaneously, model-free proximal policy optimization (PPO) selects semantic features from the pruned search space.

[0137] The embodiments of the present invention have the following advantages compared with the prior art:

[0138] 1) Enhanced dynamic adaptability:

[0139] Existing technologies mostly rely on static knowledge bases or single deep models, which are difficult to adapt to dynamic network environments and have high fine-tuning costs. This invention uses pre-trained GAI (Stable Diffusion) for semantic recovery, which does not require retraining for network changes and can adapt to different scenarios simply by prompting, with good real-time performance.

[0140] 2) Semantic-transport cross-layer joint optimization:

[0141] Traditional methods treat semantic control (feature selection) and transmission control (power, scheduling, NOMA decoding order) separately. This invention proposes the JTSC framework, which combines the selection of semantic features with the NOMA decoding order and receive beamforming into the same objective function for joint optimization, significantly reducing end-to-end latency and improving recovery accuracy.

[0142] 3) Efficient learning driven by model-data hybrid approach:

[0143] Pure DRL methods converge slowly in high-dimensional action spaces and easily ignore explicit information such as channel / interference. This invention designs an importance-matching PPO algorithm, which uses "importance-awareness" cross-modal matching to first prune low-value semantic features and reduce the action space. It assigns modelable transmission variables (beamforming, decoding order) to convex optimization and leaves black-box semantic variables to PPO, achieving the complementary advantages of "model knowledge + deep reinforcement learning" and significantly improving training speed.

[0144] Finally, the method designed in this invention was verified in simulation experiments. We used the proposed importance-matching PPO algorithm to evaluate the performance of the Joint Transmission and Semantic Control (JTSC) scheme in generative artificial intelligence (GAI)-assisted semantic nonorthogonal multiple access (NOMA) transmission. On the access point (AP) side, the core architecture of the Stable Diffusion Model (SDM) adopts Stable Diffusion v1-5, and the original images used for semantic transmission are from the COCO-2017 dataset, which is widely used in computer vision and semantic communication research. The specific simulation experimental parameters are shown in Table 1:

[0145] Table 1 Simulation parameter settings

[0146] To verify the convergence of the importance-aware and model-driven proximal policy optimization framework, we evaluate its learning performance and compare it with two benchmark methods: the M-PPO method and the basic PPO method. Specifically, the M-PPO method integrates a model-based optimization module but does not apply cross-modal matching for semantic feature trimming. The basic PPO method employs a model-free proximal policy optimization approach to directly optimize all control variables without utilizing any model information.

[0147] Figure 7 (a) of the paper investigated the convergence performance of different algorithms. It was observed that the proposed IM-PPO achieved the fastest convergence, requiring approximately 2,500 rounds, compared to approximately 6,000 rounds for M-PPO and over 7,500 rounds for Plain-PPO. These results demonstrate that incorporating model information effectively reduces inefficient exploration, thereby guiding the deep reinforcement learning agent towards the optimal solution more efficiently. Furthermore, the cross-modal matching method significantly reduces the action space of the PPO agent, further accelerating its learning efficiency. Figure 4In the left plot of subplot a, the blue dashed line represents the average reward after Plain-PPO convergence. We observe that both IM-PPO and M-PPO achieve higher final rewards, suggesting that integrating model information helps avoid local optima during the learning process. We also investigated the stability of IM-PPO by varying its learning rate, such as... Figure 7 As shown in (b) of the figure. The results show that IM-PPO converges stably and uniformly under various settings. This robustness is due to the optimization module and cross-modal matching, which together reduce the action space and mitigate the sensitivity to hyperparameter tuning.

[0148] In addition to overall reward performance, we further investigated the learning performance of IM-PPO and M-PPO on individual metrics (i.e., LPIPS and transmission delay). Figure 7 As shown in (c), both methods converged successfully on each metric. Interestingly, although IM-PPO pruned some semantic features through cross-modal matching, it still achieved recovery accuracy comparable to M-PPO. This verifies that the proposed cross-modal matching method effectively preserves the most informative semantic features. To further analyze the interaction between semantic control and transmission capability, we examined the semantic feature selection ratio for each SU, as shown in (c). Figure 7 As shown in (d) of the diagram, we consider a fair scenario where each SU transmits the same image. After implementing the IM-PPO framework, the NOMA decoding order is assigned as SU-1, SU-2, and SU-3. We observe that SUs with earlier decoding orders tend to select fewer semantic features to accommodate their transmission capabilities. This validates the effectiveness of IM-PPO in jointly optimizing semantic control and transmission control.

[0149] Then, we discuss the system performance improvements achieved by our proposed JTSC scheme. To highlight its advantages, we compare JTSC with three benchmark schemes: the all-select scheme, the random-select scheme, and the position-decoding scheme. In the all-select scheme, all semantic features are transmitted. The random-select scheme transmits a randomly selected subset of semantic features, regardless of their importance. In the position-decoding scheme, the NOMA decoding order is determined solely based on the distance between the SU and AP, without joint optimization with the semantic feature selection strategy. Figure 8 (a) evaluates system performance under different transmission capabilities, where higher bandwidth indicates stronger transmission capability per unit (SU). We define weighted LL as a comprehensive metric to reflect LPIPS and transmission delay performance; a lower value indicates better overall performance. Figure 8In the left-hand plot of (a), the weighted LL performance of all schemes improves with increasing bandwidth. The JTSC scheme achieves the best weighted LL performance. This is because the JTSC scheme adaptively aligns the semantic feature selection strategy with the transmission capability of the SU. Conversely, the random selection scheme randomly selects semantic features, leading to the loss of key semantic information. Although the full selection scheme ensures high semantic fidelity by transmitting all features, it introduces significant transmission latency, resulting in suboptimal overall performance. The positional decoding scheme fails to adjust the NOMA decoding order of the SU according to the current semantic feature selection strategy, also leading to increased transmission latency. As the transmission capability of the SU improves, the performance gap between the JTSC scheme and the full selection and positional decoding schemes narrows due to the alleviation of transmission bottlenecks. Figure 8 The right-hand plot of (a) evaluates the overall latency and individual LPIPS. With the increase in SU transmission capacity, each SU transmits richer semantic features, thereby improving semantic recovery accuracy. Simultaneously, latency is reduced accordingly due to the improved transmission capacity. Figure 8 In section (b), we further investigated LPIPS and latency performance under different numbers of SUs. As the number of SUs increases, access contention among users becomes more intense. Figure 8 In the left-hand plot of (b), it is observed that the full selection scheme consistently achieves the highest semantic recovery accuracy across different numbers of SUs. However, as... Figure 8 As shown in the right-hand figure (b), this comes at the cost of a significant increase in transmission delay, making it impractical in real-time transmission scenarios. Conversely, the JTSC scheme balances LPIPS and delay performance, ensuring that both metrics decrease moderately even as access contention intensifies. This complementary design allows JTSC to effectively adapt to changing transmission conditions, thereby enhancing overall robustness. It is noteworthy that, due to the lack of model information guidance, the LPIPS performance of the random selection scheme does not show a significant trend with increasing SU numbers. To further investigate the inter-SU impact, we gradually increase the transmit power of SU-3 and compare the semantic feature selection ratios of the three SUs, such as... Figure 8As shown in (c), when the transmit power of SU-3 is low (below 22.5 dBm), the optimized NOMA decoding order is SU-2, SU-1, SU-3. In this case, as the transmit power of SU-3 increases, its transmission capability also improves, allowing it to select more semantic features to enhance recovery accuracy. Meanwhile, the semantic selection strategy of SU-2 remains unchanged because the increased interference from SU-3 forces it to select only the most necessary semantic features to ensure reliable semantic recovery. When the transmit power of SU-3 exceeds 25 dBm, its interference to the previously decoded SU-1 and SU-2 becomes significant, rendering the current NOMA decoding order inapplicable. Therefore, the JTSC scheme adaptively reallocates the NOMA decoding order to SU-3, SU-2, SU-1. Notably, after moving to the first decoding position, the transmission capability of SU-3 decreases significantly, forcing it to reduce the number of selected semantic features to maintain effective adaptation. As the transmit power of SU-3 continues to increase, its transmission capability gradually improves again. Simultaneously, SU-2 begins to increase its semantic feature selection ratio to improve recovery accuracy. This is because the increased transmission capacity of SU-3 effectively relaxed the overall latency requirements, allowing SU-2 to transmit more semantic features without causing excessive latency. We observed that the semantic feature selection ratio of SU-1 remained constant throughout the process, as its optimal semantic selection strategy always remained consistent with its transmission capacity.

[0150] exist Figure 9 In our study, we visualized the impact of semantic feature selection on semantic reconstruction accuracy. The "Select All" approach chose all semantic features, while the JTSC approach adaptively selected only the features with the most information content. Interestingly, as the number of denoising steps increased, the JTSC approach gradually surpassed the "Select All" approach in semantic reconstruction accuracy, achieving more stable performance after more than five denoising steps. The principle behind this result is that transmitting too many semantic features may introduce redundant or even conflicting information during the reconstruction process, thereby weakening its ability to accurately capture and recover key semantic details. Conversely, transmitting fewer but more information-rich semantic features allows the reconstruction process to focus on the most essential information, resulting in more accurate and faithful semantic reconstruction. For example, as shown by the red circle in the reconstructed image, the JTSC approach correctly reconstructed the arm in the image, while the "Select All" approach generated an inaccurate arm due to redundant semantic information in that region.

[0151] Reference Figure 2 A semantic communication system based on generative artificial intelligence, comprising:

[0152] The first module 201 is used to filter the semantic features of the original image based on the bootstrap language-image pre-trained model and the unified perception parsing network model, and introduces the Grad-CAM method to obtain the refined semantic features.

[0153] The second module 202 is used to perform semantic feature selection on the refined semantic features based on the importance matching PPO algorithm, and obtain the selected semantic features.

[0154] The third module 203 is used to jointly solve the receiving beamforming and NOMA decoding order through a non-orthogonal multiple access transmission method, and then wirelessly transmit the selected semantic features to the base station.

[0155] The fourth module 204 is used to perform recovery processing on the selected semantic features based on the generative artificial intelligence semantic recovery method, and output the recovered image.

[0156] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0157] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A semantic communication method based on generative artificial intelligence, characterized in that, Includes the following steps: Based on the bootstrap language-image pre-trained model and the unified perception parsing network model, and by introducing the Grad-CAM method, the semantic features of the original image are filtered to obtain refined semantic features. An importance-matching PPO algorithm is constructed, which includes a model-free PPO module and a model-based optimization module. Based on the model-free PPO module, a state space, an action space, and a reward function are defined. The state space includes the channel conditions of all semantic user equipment and the non-orthogonal multiple access decoding order. The action space includes the semantic feature selection strategy, the receive beamforming strategy, and the non-orthogonal multiple access decoding order. Given an action from the modelless PPO module, optimize it using the model-based optimization module, and output the optimized action. Merge the actions of the model-free PPO module with the optimized actions, and execute the merged actions in the wireless network to obtain the next state and feedback reward; The importance matching PPO algorithm is executed in a loop based on the next state and feedback reward to select semantic features from the refined semantic features. A receive beamforming strategy is defined. Based on the non-orthogonal multiple access decoding order, continuous interference cancellation is used to decode the superimposed signal. The problem is modeled as a standard generalized eigenvalue problem. The optimal receive beamforming vector is obtained by solving the eigenvector corresponding to the maximum eigenvalue. Based on the optimal receive beamforming vector, auxiliary variables and a quadratic transformation are introduced to approximate SINR and transform it into a convex optimization problem. The discrete decoding order constraint is linearized using a first-order Taylor expansion, and iterative solutions are performed to determine the optimal non-orthogonal multiple access decoding order to match the semantic traffic requirements of each user equipment. Based on the optimal non-orthogonal multiple access decoding order, the semantic traffic requirements of each user device are matched, and the selected semantic features are wirelessly transmitted to the base station. Based on the generative artificial intelligence semantic restoration method, the selected semantic features are restored and the restored image is output.

2. The semantic communication method based on generative artificial intelligence according to claim 1, characterized in that, The step of using a bootstrap language-image pre-trained model and a unified perceptual parsing network model, and introducing the Grad-CAM method to filter the semantic features of the original image to obtain refined semantic features, specifically includes: The original image is input into the image encoder to extract high-level visual features and guide the decoder to generate corresponding semantic feature elements, which include text feature elements and visual feature elements. Calculate the gradient of each semantic feature element relative to the high-level visual features, quantify the contribution of the semantic feature element to the corresponding image region, and generate a set of heatmaps. Construct a correlation matrix and a contribution matrix, and combine them with a heatmap set to obtain the importance scores of semantic feature elements; By introducing a predefined threshold, semantic feature elements are filtered based on their importance scores to obtain refined semantic features.

3. The semantic communication method based on generative artificial intelligence according to claim 2, characterized in that, The specific expression for the reward function is as follows: In the above formula, Represents the penalty function. Represents the reward function, Indicates the actions after the merger. This refers to the perceived similarity metric used to measure user... The accuracy of semantic information recovery The weighting coefficients are used to balance semantic recovery performance and transmission latency performance. Indicates transmission delay. Indicates user index, This indicates the number of users.

4. The semantic communication method based on generative artificial intelligence according to claim 3, characterized in that, The generative artificial intelligence-based semantic restoration method, in its step of restoring selected semantic features and outputting the restored image, specifically includes: Semantic features are used as conditions to guide the denoising process. The selected semantic features are restored through a multi-step denoising process to obtain a preliminary restored image. Perceptual similarity is measured by comparing the differences between images in the deep feature representation space. The semantic restoration performance is evaluated by calculating the LPIPS value between the original image and the restored image, and the restored image is output.

5. The semantic communication method based on generative artificial intelligence according to claim 4, characterized in that, The specific expression for restoring the selected semantic features is as follows: In the above formula, Indicates control parameters, Representing feature information, This represents a denoising network. and Indicates user In the noise reduction process, the first and The noise situation of the step, Indicates the speed control coefficient. The index representing the number of denoising steps.

6. The semantic communication method based on generative artificial intelligence according to claim 5, characterized in that, The specific expression for calculating the LPIPS value is as follows: In the above formula, Indicates the LPIPS value. and The feature extractor and the corresponding evaluation network are represented by the first... Layer weights, Represents the original image. Indicates image restoration. This indicates the total number of layers in the network being evaluated.

7. A semantic communication system based on generative artificial intelligence, characterized in that, Includes the following modules: The first module is used to filter the semantic features of the original image based on the bootstrap language-image pre-trained model and the unified perception parsing network model, and introduces the Grad-CAM method to obtain the refined semantic features. The second module is used to construct an importance-matching PPO algorithm, which includes a model-free PPO module and a model-based optimization module. Based on the model-free PPO module, a state space, an action space, and a reward function are defined. The state space includes the channel conditions of all semantic user equipment and the non-orthogonal multiple access (NOMA) decoding order. The action space includes the semantic feature selection strategy, the receive beamforming strategy, and the non-orthogonal multiple access decoding order. Given an action from the modelless PPO module, optimize it using the model-based optimization module, and output the optimized action. Merge the actions of the model-free PPO module with the optimized actions, and execute the merged actions in the wireless network to obtain the next state and feedback reward; The importance matching PPO algorithm is executed in a loop based on the next state and feedback reward to select semantic features from the refined semantic features. The third module is used to define the receiving beamforming strategy. Based on the non-orthogonal multiple access decoding order, continuous interference cancellation is used to decode the superimposed signal. The problem is modeled as a standard generalized eigenvalue problem. The optimal receiving beamforming vector is obtained by solving the eigenvector corresponding to the maximum eigenvalue. Based on the optimal receive beamforming vector, auxiliary variables and a quadratic transformation are introduced to approximate SINR and transform it into a convex optimization problem. The discrete decoding order constraint is linearized using a first-order Taylor expansion, and iterative solutions are performed to determine the optimal non-orthogonal multiple access decoding order to match the semantic traffic requirements of each user equipment. Based on the optimal non-orthogonal multiple access decoding order, the semantic traffic requirements of each user device are matched, and the selected semantic features are wirelessly transmitted to the base station. The fourth module is used to perform recovery processing on the selected semantic features based on generative artificial intelligence semantic recovery methods, and output the recovered image.

Images