Semantic communication optimization method and system based on sparse reward re-optimization

Abstract

The invention discloses a semantic communication optimization method and system based on sparse reward re-optimization, and belongs to the technical field of wireless communication, and the method comprises the steps: constructing a semantic communication optimization model; establishing a reward prediction model, and predicting a reward value at a future moment in reinforcement learning; reconstructing a reward sequence based on a reward value prediction result; and performing reinforcement learning on the semantic communication channel based on the reconstructed reward sequence to realize transmission strategy updating. According to the method, the problems of sparse rewards and unstable training in semantic communication reinforcement learning can be effectively relieved under the conditions of complex channels and limited feedback, joint optimization of a semantic layer and a channel layer is realized, and an intelligent communication system is promoted to develop towards an efficient, self-adaptive and interpretable semantic transmission direction while low delay and high reliability are ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wireless communication, in particular to a semantic communication optimization method and system based on sparse reward re-optimization. BACKGROUND

[0002] In the face of the continuous outbreak of data flood in the intelligent era, global mobile communication networks are entering a new stage characterized by ultra-large-scale data transmission and real-time semantic interaction. As an innovative paradigm that breaks through the Shannon communication limit, semantic communication extracts and transmits the most relevant semantic information to the task target, fundamentally reduces the amount of redundant bit transmission, and significantly improves the spectrum utilization and energy efficiency ratio. It has become a key technology direction to alleviate the shortage of spectrum resources and realize the efficient evolution of intelligent communication systems.

[0003] Existing semantic communication system research can be mainly divided into four categories: DL-based SE (Deep Learning-based SE), KB-assisted SE (Knowledge Base-assisted SE), RL-based SE (Reinforcement Learning-based SE) and Semantic-native SE. Among them, DL-based SE realizes the automatic extraction and restoration of semantic information through end-to-end neural network modeling, but has the problems of poor interpretability and limited model generalization ability; KB-assisted SE relies on pre-constructed knowledge base to assist the transmission of task-specific information, but the system deployment is complex and lacks real-time adaptability; Semantic-native SE realizes active semantic modeling through collaborative learning and context reasoning between communication bodies, but is still in the concept verification stage and is difficult to deploy in practice. In contrast, RL-based SE introduces a reinforcement learning mechanism on the basis of DL-based SE, directly guides model training through non-differentiable indicators such as semantic similarity, BLEU, AoI (Age of Information), etc., and can realize more accurate optimization at the semantic level, while also having the ability to adapt to dynamic environments, so it has become an important direction of current semantic communication research.

[0004] Then, in the semantic transmission task, the RL-based SE can only obtain a global reward feedback according to a semantic similarity index (such as BLEU, BERT Score or semantic distance) after decoding the whole information, and cannot perform fine-grained evaluation on the transmission behavior in the middle stage. This delayed and sparse reward signal makes it difficult for the agent to determine which local decision truly promotes semantic understanding and information recovery, resulting in high variance in the learning process and slow training convergence. In typical scenarios such as autonomous driving cooperative perception, remote medical surgery, industrial visual inspection and multi-modal human-computer interaction, the communication system must achieve fast adaptation and stable convergence in a dynamic channel environment. If the reward sparsity problem is not solved, it will directly limit the real-time performance and reliability of semantic communication in these key tasks. Therefore, it is urgent to design an optimization mechanism that can alleviate the sparse reward problem in semantic communication reinforcement learning, convert the global semantic feedback signal into local guidance information that can be used for intermediate timing decision, and realize the landing of an efficient intelligent semantic communication system with ultra-low latency. SUMMARY

[0005] The present application provides a semantic communication optimization method and system based on sparse reward re-optimization, to solve the technical problems of low learning efficiency of the agent, unstable training and difficulty in converging semantic transmission strategies in the existing scheme of implementing semantic communication system optimization through reinforcement learning due to sparse reward feedback.

[0006] To solve the above technical problems, the present application provides the following technical solutions: In one aspect, the present application provides a semantic communication optimization method based on sparse reward re-optimization, comprising: constructing a semantic communication optimization model; wherein the semantic communication optimization model comprises a semantic sending end, a semantic communication channel and a semantic receiving end; the semantic sending end is used for timing modeling of input semantic information to generate encoded information; the semantic communication channel is used for generating a transmission strategy based on a reinforcement learning algorithm to transmit the encoded information; and the semantic receiving end is used for decoding the received encoded information to reconstruct the original semantic information; establishing a reward prediction model to predict the reward value at a future time in the reinforcement learning algorithm; reconstructing a reward sequence based on the prediction result of the reward value at the future time; the semantic communication channel performs reinforcement learning based on the reconstructed reward sequence to realize transmission strategy updating.

[0007] Further, the semantic sending end uses a long short-term memory network to perform timing modeling of the input semantic information.

[0008] Further, the semantic receiving end uses a long short-term memory network to decode the received encoded information.

[0009] Further, the reward function for calculating the reward value in the reinforcement learning algorithm is: R wherein, sim represents the semantic similarity between the reconstructed semantic information and the original semantic information; t represents the transmission delay; D w represents the weight of the semantic similarity; w represents the weight of the transmission delay.

[0010] Further, the input of the reward prediction model is the historical reward sequence and the state transition sequence generated in the execution process of the reinforcement learning algorithm, and the output is the prediction result of the reward value at future time points.

[0011] Further, the reward prediction model is a recurrent neural network structure.

[0012] Further, based on the prediction result of the reward value at the future time point, the reward sequence is reconstructed, including: calculating the actual increase of the reward at each time step in the prediction result of the reward value at the future time point for each historical action; wherein, the calculation formula of the actual increase is: wherein, represents the actual increase of the reward at the i th time step for the j th action; represents the prediction result of the reward at the i th time step after the j th action; t represents the prediction result of the reward at the i th time step after the j th action; when the actual increase of the reward at a certain time step for a historical action is greater than a first preset threshold, the nomination times corresponding to the historical action are increased by 1; when the nomination times corresponding to a historical action are the largest or greater than a second preset threshold, the historical action is recorded as a key action; the reward corresponding to the key action is weighted and distributed forward and backward in time according to the influence range before and after the occurrence of the key action, to form a new reward sequence. t Further, the formula for weighting and distributing the reward corresponding to the key action forward and backward in time according to the influence range before and after the occurrence of the key action is: t

[0013] ​​​​​​​​​​​in, Indicates the weighted allocation of the first... t Rewards for each time step; Indicates the reward corresponding to the key action; and This indicates the preset attenuation coefficient; Indicates the moment when the key action occurs; D Indicates the preset forward-allocated window length; O This indicates the preset window length to be allocated backwards.

[0014] Furthermore, the reinforcement learning based on the reconstructed reward sequence to update the transmission policy includes: Construct an experience replay pool, and store the states, actions, channel observations, decoding results, and reconstructed reward sequences generated by the agent in the semantic communication environment into the experience replay pool in chronological order; At each policy update, a batch of samples is randomly sampled from the experience replay pool; the sampled reward sequence is input into the policy network and the value network, and the network parameters are updated in batches according to the preset reinforcement learning update rules; the iterative optimization process is executed until the preset termination condition is reached, and the optimal transmission policy is output.

[0015] On the other hand, the present invention also provides a semantic communication optimization system based on sparse reward re-optimization, the semantic communication optimization system based on sparse reward re-optimization comprising: The semantic communication optimization model construction module is used to construct a semantic communication optimization model. The semantic communication optimization model includes a semantic transmitter, a semantic communication channel, and a semantic receiver. The semantic transmitter is used to perform temporal modeling on the input semantic information and generate encoded information. The semantic communication channel is used to generate a transmission strategy based on a reinforcement learning algorithm and transmit the encoded information. The semantic receiver is used to decode the received encoded information and reconstruct the original semantic information. The reinforcement learning control module is used for: Establish a reward prediction model to predict the reward value at future time steps in the reinforcement learning algorithm; Reconstruct the reward sequence based on the predicted reward values ​​at future moments; The semantic communication channel uses reinforcement learning based on the reconstructed reward sequence to update the transmission policy.

[0016] In another aspect, the present invention also provides an electronic device comprising a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described method.

[0017] In another aspect, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the above method.

[0018] The beneficial effects of the technical solution provided by this invention include at least the following: This invention constructs an interpretable key action recognition mechanism, making the optimization process of semantic transmission strategies more interpretable rather than a black box. By comparing the temporal incremental changes in predicted rewards, it can automatically identify key actions in the semantic transmission link that lead to a significant increase in semantic similarity and dynamically adjust the reward weights, thus achieving traceability and interpretability of semantic transmission decisions. This mechanism effectively reveals the causal relationship of agent decisions in semantic layer optimization, providing theoretical support for the performance tuning and online deployment of semantic communication systems.

[0019] This invention achieves joint optimization of semantic encoding / decoding and channel transmission, constructing a cross-layer adaptive semantic transmission framework. By introducing a joint optimization strategy into the reinforcement learning control unit, the semantic encoder, channel transmission strategy, and semantic decoder form an end-to-end collaborative learning mechanism. During training, the agent can dynamically adjust the encoding compression rate, channel resource allocation, and decoding mapping function based on the prediction reward, thereby achieving real-time adaptation to changes in channel state at the semantic level.

[0020] This invention proposes a semantic communication reinforcement learning method based on sparse reward re-optimization, which solves the bottleneck of reward sparsity in semantic communication and realizes the decomposability of training signals. By introducing a reward prediction model, continuous future reward estimates are generated in the temporal process of semantic transmission, and a credit allocation mechanism is combined to weight key actions and adjacent actions, so that the global reward can be decomposed into learnable local feedback, fundamentally improving the trainability and convergence efficiency of reinforcement learning in semantic communication scenarios. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of the semantic communication optimization method based on sparse reward re-optimization provided in an embodiment of the present invention; Figure 2 This is a diagram of the source-channel joint optimization semantic communication model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of key action recognition provided in an embodiment of the present invention; Figure 4 This is a system block diagram of the electronic device provided in the embodiments of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0024] First, it should be noted that in the embodiments of the present invention, the words "exemplarily," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplarily" is intended to present the concept in a specific manner. Furthermore, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either one or the other.

[0025] First Embodiment This embodiment provides a semantic communication optimization method based on sparse reward re-optimization. This method can be implemented by an electronic device, and its execution flow is as follows: Figure 1 As shown, it includes the following steps: S1, Construct a semantic communication optimization model; wherein, the semantic communication optimization model includes a semantic sender, a semantic communication channel, and a semantic receiver; the semantic sender is used to perform temporal modeling on the input semantic information and generate encoded information; the semantic communication channel is used to generate a transmission strategy based on a reinforcement learning algorithm and transmit the encoded information; the semantic receiver is used to decode the received encoded information and reconstruct the original semantic information; Specifically, in this embodiment, the semantic sending end is equipped with a semantic encoding neural network, which is used to process the original semantic information. Convert to low-dimensional semantic representation vector This neural network employs a Long Short-Term Memory (LSTM) network or Transformer structure to capture the contextual dependencies of semantic features. The semantic communication channel is modeled as a noisy transmission environment for encoding semantic vectors. For channel transmission, considering channel bandwidth, noise power, and delay constraints, the signal is subject to random disturbances during transmission. The received signal is represented as The semantic receiver is equipped with a semantic decoding neural network, used to decode the received signal. Reconstructing semantic information It also calculates semantic similarity metrics to assess the quality of semantic information transmission.

[0026] The construction process of the semantic communication optimization model is as follows: S11, Initialize the semantic communication environment, including the agent. Communication links Transmission Channel and semantic information source The agent is trained and its strategies are adjusted based on feedback from the environment. S12, Design and initialize the transmission link for semantic information, including modeling the environmental state and considering the noise of the communication channel. ,bandwidth ,Delay Factors such as these are considered, and the communication link is optimized using a neural network model. This yields a joint source-channel optimized semantic communication model, with the overall system model framework as follows: Figure 2 As shown, it includes:

[0027] (1) Semantic information encoding network, which adopts the Long Short-Term Memory (LSTM) model. To the semantic information of the input Perform timing modeling to generate coded information suitable for communication transmission. ; (2) Semantic information transmission network, which is responsible for transmitting the encoded information through the transmission channel. Transmission is performed. The transmission process takes into account the channel bandwidth. and transmission delay And adjust the transmission strategy through reinforcement learning;

[0028] (3) Semantic information decoding network, which employs a decoder Its structure corresponds to that of the coding network, using the LSTM model to process the received signal. Decode the original semantic information to reconstruct it. .

[0029] S13, Based on the above transmission modeling, a reward feedback mechanism, namely a reinforcement learning control unit, is established to evaluate the effectiveness of semantic information transmission. The reward function is... The design takes into account the accuracy of semantic information and the transmission latency, specifically as follows: in, For semantic similarity, Weights for similarity As a weight for latency, This is to reduce transmission latency. Through this reward mechanism, the agent optimizes its information transmission strategy during the learning process.

[0030] S2, Establish a reward prediction model to predict the reward value at future time steps in the reinforcement learning algorithm; Specifically, in this embodiment, the reward prediction network uses historical reward signals... and state transition sequence Modeling is performed to predict reward values ​​at multiple future time points. ,in, , This indicates the time step length for prediction. The reward prediction model is based on a recurrent neural network structure, consisting of an input layer, hidden layers, and an output layer. The prediction process is as follows: the input layer receives past... The state reward sequence at time 1 The hidden layer uses Long Short-Term Memory (LSTM) network units to capture the dependencies between state and reward over time; the output layer predicts the future through a fully connected layer. The sequence of reward values ​​at each time point The prediction process is described by the following formula:

[0031] in, This represents the reward prediction model function, which includes time-step expanded LSTM units and a linear output layer. The prediction module outputs a reward estimate at each time step, forming a continuous reward estimate sequence, which provides time-related reward input for the subsequent credit allocation module.

[0032] S3, Reconstruct the reward sequence based on the predicted reward values ​​at future times; Specifically, in this embodiment, the reward sequence is reconstructed based on the predicted reward values ​​for future times by introducing a credit allocation mechanism. This mechanism identifies key actions using predicted rewards and allocates future rewards to key actions and adjacent states based on their temporal relevance. The specific process is as follows:

[0033] S31, Construct a credit allocation process based on key action recognition, as follows: For the future reward sequence predicted in S2 Establish a "time-action" difference table for the reward at each future moment, such as... Figure 3 As shown, the incremental contribution of each historical action to the future reward is calculated. Specifically, at time step... Above, for all those earlier action The differential prediction reward for this action is calculated using the following formula:

[0034] in, Indicates "in the After each action, the reward "Predicted value" Indicates "in the After the action, for the same reward The difference between the predicted value and the actual value. That is the first Each action is related to the reward. The actual increase in rewards brought about.

[0035] S32, to avoid treating noise as a critical action, when Greater than the preset threshold At that time, the action corresponding to that time step. This is recorded as one "critical action candidate". The nomination process is repeated for all time steps to obtain the nomination count for each action. The action with the highest nomination count or exceeding a set percentage is identified as a critical action, and its occurrence time is recorded as... .

[0036] S33, introduce a credit allocation mechanism to achieve credit allocation based on key action recognition, as detailed below: The final round reward Based on the impact range before and after the key action, a new immediate reward sequence is formed by weighting the rewards forward and backward in time. "Preparatory actions" preceding the key action receive credits that decay with distance, while "dependent actions" following the key action receive forward credits that decrease with distance. Specifically:

[0037] in, The moment when the key action occurs. and These represent the length windows allocated forward and backward, respectively. and The preset attenuation coefficient; S34, credit rewards for all actions. The reward sequence is reconstructed to replace the original sparse reward, thereby providing denser and more discriminative feedback in subsequent semantic communication reinforcement learning training.

[0038] S4, the semantic communication channel uses reinforcement learning based on the reconstructed reward sequence to update the transmission strategy.

[0039] Specifically, in this embodiment, the above-mentioned S4 uses an experience replay mechanism to sample historical interaction trajectories during iterative optimization. Batch training and policy iteration are performed using reconstruction rewards. Adaptive joint optimization of semantic encoding, transmission, and decoding is achieved through experience replay and policy iteration. The specific implementation process is as follows:

[0040] S41 uses an experience playback mechanism to sample historical interaction trajectories.

[0041] Build an experience replay pool This involves analyzing the agent's state, actions, channel observations, decoding results, and reconstructed rewards generated in a semantic communication environment. Samples are stored in the replay pool in chronological order. A batch of samples is randomly sampled from the replay pool each time the policy is updated. To break temporal correlation, the sampled reconstructed reward sequence is input into the policy network and value network, and the network parameters are updated in batches according to the preset reinforcement learning update rules.

[0042] S42, the agent uses the reconstructed reward signal to update its policy, repeating the "sampling-update" process until the cumulative reward or semantic similarity index converges, outputting a stable semantic encoding, transmission, and decoding policy to achieve adaptive optimization of the semantic encoding, transmission, and decoding process; including: Construct a redistribution reward sequence, which is the reward signal sequence adjusted by the credit allocation mechanism. The input is fed into the reinforcement learning training process, replacing the original sparse reward; In the policy optimization process of semantic communication tasks, the agent optimizes its encoding, transmission, and decoding policies based on the redistribution of reward signals, and updates the policy parameters through reinforcement learning algorithms. Its optimization objective can be expressed as: in, As a discount factor, The reward is an immediate reward after credit allocation; By combining semantic similarity metrics with redistribution reward signals, the agent's semantic transmission strategy is optimized, enabling it to effectively improve semantic transmission quality even in low reward density environments. When the semantic similarity improvement rate or the cumulative redistribution reward gain stabilizes within a preset threshold range, the reinforcement learning process is considered to have converged, and the optimal semantic communication strategy is obtained.

[0043] Furthermore, based on the above reward reconstruction strategy, this embodiment proposes a semantic transmission algorithm based on sparse reward re-optimization, the execution process of which is as follows: Initialize the parameters of the semantic communication reinforcement learning network, including the semantic encoder. Semantic decoder and semantic transport strategy network and initialize the reward prediction model. With parameters of the credit allocation module; Construct a semantic transmission environment and clear the experience replay pool, through an environment model. Initialize parameters such as channel noise, bandwidth, and delay, and set the semantic information source. Feedback mechanism with the receiving end; Generate initial state and actions, in the initial state Next, semantically encoded actions are generated based on a random initialization strategy. and through the channel The transmitted signal is received. Perform semantic transmission and calculate immediate reward; the receiving end decodes to obtain... Calculate semantic similarity and combine latency and energy consumption to obtain an instant reward. ; Reward prediction is performed based on a predictive model. The input consists of a historical state sequence and an action sequence, from which the predicted future reward value is obtained. And calculate the predicted increment to identify key actions; The credit allocation operation is performed by assigning credits to key actions and their adjacent actions based on the future reward signals output by the reward prediction model, thereby forming a new reconstructed reward sequence. ; Update the semantic transport policy network parameters by inputting the reconstructed reward sequence into the policy network, optimizing the objective function, and adjusting the parameters based on the gradient update rule. ,in, The learning rate; Repeat the interaction and optimization process, continuously sample new semantic transmission trajectories, and update the reward prediction model, credit allocation module and policy network parameters until the semantic transmission performance (semantic similarity or cumulative reward) converges. The algorithm is trained and the optimal semantic transmission strategy is output. When the number of iterations reaches a preset threshold or the average semantic similarity gain stabilizes, the optimal semantic encoding and transmission strategy network is obtained, realizing efficient semantic communication based on sparse reward re-optimization.

[0044] In summary, this embodiment provides a semantic communication reinforcement learning method based on sparse reward re-optimization, which solves the bottleneck of reward sparsity in semantic communication and realizes the decomposability of training signals. By introducing a reward prediction model, continuous future reward estimates are generated during the temporal process of semantic transmission, and a credit allocation mechanism is combined to weight key actions and adjacent actions, so that the global reward can be decomposed into learnable local feedback, fundamentally improving the trainability and convergence efficiency of reinforcement learning in semantic communication scenarios.

[0045] Second Embodiment This embodiment provides a semantic communication optimization system based on sparse reward re-optimization, including: The semantic communication optimization model construction module is used to construct a semantic communication optimization model. The semantic communication optimization model includes a semantic transmitter, a semantic communication channel, and a semantic receiver. The semantic transmitter is used to perform temporal modeling on the input semantic information and generate encoded information. The semantic communication channel is used to generate a transmission strategy based on a reinforcement learning algorithm and transmit the encoded information. The semantic receiver is used to decode the received encoded information and reconstruct the original semantic information. The reinforcement learning control module is used for: Establish a reward prediction model to predict the reward value at future time steps in the reinforcement learning algorithm; Reconstruct the reward sequence based on the predicted reward values ​​at future moments; The semantic communication channel uses reinforcement learning based on the reconstructed reward sequence to update the transmission policy.

[0046] It should be noted that the semantic communication optimization system based on sparse reward re-optimization in this embodiment corresponds to the semantic communication optimization method based on sparse reward re-optimization in the first embodiment described above. The functions implemented by each functional module in the semantic communication optimization system based on sparse reward re-optimization in this embodiment correspond one-to-one with the process steps in the semantic communication optimization method based on sparse reward re-optimization in the first embodiment described above; therefore, they will not be described again here.

[0047] Third Embodiment This embodiment provides an electronic device, such as... Figure 4 As shown, the electronic device includes a processor and a memory; wherein the processor and the memory can be connected via a communication bus; the memory stores at least one instruction, which is loaded and executed by the processor to implement the method of the first embodiment described above. Furthermore, the electronic device may also include a transceiver, the processor and the transceiver can be connected via a communication bus, and the transceiver is used to communicate with other devices.

[0048] Below, in conjunction with Figure 4 A detailed introduction to each component of this electronic device is provided below: The processor is the control center of the electronic device. The electronic device may include multiple processors, each of which can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The term "processor" can refer to a single processor or a collective term for multiple processing elements. For example, a processor can be one or more central processing units (CPUs), other general-purpose processors, application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), one or more field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0049] In a specific implementation, as one example, the processor may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 shown are, of course, merely illustrative examples.

[0050] The memory is used to store the software program that executes the solution of the present invention, and the processor controls its execution. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0051] Optionally, the memory may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may be integrated with the processor or exist independently, and may be accessed through the interface circuit of the electronic device ( Figure 4 (Not shown in the image) is coupled to the processor; however, this embodiment of the invention does not impose specific limitations on this.

[0052] The transceiver may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function. The transceiver can be integrated with the processor or exist independently, and can be connected through the interface circuit of the electronic device (…). Figure 4 (Not shown in the image) is coupled to the processor, and this embodiment of the invention does not specifically limit this.

[0053] In addition, it should be noted that, Figure 4 The structure of the electronic device shown is not intended to limit the device. Actual devices may include more or fewer components than shown, or combine certain components, or have different component arrangements. Furthermore, the technical effects achieved by this electronic device when performing the method of the first embodiment described above can be referenced to the technical effects described in the first embodiment; therefore, they will not be repeated here.

[0054] Fourth embodiment This embodiment provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method of the first embodiment described above. The computer-readable storage medium may be a ROM, random access memory, CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc. The instruction stored therein can be loaded and executed by a processor in a terminal.

[0055] Furthermore, it should be noted that the present invention can be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention can take the form of a completely or partially hardware embodiment, a completely or partially software embodiment, or an embodiment combining software and hardware aspects. Moreover, when implemented in software, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any usable medium accessible to a computer or a data storage device such as a server or data center containing one or more sets of usable media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive (SSD).

[0056] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0058] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element. Furthermore, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Additionally, the character " / " in this text generally indicates an "or" relationship between the preceding and following objects, but it can also indicate an "AND / OR" relationship. Please refer to the context for specific interpretations. "At least one" refers to one or more items, while "more than" refers to two or more items. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be represented as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0059] Furthermore, it is understood that in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0060] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0061] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of functional modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Additionally, the functional units in the various embodiments of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0062] If the method is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0063] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments of the present invention have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make several improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A semantic communication optimization method based on sparse reward re-optimization, characterized in that, include: A semantic communication optimization model is constructed, which includes a semantic transmitter, a semantic communication channel, and a semantic receiver. The semantic transmitter is used to perform temporal modeling on the input semantic information and generate encoded information. The semantic communication channel is used to generate a transmission strategy based on a reinforcement learning algorithm and transmit the encoded information. The semantic receiver is used to decode the received encoded information and reconstruct the original semantic information. Establish a reward prediction model to predict the reward value at future time steps in the reinforcement learning algorithm; Reconstruct the reward sequence based on the predicted reward values ​​at future moments; The semantic communication channel uses reinforcement learning based on the reconstructed reward sequence to update the transmission policy.

2. The semantic communication optimization method based on sparse reward re-optimization as described in claim 1, characterized in that, The semantic sending end uses a long short-term memory network to perform temporal modeling of the input semantic information.

3. The semantic communication optimization method based on sparse reward re-optimization as described in claim 1, characterized in that, The semantic receiver uses a long short-term memory network to decode the received encoded information.

4. The semantic communication optimization method based on sparse reward re-optimization as described in claim 1, characterized in that, The reward function in the reinforcement learning algorithm calculates the reward value. R for: ； in, This indicates the semantic similarity between the reconstructed semantic information and the original semantic information. D Indicates transmission delay; Weights representing semantic similarity; The weight represents the transmission delay.

5. The semantic communication optimization method based on sparse reward re-optimization as described in claim 1, characterized in that, The input to the reward prediction model is the historical reward sequence and state transition sequence generated during the execution of the reinforcement learning algorithm, and the output is the prediction result of the reward value at multiple future time points.

6. The semantic communication optimization method based on sparse reward re-optimization as described in claim 1, characterized in that, The reward prediction model is a recurrent neural network structure.

7. The semantic communication optimization method based on sparse reward re-optimization as described in claim 4, characterized in that, The reward sequence is reconstructed based on the prediction results of reward values ​​at future times, including: Calculate the actual increase in reward at each time step in the predicted reward value for future moments from each historical action; wherein the formula for calculating the actual increase is: ； in, Indicates the first The action is related to the first t The actual increase in rewards per time step; Indicates the first After the first action, for the second... t The predicted reward for each time step; Indicates the first After the first action, for the second... t The predicted reward for each time step; When the actual increase in reward for a historical action at a certain time step is greater than the first preset threshold, the nomination count corresponding to this historical action is incremented by 1; when the nomination count corresponding to a historical action is the maximum or greater than the second preset threshold, this historical action is recorded as a key action. The rewards corresponding to key actions are weighted and allocated forward and backward in time according to the scope of influence before and after the key actions occur, forming a new reward sequence.

8. The semantic communication optimization method based on sparse reward re-optimization as described in claim 7, characterized in that, The formula for weightedly allocating rewards corresponding to key actions forward and backward in time, based on the impact range before and after the key action occurs, is expressed as follows: ； in, Indicates the weighted allocation of the first... t Rewards for each time step; Indicates the reward corresponding to the key action; and This indicates the preset attenuation coefficient; Indicates the moment when the key action occurs; D Indicates the preset forward-allocated window length; O This indicates the preset window length to be allocated backwards.

9. The semantic communication optimization method based on sparse reward re-optimization as described in claim 1, characterized in that, The reinforcement learning based on the reconstructed reward sequence to update the transmission policy includes: Construct an experience replay pool, and store the states, actions, channel observations, decoding results, and reconstructed reward sequences generated by the agent in the semantic communication environment into the experience replay pool in chronological order; At each policy update, a batch of samples is randomly sampled from the experience replay pool; the sampled reward sequence is input into the policy network and the value network, and the network parameters are updated in batches according to the preset reinforcement learning update rules; the iterative optimization process is executed until the preset termination condition is reached, and the optimal transmission policy is output.

10. A semantic communication optimization system based on sparse reward re-optimization, characterized in that, include: The semantic communication optimization model construction module is used to construct a semantic communication optimization model. The semantic communication optimization model includes a semantic transmitter, a semantic communication channel, and a semantic receiver. The semantic transmitter is used to perform temporal modeling on the input semantic information and generate encoded information. The semantic communication channel is used to generate a transmission strategy based on a reinforcement learning algorithm and transmit the encoded information. The semantic receiver is used to decode the received encoded information and reconstruct the original semantic information. The reinforcement learning control module is used for: Establish a reward prediction model to predict the reward value at future time steps in the reinforcement learning algorithm; Reconstruct the reward sequence based on the predicted reward values ​​at future moments; The semantic communication channel uses reinforcement learning based on the reconstructed reward sequence to update the transmission policy.

Images