A multi-modal communication resource allocation method based on semantic importance

Abstract

The application belongs to the field of resource allocation and wireless communication, and particularly relates to a multi-modal communication resource allocation method based on semantic importance. The method comprises the following steps: collecting audio, video and tactile signals in multi-modal services, and extracting semantic information in each mode; in view of a transmission task, considering task semantic correlation and inter-modal internal semantic correlation based on the semantic information, and evaluating semantic importance of each mode in wireless transmission; based on the semantic importance, transmission mode and channel restriction, a resource allocation optimization model for user experience is constructed; and on the basis of the resource allocation optimization model, a joint reinforcement learning method is used to optimize the resource allocation strategy, and wireless resource allocation is performed. The application effectively solves the problem of poor semantic transmission fidelity caused by channel occupation in different modal transmission based on the cross-modal semantic transmission architecture and joint reinforcement learning, and significantly improves the quality of user experience (QoE) and transmission accuracy.

Description

Technical Field

[0001] This invention belongs to the field of resource allocation and wireless communication technology, specifically relating to a multimodal communication resource allocation method based on semantic importance. Background Technology

[0002] With the continuous development and advancement of next-generation communication technologies such as 6G, the application of cross-modal services is becoming increasingly widespread, and it has become the mainstream scenario in the current communication field (e.g., telemedicine, remote industrial control, motion-sensing games). These emerging technologies have not only greatly enriched communication methods, but also promoted the integration of multiple transmission modes such as voice, video, and haptic streaming, enabling users to enjoy more efficient and flexible communication services in different scenarios.

[0003] In cross-modal service applications, the transmission method of multiple modal data has undergone fundamental changes compared to traditional multimodal communication systems. In multimodal communication systems, different types of data, such as voice, video, and haptic streams, can be transmitted independently on different radio resources, each occupying a certain frequency band or time slot, thus achieving effective resource allocation and management. However, in cross-modal services, this segmented resource allocation method is no longer applicable. For example, in a perforated architecture, the transmission of haptic streams, which have high requirements for latency and reliability, will occupy radio resources allocated to other service data (audio and video streams).

[0004] Therefore, cross-modal service scenarios require communication systems to efficiently process and transmit multiple data types simultaneously on the same frequency. This change in transmission mode presents unprecedented challenges to the allocation of data transmission resources. It is necessary not only to ensure the synchronization and stability of various modal data during transmission, but also to optimize resource allocation strategies to achieve efficient spectrum utilization and a superior service experience. Summary of the Invention

[0005] This invention provides a multimodal communication resource allocation method based on semantic importance to solve problems such as poor semantic transmission fidelity caused by channel preemption in different modal transmissions, thereby improving the quality of user experience (QoE) and transmission accuracy.

[0006] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: a multimodal communication resource allocation method based on semantic importance, comprising:

[0007] S1. Collect audio, video, and tactile signals from multimodal services and extract semantic information from each modality;

[0008] S2. For transmission tasks, based on semantic information, consider the semantic relevance of tasks and the internal semantic relevance between modes to evaluate the semantic importance of each mode in wireless transmission;

[0009] S3. Based on semantic importance, transmission method, and channel limitations, construct a resource allocation optimization model oriented towards user experience;

[0010] S4. Based on the resource allocation optimization model, a joint reinforcement learning method is used to optimize the resource allocation strategy and perform wireless resource allocation.

[0011] Furthermore, the method in step S1 includes:

[0012] S11. Collect multimodal data of audio (a), video (v), and tactile (t) from the input terminal, and extract cross-modal semantic information;

[0013] S12. Based on formulas (1)-(3), extract the semantic features of the different modalities mentioned above to obtain semantic fusion features, wherein formulas (1)-(3) are:

[0014] (1);

[0015] (2);

[0016] (3);

[0017] A=a s ⊕v s ⊕t s (4);

[0018] Among them, a s For audio semantic information, v s For video semantic information, t s For tactile semantic information, The parameters are trainable; CNN() is the convolutional neural network function; ViT() is the visual transformer function. (;) represents the feature extraction function based on GAN, and A represents the fused feature;

[0019] Furthermore, the method in step S2 includes:

[0020] S21. For the transmission task p, input the modality features to be evaluated into the pre-trained semantic encoder and decoder, and compute each element p in task p. n For the gradients of semantic feature A, based on formula (5), these gradients are globally averaged over the spatial dimensions with width and height indexed as i and j, respectively, to obtain the task semantic relevance of the k-th semantic feature. ;

[0021] (5);

[0022] Where W and H are A k Width and height, p is the activation value of the semantic feature in the i-th row and j-th column, where N is the task dimension. n It is the nth element of task p;

[0023] S22. A k Internal semantic relevance v k The average correlation coefficient between this feature and all other features is obtained. Based on formulas (6)-(7), considering the semantic features at the k-th and j-th positions, we obtain A. k With A j Internal semantic correlation between them;

[0024]

[0025] (7);

[0026] in, It is the inner product. For absolute values, sim uses cosine similarity, and C-1 is the similarity with A. k The number of relevant internal semantic features;

[0027] S23. The semantic importance is obtained by combining the semantic relevance of the task with the internal semantic relevance, as shown in formula (8):

[0028] (8);

[0029] in, For semantic importance;

[0030] Furthermore, the method in step S3 includes:

[0031] S31. To maximize the long-term user experience. u To achieve the optimization objective, a resource allocation optimization model is established as shown in the following formula:

[0032] (9)

[0033] Where u represents the user equipment set, U={1,2,…,u}; D represents the data rate threshold required by different users u; L represents the latency threshold; and R represents the reliability threshold.

[0034] Furthermore, the method in step S4 includes:

[0035] S41. Construct a joint reinforcement learning model, taking the semantic importance of the current transmission modality, the remaining semantic data to be transmitted, and the remaining transmission time as the state, the number of semantics of the currently selected transmission modality, the allocated bandwidth, and the computing resources as the action, and the user experience QoE as the reward, and establish the joint reinforcement learning model reward function as shown in the following formula;

[0036] r=U u + A u l + A u r (10)

[0037] Where r is the reward for joint reinforcement learning, A u l A u r These are the penalties for failing to meet communication latency and communication reliability requirements at the u-th user, respectively. , .

[0038] S42. Train the joint reinforcement learning model, update its network parameters, and obtain a resource allocation strategy that can achieve optimal resource allocation.

[0039] The beneficial effects of this invention are as follows: Based on cross-modal transmission architecture and joint reinforcement learning, this invention utilizes the semantic association between modalities to define the semantic importance of transmission modalities, and uses this as a guide to establish a joint reinforcement learning model, optimize resource allocation strategies, effectively solve the problem of poor semantic transmission fidelity caused by channel preemption in different modal transmissions, and significantly improve the quality of user experience (QoE) and transmission accuracy. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the semantic communication resource allocation architecture of a multimodal communication resource allocation method based on semantic importance according to an embodiment of this application;

[0041] Figure 2 This is a flowchart illustrating a multimodal communication resource allocation method based on semantic importance according to an embodiment of this application;

[0042] Figure 3 This is a schematic diagram illustrating the semantic importance evaluation of a multimodal communication resource allocation method based on semantic importance according to an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of a joint reinforcement learning resource allocation model for a multimodal communication resource allocation method based on semantic importance, according to an embodiment of this application.

[0044] Figure 5 This is a comparison chart of the mean absolute error (MAE) of tactile signal recovery in a multimodal communication resource allocation method based on semantic importance according to an embodiment of this application;

[0045] Figure 6This is a convergence curve of the training accuracy of a reinforcement learning comparison algorithm for a multimodal communication resource allocation method based on semantic importance, according to an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments.

[0047] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification.

[0048] Example 1

[0049] See Figure 1 This embodiment provides a multimodal communication resource allocation method based on semantic importance. The task-oriented semantic communication network uses CNN, ViT, and GAN to extract semantic information from audio, image, and tactile tasks, and introduces a semantic relay to assist in the decoding and forwarding of text semantics. Specifically, the semantic communication network model includes a base station, a semantic relay, and N terminal users; the terminal users include audio, image, and tactile users; the intelligent tasks in the network include audio, image, and tactile tasks; wherein, the base station equipment deploys a neural network for generating interpretable image semantic features and a DeepSC encoder neural network for extracting text semantic information, achieving interpretable image semantic transmission while reducing the occupation of transmission channel resources. DeepSC is used as the semantic encoder to encode text data into semantic symbols and send them to the semantic relay; a DeepSC decoder neural network is deployed on the semantic relay to assist in the decoding and transmission of text information, decoding and recovering the received semantic information, and transmitting the decoding result to the nearby text task terminal user through traditional bit encoding; by using the DeepSC model as a text semantic encoder, the transmission efficiency of text tasks is improved. Its core logic lies in utilizing the semantic associations between modalities to evaluate the semantic importance of transmission modalities. Guided by this, a reinforcement learning model is established to optimize resource allocation strategies, effectively solving problems such as poor semantic reliability caused by channel preemption among different modalities during transmission, and significantly improving the quality of user experience (QoE). Logically, the system can be divided into multimodal semantic extraction, semantic importance evaluation, and semantic importance-based resource allocation modules.

[0050] See Figure 2 The method execution flow of this embodiment of the invention is as follows:

[0051] S1. Collect audio (a), video (v), and tactile signals (t) from multimodal services, and extract semantic information from each modality;

[0052] Specifically, the method in step S1 includes:

[0053] S11. Extract cross-modal semantic information based on the multimodal data of audio (a), video (v), and tactile (t) from the input end;

[0054] S12. Based on formulas (1)-(3), extract the semantic features of the different modalities mentioned above to obtain semantic fusion features, wherein formulas (1)-(3) are:

[0055] (1);

[0056] (2);

[0057] (3);

[0058] A=a s ⊕v s ⊕t s (4);

[0059] Among them, a s For audio semantic information, v s For video semantic information, t s For tactile semantic information, The parameters are trainable; CNN() is the convolutional neural network function; ViT() is the visual transformer function. (;) represents the feature extraction function based on GAN, and A represents the fused feature;

[0060] S2. See Figure 3 For transmission tasks, based on semantic information, we consider the semantic relevance of tasks and the internal semantic relevance between modes to evaluate the semantic importance of each mode in wireless transmission.

[0061] Specifically, the method in step S2 includes:

[0062] S21. For the transmission task p, input the modality features to be evaluated into the pre-trained semantic encoder and decoder, and compute each element p in task p. n For the gradients of semantic feature A, based on formula (5), these gradients are globally averaged over the spatial dimensions with width and height indexed as i and j, respectively, to obtain the task semantic relevance of the k-th semantic feature. ;

[0063] (5);

[0064] Where W and H are A k Width and height, p is the activation value of the semantic feature in the i-th row and j-th column, where N is the task dimension.n It is the nth element of task p;

[0065] S22. A k Internal semantic relevance v k The average correlation coefficient between this feature and all other features is obtained. Based on formulas (6)-(7), considering the semantic features at the k-th and j-th positions, we obtain A. k With A j Internal semantic correlation between them;

[0066]

[0067] (7);

[0068] in, It is the inner product. For absolute values, sim uses cosine similarity, and C-1 is the similarity with A. k The number of relevant internal semantic features;

[0069] S23. The semantic importance is obtained by combining the semantic relevance of the task with the internal semantic relevance, as shown in formula (8):

[0070] (8);

[0071] in, For semantic importance;

[0072] S3. Based on semantic importance, transmission method, and channel limitations, construct a resource allocation optimization model oriented towards user experience;

[0073] Specifically, the method in step S3 includes:

[0074] S31. To maximize the long-term user experience. u To achieve the optimization objective, a resource allocation optimization model is established as shown in the following formula:

[0075] (9)

[0076] Where u represents the user equipment set, U={1,2,…,u}; D represents the data rate threshold required by different users u; L represents the latency threshold; and R represents the reliability threshold.

[0077] S4. Based on the resource allocation optimization model, a joint reinforcement learning method is used to optimize the resource allocation strategy and perform wireless resource allocation.

[0078] Specifically, the method in step S4 includes:

[0079] S41. Construct a joint reinforcement learning model that incorporates the semantic importance w of the current transmission modality. k The remaining semantic data A that needs to be transmitted l and the remaining transmission time T l As a state, the number of semantics M of the currently selected transmission mode is... t The allocated bandwidth B t and computing resources R t As an action, the user experience QoE is used as a reward, and a joint reinforcement learning model reward function is established as shown in the following formula;

[0080] r=U u + A u l + A u r (10)

[0081] Where r is the reward for reinforcement learning, A u l A u r These are the penalties for failing to meet communication latency and communication reliability requirements at the u-th user, respectively. , .

[0082] S42. Train the joint reinforcement learning model, update its network parameters, and obtain a resource allocation strategy that achieves optimal resource allocation. For example... Figure 4 As shown, the reinforcement learning algorithm includes a deep triple Q network (D3QN) and actor and critic networks. The actor and critic networks are used to process continuous multimodal audio-visual-tactile stream data, while D3QN is used to process discrete semantic information data.

[0083] Specifically, the method in step S42 includes:

[0084] S421. Define the state space, action space, and reward function for reinforcement learning;

[0085] The state space of the system is s = {(W, A, T)}, where W = [w 1, ..., w k ,... ,w u [A] represents the semantic importance of the current transmission mode of the system user, where A = [a1, ..., a2]. k ,... ,a u [] represents the remaining transmission semantics of the system user, T=[t1, ..., t] k ,... ,tu [] indicates the remaining transmission time for system users.

[0086] S422. Initialize the parameters of the reinforcement learning algorithm and clear the experience replay pool;

[0087] S423. Initialize actions according to a random strategy, and generate random initialization states based on the initial actions;

[0088] The current state s t ={W t A t ,T t The algorithm model takes a D3QN network and an actor / critic network as input. The D3QN network takes a discrete action a={p} based on the current state, and the actor / critic network takes a continuous action a={q} based on the current state. Here, p represents the number of semantic symbols transmitted by the modality with the highest semantic importance, and q represents the allocated loan and computing resources. Then, the agent executes the action and the state transitions to s. t+1 ={W t+1 A t+1 ,T l+1}, calculate the immediate reward r, and (s t ,a t ,r t ,s t+1 Stored in the experience replay buffer;

[0089] Wherein, the loss function of D3QN :

[0090] (11)

[0091] in, This serves as a buffer for replaying discrete action experiences. Here, θ is the discount factor, and θ is the parameter used to evaluate the Q-network. - For the target Q network parameters, It is the immediate reward obtained after performing the action a={p}. ' is the dominant state after performing the action a={p}. ' is the dominant action following the action a={p}. It is an action selection operator;

[0092] The loss functions for actors and critics are:

[0093] (12)

[0094] Where Z is the continuous action experience replay buffer. As a discount factor, To evaluate the Q-network parameters, θ* For the target Q network parameters, t+1 It is the next state after the action a={q} is executed. It is the immediate reward obtained after performing the action a={q}. It is the next action after the action a={q} is executed;

[0095] S424. Then, in each iteration, update the parameters of the D3QN network and the actor critic network in the joint reinforcement learning architecture.

[0096] (13)

[0097] (14)

[0098] To verify the role of this invention in network resource allocation, this embodiment developed the VisTouch dataset, a large-scale multimodal dataset created through audio, video, and tactile interactions between various materials and tactile gloves. This dataset contains over 1000 pairs of synchronized audio, video, and tactile signals, covering 47 material categories. To compare the recovered signals with the original signals to evaluate the quality of master-slave signal transmission, specifically for tactile signals, audio, visual, and tactile stream data for eight materials (including spandex, paper, polyester, brass, fabric, wood, silk, and stone) under different communication architectures were collected from the master end. Based on the above dataset, a cross-modal semantic communication framework was constructed to recover the tactile signals from the receiving end.

[0099] MAE comparison chart of tactile signal recovery under different transmission modes is shown below Figure 5 As shown in the figure, the results demonstrate that the cross-modal semantic transmission method, which incorporates cross-modal semantic importance into the transmission, achieves better tactile signal recovery than (comparison methods: bitstream transmission and semantic transmission).

[0100] The convergence speed of different reinforcement learning methods in resource allocation, such as Figure 6 As shown, our proposed framework generates better cumulative rewards as the number of training epochs increases due to its faster convergence and higher stability. By changing the policy constraint, our method ensures training stability, which is significantly superior to methods such as the Double Delay Deep Deterministic Policy Gradient (TD3), Deep Deterministic Policy Gradient (DDQN), and Deep Deterministic Policy Gradient (DDPG).

[0101] Example 2

[0102] The computer-readable storage medium of this embodiment stores a computer program that, when executed by a processor, implements the steps of the robust multi-view skeleton fusion method based on information priority selection and effective joint masking in Embodiment 1.

[0103] The computer-readable storage medium in this embodiment can be an internal storage unit of the terminal, such as the terminal's hard disk or memory; the computer-readable storage medium in this embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc. equipped on the terminal; furthermore, the computer-readable storage medium can include both the terminal's internal storage unit and external storage devices.

[0104] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0105] Example 3

[0106] The computer device of this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the robust multi-view skeleton fusion method based on information priority selection and effective joint masking of Embodiment 1.

[0107] In this embodiment, the processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The memory can include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0108] Those skilled in the art will understand that the content disclosed in the embodiments can be provided as a method, system, or computer program product. Therefore, this solution can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this solution can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage) containing computer-usable program code.

[0109] This solution is described with reference to flowchart illustrations and / or block diagrams of methods and computer program products according to embodiments of this solution. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0113] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A multimodal communication resource allocation method based on semantic importance, characterized in that, include: S1. Collect audio, video, and tactile signals from multimodal services and extract semantic information from each modality; S2. For transmission tasks, based on semantic information, consider the semantic relevance of tasks and the internal semantic relevance between modes to evaluate the semantic importance of each mode in wireless transmission; S3. Based on semantic importance, transmission method, and channel limitations, construct a resource allocation optimization model oriented towards user experience; S4. Based on the resource allocation optimization model, a joint reinforcement learning method is used to optimize the resource allocation strategy and perform wireless resource allocation.

2. The method according to claim 1, characterized in that, The method in step S1 includes: S11. Collect multimodal data of audio (a), video (v), and tactile (t) from the input terminal, and extract cross-modal semantic information; S12. Based on the following formula, extract the semantic features of the different modalities mentioned above to obtain the semantic fusion features; ； ； ； A=a s ⊕v s ⊕t s ; Among them, a s For audio semantic information, v s For video semantic information, t s For tactile semantic information, θ(·) are trainable parameters, CNN(;) is a convolutional neural network function, and ViT(;) is a visual transformer function. (;) represents the feature extraction function based on GAN, and A represents the fused feature.

3. The method according to claim 1, characterized in that, In step S2, the semantic importance of each modality in wireless transmission is evaluated, including: S21. Based on the following formula, for the transmission task p, input the modality features to be evaluated into the pre-trained semantic encoder and decoder, and compute each element p in task p. n For the gradients of semantic feature A, calculate the global average of these gradients across the spatial dimensions indexed i and j for width and height, respectively, to obtain the task semantic relevance of the k-th semantic feature. ; ； Where W and H are A k Width and height, p is the activation value of the semantic feature in the i-th row and j-th column, where N is the task dimension. n It is the nth element of task p; S22. A k Internal semantic relevance v k A is obtained by taking into account the average correlation coefficient between this feature and all other features, and considering the semantic features at the k-th and j-th positions based on the following formula. k With A j Internal semantic correlation between them; ； ； in, It is the inner product. For absolute values, sim uses cosine similarity, and C-1 is the similarity with A. k The number of relevant internal semantic features; S23. Semantic importance is obtained by combining task semantic relevance and internal semantic relevance; ； in, For semantic importance.

4. The method according to claim 1, characterized in that, In step S3, a resource allocation optimization model oriented towards user experience is constructed, specifically as follows: S31. To maximize the long-term user experience. u To achieve the optimization objective, a resource allocation optimization model is established as shown in the following formula: Where u represents the user equipment set, U={1,2,…,u}; D represents the data rate threshold required by different users u; L represents the latency threshold; and R represents the reliability threshold.

5. The method according to claim 4, characterized in that, In step S4, the allocation of wireless resources specifically includes: S41. Construct a joint reinforcement learning model, taking the semantic importance of the current transmission modality, the remaining semantic data to be transmitted, and the remaining transmission time as the state, the number of semantics of the currently selected transmission modality, the allocated bandwidth, and the computing resources as the action, and the user experience QoE as the reward, and establish the joint reinforcement learning model reward function as shown in the following formula; r=U u + A u l + A u r Where r is the reward for joint reinforcement learning, A u l A u r These are the penalties for failing to meet communication latency and communication reliability requirements at the u-th user, respectively. , ; S42. Train the joint reinforcement learning model, update its network parameters, and obtain a resource allocation strategy that can achieve optimal resource allocation.

Images