A semantic-driven multi-access adaptive transmission scheduling method for visual question answering classification

Abstract

The application discloses a kind of semantic drive multi-access adaptive transmission scheduling methods for visual question classification, belong to wireless resource scheduling technical field;Method is: obtaining the task bimodal data generated by multiple user equipments in current time slot;Build semantic encoder, text data and image data in task bimodal data are processed, obtain different modal semantic token representation, and map to unified semantic space;Build resource scheduling problem with the minimum of semantic importance weighted comprehensive transmission cost as target, and solve resource scheduling problem before wireless transmission, obtain scheduling result.The application obtains modal level semantic importance by the way of combining offline Teacher with online Student, directly introduces semantic importance into comprehensive transmission cost index, makes the mode flow more critical to visual question classification result to obtain priority guarantee, improves the effectiveness of receiving end task reasoning;Realize joint trade-off between delay, semantic distortion and power consumption, improve resource utilization efficiency.

Description

Technical Field

[0001] This invention belongs to the field of wireless resource scheduling technology, specifically relating to a semantic-driven multi-access adaptive transmission scheduling method for visual question answering classification. Background Technology

[0002] With the development of edge intelligence, intelligent sensing, and task-driven wireless services, the optimization goal of communication systems is gradually shifting from traditional bit-level high-fidelity transmission to semantic-level efficient transmission oriented towards downstream task performance. In tasks such as visual question answering, the transmitter often needs to jointly transmit image information and question text, while the receiver completes classification reasoning based on the received bimodal semantic representation. Compared with traditional communication methods that only focus on bit error rate, throughput, and capacity, semantic communication emphasizes the actual contribution of the transmitted information to the task result, thus having higher task adaptability value in resource-constrained and channel-fluctuating environments.

[0003] Existing technologies have focused on semantic communication, including semantic representation construction, cross-modal fusion, resource allocation, and online scheduling. One approach emphasizes mapping inputs such as images and text into transmissible semantic representations and performing cross-modal fusion inference at the receiving end. Another approach attempts to correlate physical layer parameters such as power, bandwidth, and modulation / coding with task performance to improve transmission efficiency and system robustness. Furthermore, multi-access transmission mechanisms can provide greater link flexibility for wireless systems, with different access methods offering varying latency, bandwidth, and reliability under different load and channel conditions.

[0004] However, existing technologies still have significant shortcomings in multi-user bimodal semantic transmission scenarios for visual question-answering classification tasks. First, existing solutions are mostly based on single-user, single-link, or single-modal settings, making it difficult to handle the unified scheduling problem when multiple users concurrently upload text and image bimodal data. Second, different access methods typically have different fixed latency overheads, available bandwidth, and resource utilization efficiencies, and the choice of modulation and coding scheme is strongly coupled with bandwidth allocation and transmit power allocation. Existing methods struggle to simultaneously address access selection, modulation and coding scheme selection, and continuous resource allocation within the same framework. Third, in visual question-answering classification tasks, the contributions of text and image modalities to the final classification result are usually different and vary with sample content, channel state, and task context. If resource allocation is still based on traditional averaging or solely on link quality, it can easily lead to insufficient protection of critical semantic information transmission, thus affecting the classification performance of the receiving end.

[0005] Furthermore, in real-world wireless environments, scheduling decisions typically need to be made rapidly at the time slot level. The joint optimization problem of multi-user, multi-access, discrete modulation and coding schemes and continuous resource allocation has obvious hybrid coupling characteristics. It needs to satisfy time delay constraints, semantic distortion constraints, power budget constraints and bandwidth sharing constraints, while also taking into account online deployability.

[0006] Therefore, the technical problem this invention aims to solve is how to address the difficulty in prioritizing the transmission of key semantic information under resource constraints and channel fluctuations in visual question answering classification tasks, thereby affecting the accuracy of visual question answering classification tasks at the receiving end and the efficiency of system resource utilization, and thus improving the transmission efficiency, robustness, and online deployment capability of task-oriented semantic communication systems. Summary of the Invention

[0007] The purpose of this invention is to provide a semantically driven multi-access adaptive transmission scheduling method for visual question answering classification, so as to solve the problems mentioned in the background art.

[0008] The objective of this invention is achieved as follows: a semantically driven multi-access adaptive transmission scheduling method for visual question-answering classification, characterized by the following steps:

[0009] Step S1: Acquire task dual-modal data generated by multiple user devices in the current time slot;

[0010] Step S2: Construct a semantic encoder to process the text and image data in the task's bimodal data, obtain semantic token representations of different modalities, and map them to a unified semantic space;

[0011] Step S3: Obtain the semantic importance of different modal flows;

[0012] Step S4: Construct a resource scheduling problem with the goal of minimizing the semantic importance-weighted comprehensive transmission cost, and solve the resource scheduling problem before wireless transmission to obtain the scheduling result;

[0013] Step S5: Map the scheduling results and perform wireless transmission via the access method.

[0014] Preferably, a semantic encoder is constructed to process the text and image data in the task's bimodal data, obtain semantic token representations of different modalities, and map them to a unified semantic space, specifically as follows:

[0015] Semantic encoders include text semantic encoders and image semantic encoders. The text semantic encoder uses DistilBERT to extract context-dependent semantic features. Let the length of the input text after word segmentation be... The token sequence, whose embedding representation is: ;

[0016] in, For word embedding, For positional embedding; after feeding the input sequence into the text encoder, a context-sensitive representation is obtained: ;

[0017] To control transmission load and preserve key semantic information, the text token sequence is compressed and filtered, resulting in:

[0018] ;

[0019] in, This represents the number of compressed tokens; it is then mapped to a unified semantic dimension via linear projection. :

[0020] ;

[0021] in, and Let represent the linear projection parameters for the text modality and the image modality, respectively; finally, As a sequence of text modal semantic tokens.

[0022] Preferably, the image semantic encoder uses a ResNet50-based convolutional network to extract high-level visual semantic features, denoted as the original image input. The image feature extraction process is represented as follows:

[0023] ;

[0024] Then, the feature map is flattened into a token sequence:

[0025] ;

[0026] in, Indicates the number of visual tokens. This represents the feature dimension of each visual token; to achieve a unified representation across modalities, it is linearly mapped:

[0027] ; Represents a sequence of image modal semantic tokens.

[0028] Preferably, obtaining the semantic importance of different modal flows in step S3 specifically involves:

[0029] Semantic importance is obtained by combining offline Teacher labeling with online Student prediction. In the offline stage, modal importance is estimated by perturbation sensitivity, and in the online stage, the modal-level semantic importance distribution is output by the Student predictor.

[0030] During the offline phase, samples The Modal representation Let the set of multimodal tokens before perturbation be denoted as . ;

[0031] Only for the first The set of scrambled modes is denoted as . Its disturbance sensitivity is estimated as follows:

[0032] ;

[0033] in, This represents the task loss function. It is a numerically stable term; This is a pre-trained visual question answering classification task function used to map multimodal semantic representations to visual question answering classification results; For the first The true answer labels corresponding to each task sample are used to calculate the change in task loss before and after the perturbation during the perturbation sensitivity estimation process; This represents a multimodal semantic fusion function, used to fuse text modal semantic representations and image modal semantic representations to obtain a joint semantic representation for visual question answering classification tasks; Indicates the first In the task sample, the first The variance of the Gaussian perturbation applied to each modal semantic representation is used to characterize the strength of the perturbation applied to that modal semantic representation;

[0034] The corresponding disturbance is represented as:

[0035] ;

[0036] in, The pre-defined perturbation variance is used to characterize the variance of the first... The first user device Each modal semantic represents the strength of the applied perturbation; It is an identity matrix, and its dimensions are consistent with the feature dimensions of the modal semantic representation; This represents the number of perturbation samplings, used to obtain a stable modal sensitivity estimate through multiple perturbation samplings;

[0037] To suppress the over-concentration of importance caused by extreme samples, a compression mapping is performed on the sensitivity:

[0038] ;

[0039] Subsequently, in the sample Normalization is performed on the modality set to obtain the original teacher importance distribution: ;

[0040] To avoid excessive skewness in resource allocation, a uniform prior mixture is introduced to obtain the final Teacher importance:

[0041] ;

[0042] Introducing a lightweight Student predictor in the online phase The input features of Student are composed of global semantic summaries of the text modality and the image modality, as well as key token summaries. The input feature vector is:

[0043] ;

[0044] in, and These represent the global pooling features of the semantic token sequences in the text modality and image modality, respectively. and These represent the two types of modalities after Top- Key summary features obtained from semantic token pooling;

[0045] The Student predictor is based on the input feature vector. Output modal-level semantic importance distribution:

[0046] ;

[0047] in, ;

[0048] The Student learns the Teacher importance distribution through knowledge distillation, and the training objective uses KL divergence.

[0049] ;

[0050] Introducing a lightweight Student predictor in the online phase Based on the semantic representation of the current sample, the modality-level importance distribution is directly output:

[0051] ;

[0052] in, The input feature vector for Student. This represents the predicted semantic importance distribution.

[0053] Preferably, in step S4, a resource scheduling problem is constructed with the objective of minimizing the semantically important weighted comprehensive transmission cost, and the resource scheduling problem is solved before wireless transmission to obtain the scheduling result, specifically as follows:

[0054] Step S4-1: Determine the resource scheduling problem:

[0055] Under conditions of multiple users, multiple modes, and multiple access points, it is necessary to jointly optimize the access methods. MCS Index Bandwidth share and transmission power To minimize the overall task-oriented transmission cost while satisfying power budget, bandwidth sharing, and QoS constraints;

[0056] For the first The first user device Each modal stream, based on the obtained semantic importance Define semantic distortion weight and transmission delay weight:

[0057] ;

[0058] After normalizing semantic distortion, transmission delay, and transmit power, the semantic-aware modal-level transmission metric is expressed as:

[0059] ;

[0060] The resource scheduling problem is described as follows:

[0061] ;

[0062] ;

[0063] ;

[0064] ;

[0065] ;

[0066] ;

[0067] ;

[0068] in, Indicates the first The first user device The transmit power allocated to each modal stream; Indicates the first The maximum transmit power budget that each user equipment is allowed to use in the current scheduling time slot; Indicates assignment to the first The first user device Bandwidth share of each modal stream; This represents the upper bound of the Shannon reachability spectral efficiency of the modal stream under the current channel conditions, used to constrain the realizability of the selected MCS level at the physical layer; Indicates the first The first user device Semantic distortion in a modal stream due to link unreliability; This represents the maximum semantic distortion threshold allowed by the system. Indicates the first The first user device End-to-end transmission delay of a modal stream; This indicates the maximum allowable transmission delay threshold of the system;

[0069] constraint This indicates a user-level transmit power budget constraint, meaning that the sum of the transmit power of all modal streams within the same user equipment must not exceed the maximum transmit power budget of that user equipment; constraint This indicates a bandwidth sharing constraint on the access side, meaning that the sum of the bandwidth shares occupied by each modal stream using the same access method must not exceed the available bandwidth resources of that access method; constraint The constraints represent physical layer realizability constraints, namely, the effective spectral efficiency corresponding to the selected MCS level must not exceed the upper bound of the Shannon reachable spectral efficiency under the current channel state; constraint C4 represents semantic distortion and transmission delay constraints, namely, the semantic distortion and end-to-end transmission delay of each modal stream must not exceed the preset maximum semantic distortion threshold and maximum transmission delay threshold, respectively; constraint C5 represents discrete decision variable value constraints, namely, the access mode index and MCS index are selected from the preset discrete set; constraint C6 represents continuous decision variable value constraints, namely, the bandwidth share is within the preset value range, and the transmit power is a non-negative continuous variable;

[0070] Step S4-2: Decompose the resource scheduling problem into an outer discrete solution and an inner continuous resource allocation problem using a hierarchical solution approach;

[0071] Step S4-3: Solve the two sub-problems of outer-layer discrete decision-making and inner-layer continuous resource allocation respectively.

[0072] Preferably, in step S4-2, a hierarchical solution method is used to decompose the resource scheduling problem into an outer-layer discrete decision-making problem and an inner-layer continuous resource allocation problem, specifically as follows:

[0073] Flow The discrete actions are: ;

[0074] in, Indicates the access method. Represents the MCS index; given a set of discrete actions. After that, the inner layer only needs to optimize continuous variables. ;

[0075] Access method Next, the The first user device The received signal-to-noise ratio of each modal stream is expressed as:

[0076] ;

[0077] Therefore, we can conclude that: ;

[0078] definition:

[0079] The equivalent expression for the transmit power is:

[0080] ;

[0081] in, Indicates the first The first user device Each modal stream is accessed via... During transmission, the equivalent conversion coefficients between transmit power and bandwidth share, and receive signal-to-noise ratio; Indicates the access method Available system bandwidth; Indicates the first Individual user equipment access methods Equivalent channel power gain during transmission; This represents the noise power spectral density of the wireless channel.

[0082] Within a single scheduling slot, and Treated as a known constant, therefore, in bandwidth share Given the following conditions, the transmit power With the received signal-to-noise ratio There is a one-to-one correspondence; the inner optimal response is written as:

[0083] ;

[0084] in, Let be the continuous feasible region after a given discrete action. For the corresponding total cost;

[0085] Therefore, the outer problem can be expressed as: ;

[0086] In discrete actions Given the given information, the inner layer problem only needs optimization. and ;

[0087] First, due to time delay constraints The lower bound of bandwidth share can be obtained:

[0088] ;

[0089] Secondly, due to distortion constraints Using Shannon realizability constraints, we obtain the lower bound of SNR: ;

[0090] in, ;

[0091] in, This represents the maximum semantic distortion threshold allowed by the system. Indicates the selected MCS gear. The positional parameters of the corresponding block bit error rate-signal-noise ratio curve are used to determine the turning point of the curve; Indicates the selected MCS gear. The slope parameter of the corresponding block bit error rate-signal-noise ratio curve is used to characterize the rate at which the block bit error rate decreases as the received signal-noise ratio increases, and ;

[0092] Therefore, the inner continuous feasible region is jointly determined by the lower bound of bandwidth, the lower bound of SNR, the access-side bandwidth sharing constraint, and the UE-level power budget;

[0093] Due to the given access method and channel conditions, the transmit power With the received signal-to-noise ratio There is a one-to-one correspondence, therefore the original power cost item can be... Equivalent rewriting about In the form of After rewriting, given discrete actions The inner contiguous resource allocation target is written as:

[0094] ;

[0095] in, Weighting coefficients representing the cost of semantic distortion; Weighting coefficients representing transmission delay costs; The weighting coefficient representing the cost of transmitting power; Indicates the first The first user device The semantic distortion weights corresponding to each modal stream are obtained from the modal semantic importance in step S3. This is determined to reflect the weighted effect of the semantic importance of this modality on the cost of semantic distortion; Indicates the first The first user device The transmission delay weights corresponding to each modal stream are derived from the obtained modal semantic importance. This is determined to reflect the weighted effect of the semantic importance of this modality on the transmission delay cost; Indicates the first The first user device The amount of semantic data to be transmitted is formed by semantic encoding and quantization of a modal stream; This represents the mapping coefficient between block error rate and semantic distortion, used to convert the block error rate into the corresponding semantic distortion; Indicates the selected access method The corresponding fixed access latency; This indicates the maximum allowed end-to-end transmission delay threshold of the system.

[0096] Preferably, in step S4-3, the two sub-problems of outer-layer discrete decision-making and inner-layer continuous resource allocation are solved separately, specifically as follows:

[0097] Model the outer discrete decision as a contextual multi-armed slot machine problem: convection. In the time slot candidate actions Construct context feature vectors: ;

[0098] in, To aggregate observations of the current flow's link state, load information, and semantic importance; the immediate reward after an action is executed is defined as: ;

[0099] That is, the smaller the total cost, the greater the corresponding reward; the outer learning objective is transformed into: under the condition of unknown reward mapping, gradually learn a better discrete action selection strategy based on historical interaction data.

[0100] Preferably, the discrete action selection strategy learned progressively from historical interaction data employs LinUCB processing for the outer online decision-making layer, for each UE. Maintaining Ridge Regression Statistics:

[0101] ;

[0102] The uncertainty term is: ;

[0103] Its upper confidence boundary is: ;

[0104] in, The trade-off between controlling exploration and exploitation; Represents the ridge regression regularization coefficients, used to initialize the ridge regression statistical matrix and ensure the matrix... Positive definite and reversible; This represents the context feature vector corresponding to the candidate discrete action. The context feature vector includes the link state of the current modal flow, semantic data load, modal semantic importance, and resource state information. Represents the transpose of a vector or matrix;

[0105] Introducing a price variable feeds back the global resource scarcity to the outer scoring function; (Note the access method.) The bandwidth price is UE The power price is Then the candidate action The outer layer score is defined as:

[0106] ;

[0107] in Indicates action The approximate power demand in the current context; therefore, the outer selection rule is: ; This represents the exploration coefficient of the LinUCB algorithm, used to control the trade-off between exploration and exploitation. The larger the value, the more the algorithm tends to explore candidate actions with higher uncertainty. The smaller the value, the more the algorithm tends to select candidate actions with higher current predicted rewards;

[0108] The price variable is updated using projected subgradients, and the bandwidth price is represented as follows: ;

[0109] in, Step size, ;

[0110] Projected subgradient update of power price for user equipment (UEk):

[0111] ;

[0112] in, The power price update step size is indicated; the bandwidth price is used to reflect the degree of bandwidth resource congestion for the corresponding access method, and the power price is used to reflect the tightness of the power budget for user equipment; after the price variable update is completed, a single alternating optimization (AO) is used to solve the inner continuous resource allocation problem.

[0113] Compared with the prior art, the present invention has the following improvements and advantages:

[0114] 1. This invention obtains modal-level semantic importance by combining offline Teacher and online Student, and directly incorporates semantic importance into the comprehensive transmission cost index, so that modal streams that are more critical to the visual question answering classification results are given priority, thereby improving the effectiveness of task inference at the receiving end.

[0115] 2. The invention constructs semantically aware modal-level transmission indicators and combines the mapping relationship between received signal-to-noise ratio, block error rate and semantic distortion to achieve a joint trade-off between latency, semantic distortion and power consumption, thereby improving system resource utilization efficiency and task robustness.

[0116] 3. By unifying the dual-modal semantic transmission of text and images, access method selection, modulation and coding scheme selection, and continuous resource allocation into the same scheduling framework, this invention can better adapt to the task-oriented transmission requirements in multi-user concurrent and heterogeneous access environments.

[0117] 4. This invention adopts a hierarchical scheduling mechanism that combines outer-layer discrete decision-making with inner-layer continuous resource allocation, and reflects the global resource scarcity through a price coordination mechanism. This reduces the online solution complexity of multi-user, multi-access, and strongly coupled discrete and continuous variable problems, and facilitates engineering deployment. Attached Figure Description

[0118] Figure 1 This is a schematic diagram of the architecture of a multi-user uplink semantic communication system.

[0119] Figure 2 This diagram illustrates the convergence process of SAMTI under different scheduling strategies as the scheduling time slot changes.

[0120] Figure 3 This diagram illustrates the average SAMT results for various scheduling strategies under different user sizes.

[0121] Figure 4 This diagram illustrates the impact of semantic importance on resource allocation results.

[0122] Figure 5 A schematic diagram illustrating the results of access mode selection under different system load and semantic importance conditions.

[0123] Figure 6 This diagram illustrates the overall scheduling cost and per-slot computational overhead under different methods.

[0124] Figure 7 This diagram illustrates the relationship between average SAMTI and relative VQA accuracy. Detailed Implementation

[0125] The invention will be further summarized below with reference to the accompanying drawings.

[0126] A joint optimization method for 6G network power and beamforming for collaborative sensing and AI tasks includes the following steps:

[0127] Step S1 involves acquiring dual-modal task data generated by multiple user devices in the current time slot, specifically as follows:

[0128] like Figure 1 As shown, in order to acquire task bimodal data generated by multiple user devices in the current time slot, a multi-user uplink semantic communication system is constructed, consisting of... The system consists of a user equipment (UE) and an edge server (ES). The multi-user uplink semantic communication system operates in time slots and assumes that the channel state remains unchanged within a time slot, but can change with user movement and environmental interference across time slots.

[0129] Each user device generates data samples related to the visual question answering task in the current time slot. The data samples include question text and image data. The edge server does not aim for bit-level error-free reconstruction, but rather aims to recover semantic representations sufficient to support the visual question answering classification task and complete downstream classification.

[0130] In step S2, a semantic encoder is constructed to process the text and image data in the task's bimodal data, obtain semantic token representations of different modalities, and map them to a unified semantic space. Specifically:

[0131] Semantic encoders include text semantic encoders and image semantic encoders. The text semantic encoder uses DistilBERT to extract context-dependent semantic features. Let the length of the input text after word segmentation be... The token sequence, whose embedding representation is: ;

[0132] in, For word embedding, For positional embedding; after feeding the input sequence into the text encoder, a context-sensitive representation is obtained: ;

[0133] To control transmission load and preserve key semantic information, the text token sequence is compressed and filtered, resulting in:

[0134] ;

[0135] in, This represents the number of compressed tokens; it is then mapped to a unified semantic dimension via linear projection. :

[0136] ;

[0137] in, and Let represent the linear projection parameters for the text modality and the image modality, respectively; finally, As a sequence of text modal semantic tokens.

[0138] The image semantic encoder uses a ResNet50-based convolutional network to extract high-level visual semantic features. Let the original image input be... The image feature extraction process is represented as follows:

[0139] ;

[0140] Then, the feature map is flattened into a token sequence:

[0141] ;

[0142] in, Indicates the number of visual tokens. This represents the feature dimension of each visual token; to achieve a unified representation across modalities, it is linearly mapped:

[0143] ; Represents a sequence of image modal semantic tokens.

[0144] The linear projection operation described above transforms the original feature dimensions output by the text encoder and image encoder into the same semantic dimension. Therefore, although the text modal semantic token sequence and the image modal semantic token sequence have different numbers of tokens, each token is located in the same dimension of semantic representation space, thereby realizing the mapping of different modal semantic token representations to a unified semantic space.

[0145] Step S3 obtains the semantic importance of different modal flows, specifically as follows:

[0146] Semantic importance is obtained by combining offline Teacher labeling with online Student prediction. In the offline stage, modal importance is estimated by perturbation sensitivity, and in the online stage, the modal-level semantic importance distribution is output by the Student predictor.

[0147] For the Each user equipment generates a text modal stream and an image modal stream in the current time slot, i.e. .in, Represents a text modal stream. Representing the image modal stream, the bimodal semantic token sequences obtained in step S2 are denoted as follows:

[0148] ;

[0149] During the offline phase, samples The Modal representation Let the set of multimodal tokens before disturbance be denoted as . ;

[0150] Only for the first The set of scrambled modes is denoted as . Its disturbance sensitivity is estimated as follows:

[0151] ;

[0152] in, This represents the task loss function. It is a numerically stable term; This is a pre-trained visual question answering classification task function used to map multimodal semantic representations to visual question answering classification results; For the first The true answer labels corresponding to each task sample are used to calculate the change in task loss before and after the perturbation during the perturbation sensitivity estimation process; This represents a multimodal semantic fusion function, used to fuse text modal semantic representations and image modal semantic representations to obtain a joint semantic representation for visual question answering classification tasks; Indicates the first In the task sample, the first The variance of the Gaussian perturbation applied to each modal semantic representation is used to characterize the strength of the perturbation applied to that modal semantic representation;

[0153] The corresponding disturbance is represented as:

[0154] ;

[0155] in, The pre-defined perturbation variance is used to characterize the variance of the first... The first user device Each modal semantic represents the strength of the applied perturbation; It is an identity matrix, and its dimensions are consistent with the feature dimensions of the modal semantic representation; This represents the number of perturbation samplings, used to obtain a stable modal sensitivity estimate through multiple perturbation samplings;

[0156] To suppress the over-concentration of importance caused by extreme samples, a compression mapping is performed on the sensitivity:

[0157] ;

[0158] Subsequently, in the sample Normalization is performed on the modality set to obtain the original teacher importance distribution: ;

[0159] To avoid excessive skewness in resource allocation, a uniform prior mixture is introduced to obtain the final Teacher importance:

[0160] ;

[0161] While offline teacher calibration offers strong interpretability, it requires multiple perturbation samplings, making it difficult to directly apply to online scheduling. Therefore, a lightweight student predictor is introduced in the online phase. The input features of Student are composed of global semantic summaries of each modality and key token summaries; the input feature vector It can be written as:

[0162] ;

[0163] in, Indicates the first Global pooling features for each modality, Indicates Top- Key summary features obtained by semantic token pooling thus take into account both the global semantic overview and high-value local information;

[0164] The Student predictor is based on the input feature vector. Output bimodal semantic importance distribution:

[0165] ;

[0166] And satisfy: ;

[0167] in, The input feature vector for Student. For the predicted semantic importance distribution, This indicates the relative contribution of the question text to the current visual question answering classification task. This indicates the relative contribution of the image to be analyzed to the current visual question answering classification task;

[0168] The Student learns the semantic importance distribution obtained from offline Teacher labeling through knowledge distillation, and the training objective uses KL divergence.

[0169] ;

[0170] Since modal importance essentially reflects relative proportions rather than independent absolute scores, it is more reasonable to use KL divergence constraints to ensure consistency between the Student and Teacher distributions.

[0171] Step S4 involves constructing a resource scheduling problem with the objective of minimizing the semantically important weighted comprehensive transmission cost. This resource scheduling problem is solved before wireless transmission to obtain the scheduling results. Specifically:

[0172] Step S4-1: Determine the resource scheduling problem:

[0173] Under conditions of multiple users, multiple modalities, and multiple access points, it is necessary to jointly optimize access methods. MCS Index Bandwidth share and transmission power To minimize the overall task-oriented transmission cost while meeting power budget, bandwidth sharing, and QoS constraints;

[0174] For any user equipment (UE) It generates in the current time slot Let the set of modal flows related to the task be denoted as . For User Equipment (UE) The Striped modal flow , Let its original modal observation be denoted as ;

[0175] make Represents a stream The size of the semantic payload to be transmitted in the current time slot is obtained after semantic encoding, then the stream... The effective transmission rate can be written as:

[0176] ;

[0177] in, For the selected MCS gear The corresponding effective spectral efficiency; Index the selected access method; Access method Available system bandwidth; To be assigned to the The first user equipment The bandwidth ratio of each modal stream; Access method Resource utilization coefficient; The effective available bandwidth after considering resource utilization factors for this modal flow;

[0178] Without considering retransmission mechanisms, the end-to-end transmission delay consists of a fixed delay and a net transmission delay, namely:

[0179] ;

[0180] To ensure that the semantic payload can be transmitted within the allowable latency, it is required that ;

[0181] To describe link reliability under discrete MCS settings, a block error rate (BER) model is introduced. For any modal stream, given the MCS setting and received signal-to-noise ratio (SNR), the BLER–SNR variation is approximated using a logistic function.

[0182] ;

[0183] in, Indicates the steepness of the curve. This indicates the reliability inflection point for the corresponding MCS level. Indicates the received signal-to-noise ratio; as As the BLER increases, it will gradually decrease, indicating that the probability of link decoding failure decreases.

[0184] In task-oriented communication scenarios, link decoding failures will result in some semantic representations being unreliable and thus affecting the receiver's fusion inference results. BLER is used as a resolvable surrogate quantity for semantic transmission reliability, and semantic distortion induced by link unreliability is defined accordingly:

[0185] ;

[0186] in, This is the distortion scaling factor; based on BLER, this definition constructs a proxy distortion metric that can be embedded in resource scheduling optimization, used to reflect the impact of changes in link reliability on semantic transmission quality.

[0187] To ensure semantic quality, further requirements are needed. ;

[0188] This constraint indicates that, given the access method and MCS, the link SNR must be sufficient to keep semantic distortion within an acceptable range;

[0189] For any UE Its total transmit power is limited by the user-level power budget:

[0190] ;

[0191] Within a given time slot, the flow Access method The equivalent gain of the link under the following is denoted as This gain is determined by both large-scale path loss and small-scale fading, and is estimated from the pilot signal at the beginning of the time slot, where it is considered constant. Specifically, the channel gain is modeled as follows: ;

[0192] in For large-scale fading terms determined by path loss and shadow fading, Rayleigh small-scale power gain is represented; path loss is modeled using a logarithmic distance model.

[0193] ;

[0194] in, This indicates shadow fading; therefore, the large-scale gain is:

[0195] ;

[0196] Access method The received signal-to-noise ratio is defined as follows:

[0197] ;

[0198] in, The noise power spectral density is given; to ensure that the selected MCS is realizable at the physical layer, an upper bound on the Shannon capacity is introduced as a necessary condition:

[0199] ;

[0200] because The above formula is equivalent to That is, the spectral efficiency of the selected MCS cannot exceed the achievable upper limit under the current channel conditions;

[0201] In addition, for each access method Its bandwidth resources are shared by all flows that choose this access method, therefore satisfying:

[0202] ;

[0203] To unify the dimensions of the three types of costs—semantic distortion, transmission delay, and transmit power—normalization indices are defined for each:

[0204] ; ; ;

[0205] In the offline phase, Teacher importance is obtained based on perturbation sensitivity; in the online phase, modal importance is output by a lightweight Student predictor. And it is directly used for scheduling objectives; to reflect the different effects of semantic importance on distortion and latency, the following definition is made: ;

[0206] in, Indicates semantic fidelity priority. Indicates latency sensitivity; index Used to characterize the diminishing marginal returns of latency;

[0207] Accordingly, the Semantic Aware Modal Level Transmission Metric (SAMTI) is defined as follows:

[0208] ;

[0209] in, Weighting coefficients representing the cost of semantic distortion; Weighting coefficients representing transmission delay costs; The weighting coefficient representing the cost of transmitting power; Indicates the first The first user device The semantic distortion weights corresponding to each modal stream are obtained from the modal semantic importance in step S3. This is determined to reflect the weighted effect of the semantic importance of this modality on the cost of semantic distortion; Indicates the first The first user device The transmission delay weights corresponding to each modal stream are derived from the obtained modal semantic importance. This is determined to reflect the weighted effect of the semantic importance of this modality on the transmission delay cost; Indicates the first The first user device The amount of semantic data to be transmitted is formed by semantic encoding and quantization of a modal stream; This represents the mapping coefficient between block error rate and semantic distortion, used to convert the block error rate into the corresponding semantic distortion; Indicates the selected access method The corresponding fixed access latency; This indicates the maximum allowed end-to-end transmission delay threshold of the system.

[0210] When the semantic importance of a certain modal stream is high, its semantic distortion and transmission delay have a correspondingly increased weight in SAMTI, so that the resource scheduling process prioritizes the transmission reliability and timeliness of that modal stream.

[0211] In the objective function, semantic importance directly affects distortion and latency, thus prioritizing the optimization of modes more critical to the task under resource constraints; the power term is used to suppress energy consumption and avoid unnecessary power spikes. The resource scheduling problem is formulated as follows:

[0212] ;

[0213] ;

[0214] ;

[0215] ;

[0216] ;

[0217] ;

[0218] ;

[0219] in, Indicates the first The first user device The transmit power allocated to each modal stream; Indicates the first The maximum transmit power budget that each user equipment is allowed to use in the current scheduling time slot; Indicates assignment to the first The first user device Bandwidth share of each modal stream; This represents the upper bound of the Shannon reachability spectral efficiency of the modal stream under the current channel conditions, used to constrain the physical layer realizability of the selected MCS level; Indicates the first The first user device Semantic distortion in a modal stream due to link unreliability; This represents the maximum semantic distortion threshold allowed by the system. Indicates the first The first user device End-to-end transmission delay of a modal stream; This indicates the maximum allowable transmission delay threshold of the system.

[0220] constraint This indicates a user-level transmit power budget constraint, meaning that the sum of the transmit power of all modal streams within the same user equipment must not exceed the maximum transmit power budget of that user equipment; constraint This indicates a bandwidth sharing constraint on the access side, meaning that the sum of the bandwidth shares occupied by each modal stream choosing the same access method must not exceed the available bandwidth resources of that access method; constraint The constraints represent physical layer realizability constraints, namely, the effective spectral efficiency corresponding to the selected MCS level must not exceed the upper bound of the Shannon reachable spectral efficiency under the current channel state; constraint C4 represents semantic distortion and transmission delay constraints, namely, the semantic distortion and end-to-end transmission delay of each modal stream must not exceed the preset maximum semantic distortion threshold and maximum transmission delay threshold, respectively; constraint C5 represents discrete decision variable value constraints, namely, the access mode index and MCS index are selected from the preset discrete set, respectively; constraint C6 represents continuous decision variable value constraints, namely, the bandwidth share is within the preset value range, and the transmit power is a non-negative continuous variable.

[0221] To reduce the complexity of online solution to the above problems, a semantically aware hierarchical online scheduling method is proposed, which decomposes the original problem into two sub-problems: outer-layer discrete decision-making and inner-layer continuous resource allocation.

[0222] Step S4-2: Decompose the resource scheduling problem into an outer discrete solution and an inner continuous resource allocation problem using a hierarchical solution approach;

[0223] Flow The discrete actions are: ;

[0224] in, Indicates the access method. Represents the MCS index; given a set of discrete actions. After that, the inner layer only needs to optimize continuous variables. ;

[0225] Access method Next, the The first user device The received signal-to-noise ratio of each modal stream is expressed as:

[0226] ;

[0227] Therefore, we can conclude that: ;

[0228] definition: ;

[0229] The equivalent expression for the transmit power is:

[0230] ;

[0231] in, Indicates the first The first user device Each modal stream is accessed via... During transmission, the equivalent conversion coefficients between transmit power and bandwidth share, and receive signal-to-noise ratio; Indicates the access method Available system bandwidth; Indicates the first Individual user equipment access methods Equivalent channel power gain during transmission; This represents the noise power spectral density of the wireless channel.

[0232] Within a single time slot, and Treated as a known constant, therefore and One-to-one correspondence, no information loss is introduced; therefore, the inner optimal response can be written as:

[0233] ;

[0234] in Let be the continuous feasible region after a given discrete action. To correspond to the total cost; therefore, the outer problem can be expressed as: ;

[0235] In discrete actions Given the given information, the inner layer problem only needs optimization. and First, due to time delay constraints The lower bound of bandwidth share can be obtained:

[0236] ;

[0237] Secondly, due to distortion constraints By applying the Shannon realizability constraint, we can obtain the lower bound of SNR: ;

[0238] in, ;

[0239] Therefore, the inner continuous feasible region is jointly determined by the lower bound of bandwidth, the lower bound of SNR, the access-side bandwidth sharing constraint, and the UE-level power budget;

[0240] Due to the given access method and channel conditions, the transmit power With the received signal-to-noise ratio There is a one-to-one correspondence, therefore the original power cost item can be... Equivalent rewriting about The form;

[0241] Use the original power cost item After rewriting, given discrete actions The inner contiguous resource allocation objective can be written as:

[0242] .

[0243] Step S4-3: Solve the two sub-problems of outer-layer discrete decision-making and inner-layer continuous resource allocation respectively:

[0244] Model the outer discrete decision as a contextual multi-armed slot machine problem: for modal flow In the time slot candidate actions Construct the LinUCB context feature vector:

[0245] ;

[0246] in, Includes the link state, semantic load, access-side resource state, and semantic importance of the current modal flow. Therefore, changes in semantic importance can affect the revenue estimation of candidate actions, which in turn affects the access method and MCS selection.

[0247] That is, the smaller the total cost, the greater the corresponding reward; the outer learning objective is transformed into: under the condition of unknown reward mapping, gradually learn a better discrete action selection strategy based on historical interaction data.

[0248] LinUCB is used to process the outer layer online decision-making for each UE. Maintaining Ridge Regression Statistics:

[0249] ;

[0250] in Here is the regularization parameter; therefore, the parameter estimate for the current time slot is:

[0251] ;

[0252] For candidate actions Its predicted reward is written as: ;

[0253] The uncertainty term is: ;

[0254] Its upper confidence boundary is: ;

[0255] in, The trade-off between controlling exploration and exploitation;

[0256] If actions are selected independently for each flow based solely on UCB scores, it can easily lead to multiple flows simultaneously congesting at the same access method, or induce excessive power demand. Introducing a price variable feeds back the global resource scarcity to the outer scoring function; and recording the access method... The bandwidth price is UE The power price is Then the candidate action The outer layer score is defined as:

[0257] ;

[0258] in, Indicates action Approximate power demand in the current context;

[0259] Therefore, the outer selection rule is: ;

[0260] The price variable is updated using projected subgradients, and the bandwidth price is represented as follows: ;

[0261] in, Step size,

[0262] The inner contiguous resource allocation problem is solved using Alternating Optimization (AO):

[0263] In fixed At that time, regarding The sub-problems can be decomposed into various access methods; since the target includes and Two types of terms, given the dual variable on the access side. Then, the closed-loop update form of the bandwidth share can be obtained:

[0264] ;

[0265] in, By adjusting using binary search, the sharing constraints on the same access method are satisfied:

[0266] ;

[0267] In fixed At that time, regarding The subproblems are separable for each flow; since the BLER term follows It decreases monotonically, while the power term increases with... Since the flow increases linearly, the optimal solution for each flow can be written as:

[0268] ;

[0269] in Let the boundary be an unconstrained stationary point for this one-dimensional subproblem; if the stationary point does not exist or does not satisfy the feasibility requirement, then the boundary is directly taken. Seeking and Then, a projected subgradient update is performed on the UE-level power price:

[0270] ;

[0271] in The step size parameter; through one... - step, once With one step and one price update, an approximate feasible inner-layer solution for the current time slot can be obtained, thus meeting the complexity requirements for online deployment.

[0272] The principle of the hierarchical online scheduling process of this invention is as follows: in time slots Initially, the current context is collected, including link state, semantic importance prediction results, and access-side load information. Then, the outer layer enumerates candidate actions for each modal flow. Calculate its contextual features, LinUCB score, and price-adjusted outer layer score, and determine discrete actions accordingly. After the discrete actions are determined, the inner layer solves for continuous resource allocation based on a single AO (Automatic Allocation) operation, sequentially updating the SNR (Signal NR), bandwidth allocation, and power price, to obtain... and corresponding power Finally, the transmission is performed and the actual total cost is observed. As an immediate reward, the LinUCB statistics will be updated. At the same time, update bandwidth prices .

[0273] In step S5, the scheduling results are mapped and wireless transmission is performed via the access method, specifically as follows:

[0274] Through text semantic encoders, image semantic encoders, and linear projection heads, raw observations from different modalities are uniformly mapped into a sequence of transmissible semantic tokens with consistent dimensions, thereby providing a unified interface for subsequent wireless transmission and receiver fusion.

[0275] The receiver needs to fuse the multimodal semantic tokens recovered via the wireless link to generate a task-level semantic representation and complete downstream inference. Therefore, the fusion module adopts a bidirectional cross-attention mechanism to characterize the complementary relationship between modalities, so that each modality retains its own semantic structure while introducing auxiliary information from another modality, thereby improving the discriminability and robustness of the fused representation.

[0276] When the block error rate decreases, the edge server can more reliably recover text-modal and image-modal semantic tokens; when the block error rate increases, the recovered semantic tokens suffer from greater information loss or semantic distortion. The bimodal semantic tokens recovered by the receiver are denoted as follows:

[0277] ;

[0278] The two types of tokens obtained by the receiving end are respectively and Cross-modal interaction is achieved using bidirectional cross-attention: on one hand, text tokens are used as queries and image tokens as keys / values ​​to obtain text representations enhanced with image semantics; on the other hand, image tokens are used as queries and text tokens as keys / values ​​to obtain image representations enhanced with text semantics; the two types of semantic tokens are then fused through bidirectional cross-attention.

[0279] ;

[0280] ;

[0281] in, This indicates a cross-attention operation; and These represent the results received and recovered by the edge server. A sequence of text modal semantic tokens and a sequence of image modal semantic tokens for each user device; This represents a text modality representation enhanced with semantic information from the image; This represents the image modality representation enhanced by textual semantic information;

[0282] Residual connections are used to preserve original semantic information and enhance training stability;

[0283] The updated token sequence is pooled to obtain a modal-level aggregated representation:

[0284] ;

[0285] ;

[0286] In this approach, text employs masked average pooling to avoid invalid locations interfering with the representation; images, on the other hand, are aggregated using the mean of all visual tokens. Subsequently, the two types of aggregated features are concatenated and mapped to a unified task-level representation.

[0287] ;

[0288] Ultimately, fusion representation Input task header Used to generate prediction results for downstream tasks; express; express.

[0289] Therefore, the resource scheduling results affect the recovery quality of semantic tokens at the receiving end and the cross-modal fusion effect by influencing transmission delay, receiving signal-to-noise ratio, block error rate and semantic distortion, and ultimately affect the performance of visual question answering classification.

[0290] This invention first predicts bimodal semantic importance by Student based on semantic token sequences of text and image modalities; then, semantic importance participates in the outer discrete decision of LinUCB through SAMTI, immediate reward and contextual features, and participates in the inner bandwidth and power optimization of AO through importance weight.

[0291] The resulting resource allocation further determines the transmission rate, transmission delay, received signal-to-noise ratio, block error rate, and semantic distortion of the text modal stream and image modal stream, thereby affecting the quality of semantic token recovery at the receiver and the cross-modal fusion results, and ultimately impacting the performance of visual question answering classification.

[0292] To verify the effectiveness of the method of this invention, a simulation scenario of uplink semantic communication for multiple user devices using a single edge server was constructed. Each user generates an image stream and a text stream, and accesses the network heterogeneously via 4G, 5G, and Wi-Fi. User distances are randomly generated within the range of 10 m to 120 m, the number of scheduling time slots is 200, the maximum transmission deadline is 80 ms, and 10 random seeds are used to run independently and the average is taken. The access method parameters and MCS parameters are shown in Table 1 and Table 2, respectively.

[0293] Table 1 shows the access method parameter settings.

[0294]

[0295] Table 2 shows the modulation and coding parameter settings.

[0296]

[0297] This paper uses visual question answering as a multimodal inference task and employs an image-text joint inference scenario to evaluate the impact of different scheduling strategies on task utility. The image semantic encoder uses ResNet50, and the text semantic encoder uses DistilBERT. The two types of features are linearly projected onto a unified semantic dimension to form a transmissible semantic token representation. This paper sets the semantic dimension to 256 and the quantization bit depth to 8 bits, and uses the semantic encoder and fusion module as the task interface without further optimization of their network structure. To avoid coupling online scheduling with real-time inference of the deep model, this paper offline calculates the relative VQA accuracy under different BLER conditions and estimates task utility through interpolation in the scheduling experiments.

[0298] SAMTI was used as the primary evaluation metric, with lower values ​​indicating lower overall scheduling costs. Comparison methods included fixed 5G access, round-robin scheduling, proportional fair scheduling, maximum rate scheduling, ε-greedy strategy, semantic context-free strategy, and Lyapunov Distortion-plus-Power (D+P) strategy. To ensure fairness, all methods used the same user distribution, channel generation method, semantic load settings, access parameters, MCS parameters, and constraints.

[0299] like Figure 2 As shown in Figure 2, the convergence process of SAMTI as a function of scheduling time slots under different scheduling strategies is illustrated. The curves use a 25-time-slot moving average. It can be seen that the method of this invention gradually converges to a lower cost range after the initial exploration, indicating that the outer LinUCB decision and the inner resource allocation can form a stable scheduling behavior. Compared with various baseline strategies, the method of this invention can adaptively schedule by combining link state, semantic load, modal importance, and resource price, thus achieving a lower steady-state SAMTI.

[0300] like Figure 3 As shown, Figure 3 The average SAMTI of various scheduling strategies under different user scales is presented. As the number of users increases, bandwidth sharing, power contention, and access congestion become more pronounced, and the overall scheduling cost of each method increases. Compared with the comparative strategies, the method of this invention maintains a lower average SAMTI under different user scales, and the growth rate is slower, indicating that it can adapt to multi-user resource contention. Fixed 5G access lacks heterogeneous access adaptation capabilities, and polling and proportional fair scheduling do not explicitly consider the joint trade-off between semantic distortion, latency, and modal importance, thus limiting performance under high load conditions.

[0301] like Figure 4 As shown, Figure 4The impact of semantic importance on resource allocation was further analyzed. Experiments were conducted by dividing modal-level semantic importance into quantile intervals from low to high, and the corresponding normalized distortion and normalized transmission delay were statistically analyzed. The results show that as semantic importance increases, the method of this invention can allocate more favorable transmission resources to high-importance modal streams, resulting in lower distortion and delay. This result demonstrates that semantic importance in SAMTI can actually influence resource allocation, rather than merely being a formal weight in the objective function.

[0302] like Figure 5 As shown in Figure 5, the access mode selection results are presented under different system loads and semantic importance conditions. It can be seen that as the user scale increases, the scheduler increases the selection ratio of high-bandwidth access methods to alleviate resource competition; the differences in access selection under different semantic importance intervals also indicate that the method of the present invention can comprehensively consider semantic importance, access latency, link reliability, and resource occupancy status.

[0303] like Figure 6 As shown in Figure 6, the overall scheduling cost and computational overhead per time slot of different methods are compared. While random search and flow-based greedy algorithms can achieve lower SAMTI in some scenarios, their computation time increases significantly with the number of users. In contrast, the method of this invention quickly selects discrete actions through LinUCB and combines low-complexity continuous optimization to complete resource allocation, achieving a better trade-off between scheduling cost and online computational complexity.

[0304] like Figure 7 As shown in Figure 7, the relationship between average SAMTI and relative VQA accuracy is illustrated. The size of the scatter plot represents the semantic importance of the corresponding modal flow, with larger plots indicating higher semantic importance. It can be seen that the method of this invention generally falls within the lower SAMTI region while maintaining a relatively stable relative task accuracy, demonstrating that semantic-aware scheduling can maintain task utility while reducing system costs.

[0305] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A semantically driven multi-access adaptive transmission scheduling method for visual question answering classification, characterized in that: The method includes the following steps: Step S1: Acquire task dual-modal data generated by multiple user devices in the current time slot; Step S2: Construct a semantic encoder to process the text and image data in the task's bimodal data, obtain semantic token representations of different modalities, and map them to a unified semantic space; Step S3: Obtain the semantic importance of different modal flows; Step S4: Construct a resource scheduling problem with the goal of minimizing the semantic importance-weighted comprehensive transmission cost, and solve the resource scheduling problem before wireless transmission to obtain the scheduling result; Step S5: Map the scheduling results and perform wireless transmission through the access method.

2. The semantic-driven multi-access adaptive transmission scheduling method for visual question answering classification according to claim 1, characterized in that: In step S2, a semantic encoder is constructed to process the text and image data in the task's bimodal data, obtain semantic token representations of different modalities, and map them to a unified semantic space. Specifically: Semantic encoders include text semantic encoders and image semantic encoders. The text semantic encoder uses DistilBERT to extract context-dependent semantic features. Let the length of the input text after word segmentation be... The token sequence, whose embedding representation is: ; in, For word embedding, For positional embedding; after feeding the input sequence into the text encoder, a context-sensitive representation is obtained: ; To control transmission load and preserve key semantic information, the text token sequence is compressed and filtered, resulting in: ； in, This represents the number of compressed tokens; it is then mapped to a unified semantic dimension via linear projection. : ； in, and Let represent the linear projection parameters for the text modality and the image modality, respectively; finally, As a sequence of text modal semantic tokens.

3. The semantic-driven multi-access adaptive transmission scheduling method for visual question answering classification according to claim 2, characterized in that: The image semantic encoder uses a ResNet50-based convolutional network to extract high-level visual semantic features. The original image input is denoted as... The image feature extraction process is represented as follows: ； Then, the feature map is flattened into a token sequence: ； in, Indicates the number of visual tokens. This represents the feature dimension of each visual token; to achieve a unified representation across modalities, it is linearly mapped: ; Represents a sequence of image modal semantic tokens.

4. A semantically driven multi-access adaptive transmission scheduling method for visual question answering classification as described in claim 1, characterized in that: The semantic importance of different modal flows is obtained in step S3, specifically as follows: Semantic importance is obtained by combining offline Teacher labeling with online Student prediction. In the offline stage, modal importance is estimated by perturbation sensitivity, and in the online stage, the modal-level semantic importance distribution is output by the Student predictor. During the offline phase, samples The Modal representation Let the set of multimodal tokens before perturbation be denoted as . ; Only for the first The set of scrambled modes is denoted as . Its disturbance sensitivity is estimated as follows: ； in, This represents the task loss function. It is a numerically stable term; This is a pre-trained visual question answering classification task function used to map multimodal semantic representations to visual question answering classification results; For the first The true answer labels corresponding to each task sample are used to calculate the change in task loss before and after the perturbation during the perturbation sensitivity estimation process; This represents a multimodal semantic fusion function, used to fuse text modal semantic representations and image modal semantic representations to obtain a joint semantic representation for visual question answering classification tasks; Indicates the first In the task sample, the first The variance of the Gaussian perturbation applied to each modal semantic representation is used to characterize the strength of the perturbation applied to that modal semantic representation; The corresponding disturbance is represented as: ； in, The pre-defined perturbation variance is used to characterize the variance of the first... The first user device Each modal semantic represents the strength of the applied perturbation; It is an identity matrix, and its dimensions are consistent with the feature dimensions of the modal semantic representation; This represents the number of perturbation samplings, used to obtain a stable modal sensitivity estimate through multiple perturbation samplings; To suppress the over-concentration of importance caused by extreme samples, a compression mapping is performed on the sensitivity: ； Subsequently, in the sample Normalization is performed on the modality set to obtain the original teacher importance distribution: ; To avoid excessive skewness in resource allocation, a uniform prior mixture is introduced to obtain the final Teacher importance: ； Introducing a lightweight Student predictor in the online phase The input features of Student are composed of global semantic summaries of the text modality and the image modality, as well as key token summaries. The input feature vector is: ； in, and These represent the global pooling features of the semantic token sequences in the text modality and image modality, respectively. and These represent the two types of modalities after Top- Key summary features obtained from semantic token pooling; The Student predictor is based on the input feature vector. Output modal-level semantic importance distribution: ； in, ; The Student learns the Teacher importance distribution through knowledge distillation, and the training objective uses KL divergence. ； Introducing a lightweight Student predictor in the online phase Based on the semantic representation of the current sample, the modality-level importance distribution is directly output: ； in, The input feature vector for Student. This represents the predicted semantic importance distribution.

5. The semantic-driven multi-access adaptive transmission scheduling method for visual question answering classification according to claim 1, characterized in that: In step S4, a resource scheduling problem is constructed with the objective of minimizing the semantically important weighted comprehensive transmission cost. This resource scheduling problem is solved before wireless transmission to obtain the scheduling result. Specifically: Step S4-1: Determine the resource scheduling problem: Under conditions of multiple users, multiple modes, and multiple access points, it is necessary to jointly optimize the access methods. MCS Index Bandwidth share and transmission power To minimize the overall task-oriented transmission cost while satisfying power budget, bandwidth sharing, and QoS constraints; For the first The first user device Each modal stream, based on the obtained semantic importance Define semantic distortion weight and transmission delay weight: ； After normalizing semantic distortion, transmission delay, and transmit power, the semantic-aware modal-level transmission metric is expressed as: ； The resource scheduling problem is described as follows: ； ； ； ； ； ； ； in, Indicates the first The first user device The transmit power allocated to each modal stream; Indicates the first The maximum transmit power budget that each user equipment is allowed to use in the current scheduling time slot; Indicates assignment to the first The first user device Bandwidth share of each modal stream; This represents the upper bound of the Shannon reachability spectral efficiency of the modal stream under the current channel conditions, used to constrain the realizability of the selected MCS level at the physical layer; Indicates the first The first user device Semantic distortion in a modal stream due to link unreliability; This represents the maximum semantic distortion threshold allowed by the system. Indicates the first The first user device End-to-end transmission delay of a modal stream; This indicates the maximum allowable transmission delay threshold of the system; constraint This indicates a user-level transmit power budget constraint, meaning that the sum of the transmit power of all modal streams within the same user equipment must not exceed the maximum transmit power budget of that user equipment; constraint This indicates a bandwidth sharing constraint on the access side, meaning that the sum of the bandwidth shares occupied by each modal stream using the same access method must not exceed the available bandwidth resources of that access method; constraint The constraints represent physical layer realizability constraints, namely, the effective spectral efficiency corresponding to the selected MCS level must not exceed the upper bound of the Shannon reachable spectral efficiency under the current channel state; constraint C4 represents semantic distortion and transmission delay constraints, namely, the semantic distortion and end-to-end transmission delay of each modal stream must not exceed the preset maximum semantic distortion threshold and maximum transmission delay threshold, respectively; constraint C5 represents discrete decision variable value constraints, namely, the access mode index and MCS index are selected from the preset discrete set; constraint C6 represents continuous decision variable value constraints, namely, the bandwidth share is within the preset value range, and the transmit power is a non-negative continuous variable; Step S4-2: Decompose the resource scheduling problem into an outer discrete solution and an inner continuous resource allocation problem using a hierarchical solution approach; Step S4-3: Solve the two sub-problems of outer-layer discrete decision-making and inner-layer continuous resource allocation respectively.

6. The semantic-driven multi-access adaptive transmission scheduling method for visual question answering classification according to claim 5, characterized in that: In step S4-2, a hierarchical solution method is used to decompose the resource scheduling problem into an outer-layer discrete decision-making problem and an inner-layer continuous resource allocation problem, specifically: Flow The discrete actions are: ; in, Indicates the access method. Represents the MCS index; given a set of discrete actions. After that, the inner layer only needs to optimize continuous variables. ; Access method Next, the The first user device The received signal-to-noise ratio of each modal stream is expressed as: ； Therefore, we can conclude that: ; definition: ; The equivalent expression for the transmit power is: ； in, Indicates the first The first user device Each modal stream is accessed via... During transmission, the equivalent conversion coefficients between transmit power and bandwidth share, and receive signal-to-noise ratio; Indicates the access method Available system bandwidth; Indicates the first Individual user equipment access methods Equivalent channel power gain during transmission; Represents the noise power spectral density of a wireless channel; Within a single scheduling slot, and Treated as a known constant, therefore, in bandwidth share Given the following conditions, the transmit power With the received signal-to-noise ratio There is a one-to-one correspondence; the inner optimal response is written as: ； in, Let be the continuous feasible region after a given discrete action. For the corresponding total cost; Therefore, the outer problem can be expressed as: ; In discrete actions Given the given information, the inner layer problem only needs optimization. and ; First, due to time delay constraints The lower bound of bandwidth share can be obtained: ； Secondly, due to distortion constraints Using Shannon realizability constraints, we obtain the lower bound of SNR: ; in, ; in, This represents the maximum semantic distortion threshold allowed by the system. Indicates the selected MCS gear. The positional parameters of the corresponding block bit error rate-signal-noise ratio curve are used to determine the turning point of the curve; Indicates the selected MCS gear. The slope parameter of the corresponding block bit error rate-signal-noise ratio curve is used to characterize the rate at which the block bit error rate decreases as the received signal-noise ratio increases, and ; Therefore, the inner continuous feasible region is jointly determined by the lower bound of bandwidth, the lower bound of SNR, the access-side bandwidth sharing constraint, and the UE-level power budget; Due to the given access method and channel conditions, the transmit power With the received signal-to-noise ratio There is a one-to-one correspondence, therefore the original power cost item can be... Equivalent rewriting about In the form of After rewriting, given discrete actions The inner contiguous resource allocation target is written as: ； in, Weighting coefficients representing the cost of semantic distortion; Weighting coefficients representing transmission delay costs; The weighting coefficient representing the cost of transmitting power; Indicates the first The first user device The semantic distortion weights corresponding to each modal stream are obtained from the modal semantic importance in step S3. This is determined to reflect the weighted effect of the semantic importance of this modality on the cost of semantic distortion; Indicates the first The first user device The transmission delay weights corresponding to each modal stream are derived from the obtained modal semantic importance. This is determined to reflect the weighted effect of the semantic importance of this modality on the transmission delay cost; Indicates the first The first user device The amount of semantic data to be transmitted is formed by semantic encoding and quantization of a modal stream; This represents the mapping coefficient between block error rate and semantic distortion, used to convert the block error rate into the corresponding semantic distortion; Indicates the selected access method The corresponding fixed access latency; This indicates the maximum allowed end-to-end transmission delay threshold of the system.

7. The semantic-driven multi-access adaptive transmission scheduling method for visual question answering classification according to claim 6, characterized in that: In step S4-3, the two sub-problems of outer-layer discrete decision-making and inner-layer continuous resource allocation are solved separately, specifically as follows: Model the outer discrete decision as a contextual multi-armed slot machine problem: convection. In the time slot candidate actions Construct context feature vectors: ; in, To aggregate observations of the current flow's link state, load information, and semantic importance; the immediate reward after an action is executed is defined as: ; That is, the smaller the total cost, the greater the corresponding reward; the outer learning objective is transformed into: under the condition of unknown reward mapping, gradually learn a better discrete action selection strategy based on historical interaction data.

8. The semantic-driven multi-access adaptive transmission scheduling method for visual question answering classification according to claim 7, characterized in that: The improved discrete action selection strategy based on historical interaction data is learned step by step using LinUCB to process the outer online decision-making process for each UE. Maintaining Ridge Regression Statistics: ； The uncertainty term is: ; Its upper confidence boundary is: ; in, The trade-off between controlling exploration and exploitation; Represents the ridge regression regularization coefficients, used to initialize the ridge regression statistical matrix and ensure the matrix... Positive definite and reversible; This represents the context feature vector corresponding to the candidate discrete action. The context feature vector includes the link state of the current modal flow, semantic data load, modal semantic importance, and resource state information. Represents the transpose of a vector or matrix; Introducing a price variable feeds back the global resource scarcity to the outer scoring function; (Note the access method.) The bandwidth price is UE The power price is Then the candidate action The outer layer score is defined as: ； in Indicates action The approximate power demand in the current context; therefore, the outer selection rule is: ; This represents the exploration coefficient of the LinUCB algorithm, used to control the trade-off between exploration and exploitation. The larger the value, the more the algorithm tends to explore candidate actions with higher uncertainty. The smaller the value, the more the algorithm tends to select candidate actions with higher current predicted rewards; The price variable is updated using projected subgradients, and the bandwidth price is represented as follows: ; in, Step size, ; Projected subgradient update of power price for user equipment (UEk): ； in, The power price update step size is indicated; the bandwidth price is used to reflect the degree of bandwidth resource congestion for the corresponding access method, and the power price is used to reflect the tightness of the power budget for user equipment; after the price variable update is completed, a single alternating optimization (AO) is used to solve the inner continuous resource allocation problem.

Images