Multi-modal data transmission method and system based on dynamic priority

Abstract

The invention discloses a multi-modal data transmission method and system based on dynamic priority, and the method comprises the steps: respectively extracting image, voice and text feature data from image, voice and text data in collected multi-modal data of a service, respectively adding corresponding modal embedding vectors, and then carrying out the splicing; inputting the spliced feature data into a cross attention mechanism, and adding the attention scores of the feature data of each modal to obtain the attention score of the multi-modal fusion feature data of the service; performing joint evaluation on the attention score of the multi-modal fusion feature data of the service and a priority preset for the service to obtain a dynamic priority of the service; and preferentially transmitting the data of the service with high dynamic priority. According to the invention, transmission of data with high dynamic priority can be guaranteed preferentially, and key and important data can be guaranteed to be transmitted preferentially.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a multi-modal data transmission method and system based on dynamic priority. Background Art

[0002] With the rapid development of modern power systems, the stability, security and resource utilization efficiency of power communication and dispatching systems, as the core link of power systems, are particularly important. However, in power communication and dispatching systems, traditional secure transmission schemes often rely on preset fixed priority lists and static security policies. Moreover, traditional power communication and dispatching methods are unable to cope with multimodal data from sensors, dynamic resource allocation in power communication and dispatching, and potential security threats.

[0003] Specifically, in the prior art, the system first creates a detailed priority list, which assigns a specific priority to each type of data based on factors such as data type, source, purpose, and real-time requirements. When a data transmission request arrives, the system checks the priority of the request and determines the order of data transmission based on the priority. This mechanism ensures that critical and important data can be transmitted first.

[0004] However, this data transmission scheme based on fixed priority and static security policy also has some obvious disadvantages. First, the fixed priority list cannot be adjusted dynamically according to the real-time situation, which may result in the failure to transmit important data in time in some emergency situations, thus affecting the flexibility and response speed of the system. Second, the scheme ignores the importance differences of multimodal data in different businesses and cannot adjust the priority according to the importance score of multimodal data. Summary of the invention

[0005] In view of this, the purpose of the present invention is to propose a multimodal data transmission method and system based on dynamic priority, which can generate a dynamic priority of multimodal data according to a pre-set business priority and a calculated priority of the multimodal data, thereby giving priority to the transmission of data with a high dynamic priority, ensuring that critical and important data can be transmitted first.

[0006] Based on the above purpose, the present invention provides a multimodal data transmission method based on dynamic priority, comprising:

[0007] Extract image, speech spectrum and text feature data from the multimodal data of the collected business, respectively, and add the corresponding modality embedding vectors to them for splicing;

[0008] The spliced ​​feature data is input into the cross attention mechanism, and the attention scores of the feature data of each modality are added to obtain the attention score of the multimodal fusion feature data of the business;

[0009] Jointly evaluating the attention score of the multimodal fusion feature data of the service and the priority pre-set for the service to obtain a dynamic priority of the service;

[0010] Prioritize the transmission of data of services with high dynamic priority.

[0011] Preferably, the step of jointly evaluating the attention score of the multimodal fusion feature data of the service and the priority pre-set for the service specifically includes:

[0012] The dynamic priority w of the kth service is calculated according to the following formula 1: k :

[0013]

[0014] Among them, α k represents the priority set in advance for the kth service, A k It represents the attention score of the multimodal fusion feature data of the k-th business, K represents the total number of businesses, and β is the set weight value.

[0015] Preferably, after obtaining the dynamic priority of the service, the method further includes:

[0016] Inputting the dynamic priority of the service and the channel state information of the data transmitted by the service into the pre-trained neural network model;

[0017] Bandwidth resources and transmission power are allocated to the service according to the scheduling strategy output by the neural network model.

[0018] Preferably, the neural network model is specifically an Actor-Critic network; and

[0019] During the training process of the neural network model, the reward value is calculated according to the optimization objective function shown in Formula 2 as the reward function, and the model parameters are adjusted according to the reward value:

[0020]

[0021] Among them, B k represents the bandwidth resources allocated to the kth service, P k represents the transmission power allocated to the kth service, h k represents the channel gain matrix of the transmission channel of the kth service, h erepresents the channel gain matrix of the channel from the kth service to the eavesdropper, x k Indicates the data of the kth service transmitted, data(x k ) represents x k The amount of data, t k It is represented as the transmission delay of the data of the kth service, represents the channel gain noise power of the transmission channel of the kth service, represents the channel gain noise power of the channel from the kth service to the eavesdropper, C s,k represents the achievable secure channel capacity for data transmission of the kth service, T out (k) represents the throughput of the kth service.

[0022] Preferably, the neural network model is trained specifically according to the following method:

[0023] Train the neural network model based on different ε values ​​in multiple agents;

[0024] The model parameters of the neural network models trained by different agents are averaged to obtain the global model parameters;

[0025] The neural network model of the global model parameters is further trained.

[0026] The present invention also provides a multi-modal data transmission method based on dynamic priority, comprising:

[0027] Extract image, speech spectrum and text feature data from the multimodal data of the collected business, respectively, and add the corresponding modality embedding vectors to them for splicing;

[0028] The spliced ​​feature data is input into the cross attention mechanism, and the attention scores of the feature data of each modality are added to obtain the attention score of the multimodal fusion feature data of the business;

[0029] Jointly evaluating the attention score of the multimodal fusion feature data of the service and the priority pre-set for the service to obtain a dynamic priority of the service;

[0030] According to the dynamic priority of the service and the state of the channel for transmitting the data of the service, corresponding bandwidth resources and transmission power are scheduled for the service.

[0031] The present invention also provides a multi-modal data transmission system based on dynamic priority, comprising: a data acquisition terminal, and a dispatching center; wherein:

[0032] The data acquisition terminal is used to extract image, voice spectrum, and text feature data from the image, voice, and text data in the multimodal data of the collected business, respectively, and then add the corresponding modal embedding vectors for splicing; input the spliced ​​feature data into the cross-attention mechanism, add the attention scores of the feature data of each modality, and obtain the attention score of the multimodal fusion feature data of the business; jointly evaluate the attention score of the multimodal fusion feature data of the business and the priority pre-set for the business to obtain the dynamic priority of the business; and preferentially transmit the data of the business with a high dynamic priority to the dispatching center.

[0033] The present invention also provides a multi-modal data transmission system based on dynamic priority, comprising: a data acquisition terminal, and a dispatching center; wherein:

[0034] The data acquisition terminal is used to extract image, speech spectrum and text feature data from the image, speech and text data in the multimodal data of the collected business, respectively, and then add the corresponding modal embedding vectors to splice them; input the spliced ​​feature data into the cross-attention mechanism, add the attention scores of the feature data of each modality to obtain the attention score of the multimodal fusion feature data of the business; jointly evaluate the attention score of the multimodal fusion feature data of the business and the priority pre-set for the business, and after obtaining the dynamic priority of the business, send the data of the business and the dynamic priority of the business to the dispatch center;

[0035] The scheduling center is used to schedule corresponding bandwidth resources and transmission power for the service according to the dynamic priority of the service and the channel state information of the data transmitted by the service.

[0036] The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the steps of the above-mentioned multimodal data transmission method based on dynamic priority when executing the computer program.

[0037] The present invention also provides a computer-readable storage medium, in which a computer program is stored. The computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned multimodal data transmission method based on dynamic priority.

[0038] In the technical solution of the present invention, the image, speech, and text data in the multimodal data of the collected business are respectively extracted from the image, speech spectrum, and text feature data, and then added to the corresponding modal embedding vectors for splicing; the spliced ​​feature data are input into the cross-attention mechanism, and the attention scores of the feature data of each modality are added to obtain the attention score of the multimodal fusion feature data of the business; the attention score of the multimodal fusion feature data of the business is jointly evaluated with the priority pre-set for the business to obtain the dynamic priority of the business; and data of businesses with high dynamic priority are transmitted preferentially. The present invention can generate the dynamic priority of multimodal data according to the pre-set business priority and the calculated priority of the multimodal data, thereby giving priority to the transmission of data with high dynamic priority, ensuring that key and important data can be transmitted preferentially;

[0039] Furthermore, the technical solution of the present invention combines the dynamic priority of data with the scheduling algorithm, preferentially allocating communication resources to high-priority services to ensure their secure transmission.

[0040] Furthermore, the technical solution of the present invention can also combine the network's secure channel capacity with the scheduling algorithm, which can take into account both the security of the communication network and the overall scheduling performance (such as throughput);

[0041] In addition, using the neural network model trained by meta-reinforcement learning for resource scheduling can achieve rapid adaptation to scenarios where user behavior or channel status changes rapidly. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A schematic diagram of the architecture of a multimodal data transmission system based on dynamic priority provided by an embodiment of the present invention;

[0044] Figure 2 A flow chart of a training method for a neural network model provided by an embodiment of the present invention;

[0045] Figure 3 A flow chart of a multimodal data transmission method based on dynamic priority provided by an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0048] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present invention should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0049] The inventors of the present invention consider that the traditional power communication dispatching method is unable to cope with the multimodal data of sensors, dynamic resource allocation in power communication dispatching, and potential security threats. In other words, the data transmitted in power communication dispatching presents multimodal characteristics, including heterogeneous data such as images, voices, and texts. How to efficiently extract and utilize the relationship between these data has become an urgent problem to be solved; therefore, the multimodal data fusion scheme of the cross-attention mechanism in the technical solution of the present invention effectively overcomes the limitations of traditional methods in processing heterogeneous data.

[0050] On this basis, the inventors also found that traditional secure transmission solutions often rely on preset fixed priority lists and static security policies; however, the fixed priority list cannot be dynamically adjusted according to real-time conditions, which may result in important data not being transmitted in time in certain emergency situations, thereby affecting the flexibility and response speed of the system. Secondly, the fixed priority solution ignores the importance differences of multimodal data in different businesses and cannot adjust the priority according to the importance score of multimodal data.

[0051] Therefore, in the technical solution of the present invention, a joint priority assessment method of the cross-attention score and external priority of the cross-attention mechanism output for multimodal data is used to prioritize the secure transmission of important data.

[0052] In addition, the inventors also considered that by optimizing the capacity of the secure channel, the security performance of (power) communication scheduling can be guaranteed, and the system's defense capabilities against eavesdropping and illegal access can be improved. For example, in the scenario where illegal eavesdroppers exist in power communication scheduling, the security of data transmission can be further enhanced; and in order to improve security performance while ensuring throughput, it is necessary to jointly optimize security and throughput. Therefore, the present invention designs an optimization objective function that comprehensively considers the capacity of the secure channel and the system throughput. By optimizing the objective function, the capacity of the secure channel, the system throughput, power allocation, bandwidth allocation and delay constraints are comprehensively considered, and the resource utilization efficiency is maximized and the waste of resources is reduced, thereby improving the system throughput and transmission efficiency on the premise of meeting the security constraints, and achieving a balance between security and performance.

[0053] In addition, the inventors also considered that the use of meta-reinforcement learning methods provides the scheduling strategy with the ability to quickly adapt to changing environments; and further through the asynchronous learning of the intelligent agent and the fine-tuning of the scheduling strategy, the model can respond to changes in the communication environment in real time, maintain stable performance and efficiently adapt to new dynamic scenarios.

[0054] Therefore, combining the network's secure channel capacity with the scheduling algorithm can take into account both the security of the communication network and the overall scheduling performance (such as throughput), and can quickly adapt to scenarios where user behavior or channel status changes rapidly.

[0055] Furthermore, the technical solution of the present invention may also adopt sparse coding and physical layer security technology to improve the security performance of the system while ensuring the throughput of the communication scheduling system.

[0056] The technical solutions of the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0057] like Figure 1 As shown, a multi-modal data transmission system based on dynamic priority is proposed in an embodiment of the present invention, and the system can be applied to a power grid communication dispatching system, which includes: a dispatching center 101, and a plurality of data acquisition terminals 102 communicating with the dispatching center;

[0058] The data collection terminal 102 is used to collect business data, which may be multimodal and may include data in image mode, data in voice mode, and data in text mode;

[0059] The data collection terminal 102 is used to extract image feature data, speech spectrum feature data, and text feature data from the image data, speech data, and text data in the multimodal data of the collected business, respectively, and then add the image feature data, speech spectrum feature data, and text feature data to the corresponding modality embedding vectors and perform splicing;

[0060] Then, the data collection terminal 102 inputs the spliced ​​feature data into the cross attention mechanism to obtain the attention score of the feature data of each modality; then, the attention scores of the feature data of each modality are added together to obtain the attention score of the multimodal fusion feature data of the service; and the attention score of the multimodal fusion feature data of the service is jointly evaluated with the priority pre-set for the service to obtain the dynamic priority of the service;

[0061] Specifically, the data collection terminal 102 can calculate the dynamic priority w of the kth service according to the following formula 1: k :

[0062]

[0063] Among them, α k represents the priority set in advance for the kth service, A k It represents the attention score of the multimodal fusion feature data of the k-th business, K represents the total number of businesses, and β is the set weight value.

[0064] The data collection terminal 102 can preferentially transmit the data of the services with high dynamic priority to the dispatch center 101 , thereby giving priority to the transmission of the data with high dynamic priority, and ensuring that the key and important data can be transmitted preferentially.

[0065] Specifically, the data (signal) of the kth service sent by the data collection terminal 102 to the dispatching center 101 is represented by x k , the data (signal) received by the dispatch center 101 is expressed as: Dispatch Center 101 restores the original signal

[0066] Among them, h k When the data collection terminal 102 sends the data of the kth service to the dispatch center 101, the channel matrix of the transmission channel of the data of the kth service, n k is the channel noise of the transmission channel of the kth service data, The data of the kth business restored by the dispatch center.

[0067] In order to protect the key features after fusion, in a more preferred implementation, the data collection terminal 102 sends the x k Sparse coding is performed to send the sparsely coded signal ζ k x k ; Among them, ζ k is a random sparse matrix; since the matrix ζ kDue to the randomness and low dimensionality, it is difficult for an eavesdropper to recover the original signal, thereby improving the security of transmission; accordingly, the signal received by the dispatch center 101 is expressed as:

[0068] The dispatch center 101 uses the matrix ζ k After desparsening the received signal y, the signal is recovered

[0069] Furthermore, the data of the kth service restored in the dispatch center 101 After that, corresponding bandwidth resources and transmission power may be scheduled for the service according to the dynamic priority of the service and the channel state of the data transmitted by the service;

[0070] Specifically, the dispatch center 101 inputs the dynamic priority of the service (the kth service) and the channel state information of the data transmitting the service (the kth service) into a pre-trained neural network model; and allocates bandwidth resources and transmission power to the service (the kth service) according to the scheduling strategy output by the neural network model.

[0071] In order to further improve the security performance of the physical layer, the security channel capacity is used to measure the security performance of the communication scheduling system. The security channel capacity that can be achieved when the kth service is transmitted is: Among them, B k represents the bandwidth resources allocated to the kth service, P k represents the transmission power allocated to the kth service, h k represents the channel gain matrix of the transmission channel of the kth service, h e represents the channel gain matrix of the kth service from the data acquisition terminal to the eavesdropper, represents the channel gain noise power of the transmission channel of the kth service, represents the channel gain noise power of the kth service from the data acquisition terminal to the eavesdropper, C s,k It represents the secure channel capacity that can be achieved when transmitting data of the kth service.

[0072] In order to improve security performance and ensure throughput at the same time, it is necessary to jointly optimize security and throughput. The throughput of the kth service is expressed as: Among them, x k Indicates the data of the kth service transmitted, data(x k ) represents x k The amount of data, t k It is represented as the transmission delay of the data of the kth service.

[0073] While protecting security performance, the throughput of services needs to be considered. Under the constraints of total power, security channel capacity, delay, and transmission rate, the weighted security channel capacity is optimized to ensure that high-priority services are allocated limited power resources. The final optimized solution is the power allocated to each task. Thus, the optimization objective function constructed is shown in Formula 2 below:

[0074]

[0075] The constraints of the above optimization objective function are shown in equations a, b, c, and d above; where T k represents the transmission delay specified for the kth service, which is a value preset before the data of the kth service is sent; P total Indicates total power; B total represents the total bandwidth; K represents the total number of services; constraint (a) is the total power constraint, which ensures that the total power of all service communication data flows is less than or equal to P total ; Constraint (b) is the security channel capacity constraint, which ensures that the security channel capacity is always greater than or equal to 0; Constraint (c) is the delay constraint of the kth service, where T k It is a value preset before sending; constraint (d) is the constraint of total bandwidth.

[0076] The neural network model described above may be an Actor-Critic network; the state space s defined by the Actor-Critic network of the present invention is t ={h1,h2,…,h K ,w1,w2,...,w K}, action space a t = {P1,P2,...,P K ,B1,...B K} and the reward function

[0077] The state space input to the neural network model includes the dynamic priorities of the K services at time t and the channel state information when the data of the K services are transmitted to the dispatch center 101;

[0078] The action space output by the neural network model, that is, the scheduling strategy output by the neural network model, includes the transmission power and bandwidth resources allocated to each service at time t;

[0079] The reward function of the neural network model is the above-mentioned optimization objective function, which is used to calculate the reward value when training the neural network model. That is, during the training process of the neural network model, the reward value is calculated according to the optimization objective function shown in Formula 2 as the reward function, and the model parameters are adjusted according to the reward value.

[0080] Since the optimization objective function takes into account the dynamic priority of the service, the network's secure channel capacity and throughput, the neural network model trained based on the reward function can ensure that communication resources are allocated preferentially to high-priority services to ensure their secure transmission; at the same time, it takes into account the security of the communication network and the overall scheduling performance (such as throughput).

[0081] The asynchronous actor-critic algorithm of the multi-agent in the Actor-Critic network of the present invention is as follows: the critic network is used to guide the actor network, evaluate the quality of the action taken by the actor network, and guide the actor network to take the next action according to the quality. The global actor network is defined as:

[0082] Where V(S,μ) is the defined value function, and the self-attention mechanism is introduced at the front end of the actor network to weight the input state space. Where μ represents the global critic network parameter, represents the scaling factor in the attention mechanism, θ represents the global actor network parameter; W Q ·S·(W K V(S;μ) represents W Q S and W K ·The dot product operation between V(S; μ) is used as the similarity between the two; Norm(·) is the normalization operation, Tanh is the hyperbolic tangent activation function, MLP(·) is a number of linear layers, the purpose is to extract features from the state information, W Q With W K are the mapping matrices between the query vector and the key vector in the self-attention mechanism, θ is the actor network parameter; S represents the state space input into the actor network, and a is the action output by the actor network.

[0083] Defined critic network: To reduce the variance of the estimate, A3C (Asynchronous Advantage Actor-Critic) uses the advantage function under state S and action a: A(S,a)=Q(S,a)-V(S;μ); where V(S;μ)=Norm(Tanh(MLP(S))·π θ(a|S; θ)); μ), Norm(·) is the normalization operation, Tanh is the hyperbolic tangent activation function, MLP(·) is a number of linear layers, Q(S,a) is the Q value function considering n steps, expressed as μ is the global critic network parameter, N is the time step, γ is the discount factor, r t+k represents the immediate reward obtained from time step t+k.

[0084] In a preferred implementation, a meta-reinforcement learning method can be used to train the neural network model. The specific method flow is as follows: Figure 2 As shown, the following steps are included:

[0085] Step S201: training neural network models in multiple agents based on different ε values ​​respectively;

[0086] Specifically, K threads are opened, each thread corresponds to an agent, where each agent copies the actor and critic network in the global network to the local network, that is, the neural network model based on the above Actor-Critic network is copied to the local network; and each agent uses the strategy network with different ε values ​​of ε~greedy to explore the transmission environment from the data acquisition terminal 102 to the dispatch center 101, where ε represents the probability of the agent selecting a random action (exploration) for the action space output by the neural network model, and 1-ε represents the probability of selecting the current optimal action (utilization) according to the greedy strategy. Finally, the channel information, dynamic priority, and allocated transmission power and bandwidth resources of the transmitted service are obtained, forming the query set D query ={h1,h2,...,h K ,w1,w2,...,w K ,P1,P2,...,P K ,B1,B2,...,B K};

[0087] Use the state and action information in the query set to train the local actor and critic network of the agent, that is, train the local neural network model of the agent; during training, calculate the gradients of the two networks locally and use the gradient ascent algorithm to update the model parameters;

[0088] The parameter update process of the local actor network is shown in Equation 3:

[0089]

[0090] In Equation 3, θ', μ' are the parameters of the replicated actor network and critic network, N is the time step, γ is the discount factor, and S i , a iare the state of the i-th time step fed into the actor-critic network and the action output by the actor network, ▽ θ' represents the gradient of the local actor network parameters, dθ represents the parameter gradient of the global actor network, and r i+n Denotes the immediate reward obtained from time step i+n.

[0091] The parameter update process of the critic network is shown in Equation 4:

[0092]

[0093] In Equation 4, μ' is the parameter of the replicated critic network, N is the time step, γ is the discount factor, and S i , a i are the state of the i-th time step fed into the actor-critic network and the action output by the actor network, respectively, and dμ represents the parameter gradient of the global critic network.

[0094] Step S202: averaging the model parameters of the neural network models obtained by training different agents to obtain global model parameters.

[0095] Specifically, according to the calculation formula The parameters μ of the updated critic network and the parameters θ of the updated actor network are obtained, thereby obtaining the global model parameters of the neural network model composed of the Actor-Critic network.

[0096] Step S203: further training the neural network model of the global model parameters.

[0097] Specifically, the dispatch center 101 applies the neural network model of the global model parameters to allocate transmission power and bandwidth resources for the transmitted services; and collects the channel state information, dynamic priority, and allocated transmission power and bandwidth resources of the transmitted services to form a support set D support :

[0098] D support ={h1,h2,...,h K ,w1,w2,...,w K ,P1,P2,...,P K ,B1,B2,...,B K};

[0099] The state and action information in the support set is then used to train the neural network model.

[0100] Based on the above-mentioned multimodal data transmission system based on dynamic priority and the neural network model trained by the above-mentioned method, an embodiment of the present invention provides a multimodal data transmission method based on dynamic priority, and the specific process is as follows: Figure 3 As shown, the following steps are included:

[0101] Step S301: extracting image, speech spectrum, and text feature data from the image, speech, and text data in the collected multimodal data of the service;

[0102] In this step, the data collection terminal 102 extracts image feature data, speech spectrum feature data, and text feature data from the image data, speech data, and text data in the multimodal data of the collected service, respectively:

[0103] Specifically, the data collection terminal 102 normalizes the size of the collected service image data, and then normalizes the pixels to scale the pixel values ​​to the interval [0, 1]; and then uses a three-dimensional convolution operation to extract image feature data;

[0104] The data collection terminal 102 performs Fourier transform on the collected voice data of the service to generate a two-dimensional representation (spectrogram) of the frequency distribution changing over time; for the two-dimensional spectrogram of the voice signal, two-dimensional convolution is used to extract the voice spectrum feature data;

[0105] The data collection terminal 102 uses word embedding to convert the collected text data of the business into a high-dimensional embedding space to obtain a text embedding vector; and extracts text features from the obtained text embedding vector through a one-dimensional convolution operation.

[0106] Step S302: adding the extracted image feature data, speech spectrum feature data, and text feature data of the service to corresponding modality embedding vectors respectively and then concatenating them;

[0107] In this step, the data acquisition terminal 102 expands the image feature data, speech spectrum feature data, and text feature data of the service into a one-dimensional vector, and then adds the corresponding modality embedding vectors to concatenate them to obtain the input embedding=concat(Y I +T I ; Y S +T S ; Y T +T T );

[0108] Among them, concat(·) is a concatenation operation, and the modal types of the three modal data are embedded in the vector; Y I , Y S , Y T Respectively represent the image feature data, speech spectrum feature data, and text feature data of the one-dimensional vector; T I , T S , T TThey respectively represent the modality embedding vector of the image modality, the modality embedding vector of the speech modality, and the modality embedding vector of the text modality.

[0109] Step S303: input the concatenated feature data into the cross attention mechanism to obtain the attention score of the feature data of each modality;

[0110] In this step, the data collection terminal 102 sends the spliced ​​feature data to the cross attention mechanism: in is a learnable modality importance factor, Q, K, and V are query matrices, key matrices, and value matrices converted from the input feature data of different modalities, respectively. Represents the scaling factor.

[0111] The spliced ​​feature data is subjected to multimodal data fusion through a cross-attention mechanism to obtain fused multimodal data features, namely, multimodal fused feature data of the business, and feature data of each modality with an attention score.

[0112] Step S304: adding the attention scores of the feature data of each modality to obtain the attention score of the multimodal fusion feature data of the service;

[0113] In this step, the data collection terminal 102 adds the attention scores of the feature data of each modality of the service to obtain the attention score of the multimodal fusion feature data of the service.

[0114] Step S305: jointly assessing the attention score of the multimodal fusion feature data of the service and the priority pre-set for the service to obtain the dynamic priority of the service;

[0115] In this step, the data collection terminal 102 can calculate the dynamic priority of the service according to the above formula 1.

[0116] Step S306: preferentially transmitting data of services with high dynamic priority;

[0117] In this step, the data collection terminal 102 preferentially transmits data of services with high dynamic priorities to the dispatch center 101 according to the dynamic priorities of the services.

[0118] The service data sent by the data collection terminal 102 to the dispatch center 101 may be the collected service data or the multimodal fusion feature data of the service; in addition, when the data collection terminal 102 sends the service data to the dispatch center 101, it also sends the dynamic priority of the service.

[0119] Step S307: Scheduling corresponding bandwidth resources and transmission power for the service according to the dynamic priority of the service and the channel status of the data of the service.

[0120] In this step, after the dispatch center 101 receives the data of the service sent by the data acquisition terminal 102, it inputs the dynamic priority of the service and the channel state information of the data transmitted by the service into the neural network model pre-trained by the above method; according to the scheduling strategy output by the neural network model, bandwidth resources and transmission power are allocated to the service.

[0121] Furthermore, when the transmission network environment changes, such as when the total bandwidth resources or transmission power changes, new data can be collected to train the neural network model and fine-tune the model parameters of the neural network model until the optimal bandwidth power and bandwidth resource scheduling strategy under the safety constraints is met. The specific training method is the same as above, using the reward function to calculate the reward value and adjust the model parameters, which will not be repeated here.

[0122] In the technical solution of the present invention, the image, speech, and text data in the multimodal data of the collected business are respectively extracted from the image, speech spectrum, and text feature data, and then the corresponding modal embedding vectors are added to perform splicing; the spliced ​​feature data are input into the cross-attention mechanism, and the attention scores of the feature data of each modality are added to obtain the attention score of the multimodal fusion feature data of the business; the attention score of the multimodal fusion feature data of the business is jointly evaluated with the priority pre-set for the business to obtain the dynamic priority of the business; and the data of the business with high dynamic priority is transmitted preferentially. The present invention can generate the dynamic priority of multimodal data according to the pre-set business priority and the calculated priority of multimodal data, thereby giving priority to the transmission of data with high dynamic priority, ensuring that key and important data can be transmitted preferentially.

[0123] Furthermore, the technical solution of the present invention can also allocate bandwidth resources and transmission power to the service through a neural network model according to the dynamic priority of the service and the channel state information of the data transmitting the service; that is, the technical solution of the present invention combines the dynamic priority of the data with the scheduling algorithm to preferentially allocate communication resources to high-priority services to ensure their safe transmission.

[0124] Furthermore, the technical solution of the present invention can also combine the network's secure channel capacity with the scheduling algorithm, which can take into account both the security of the communication network and the overall scheduling performance (such as throughput);

[0125] In addition, using the neural network model trained by meta-reinforcement learning for resource scheduling can achieve rapid adaptation to scenarios where user behavior or channel status changes rapidly.

[0126] Figure 4 The hardware architecture diagram of the computer device 1300 according to the multimodal data transmission method based on dynamic priority according to the embodiment of the present application is schematically shown. In this embodiment, the computer device 1300 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. For example, it can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Figure 4 As shown, the computer device 1300 includes at least but is not limited to: a memory 1310, a processor 1320, and a network interface 1330 which can communicate with each other via a system bus. Among them:

[0127] The memory 1310 includes at least one type of computer-readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 1310 may be an internal storage module of the computer device 1300, such as a hard disk or a memory of the computer device 1300. In other embodiments, the memory 1310 may also be an external storage device of the computer device 1300, such as a plug-in hard disk equipped on the computer device 1300, a smart memory card (Smart Media Card, referred to as SMC), a secure digital (Secure Digital, referred to as SD) card, a flash card, etc. Of course, the memory 1310 may also include both the internal storage module of the computer device 1300 and its external storage device. In this embodiment, the memory 1310 is generally used to store the operating system and various application software installed on the computer device 1300, such as the program code of the multimodal data transmission method based on dynamic priority, etc. In addition, the memory 1310 can also be used to temporarily store various data that have been output or will be output.

[0128] In some embodiments, the processor 1320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 1320 is generally used to control the overall operation of the computer device 1300, such as performing control and processing related to data interaction or communication with the computer device 1300. In this embodiment, the processor 1320 is used to run the program code stored in the memory 1310 or process data.

[0129] The network interface 1330 may include a wireless network interface or a wired network interface, and the network interface 1330 is generally used to establish a communication link between the computer device 1300 and other computer devices. For example, the network interface 1330 is used to connect the computer device 1300 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 1300 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0130] It should be pointed out that Figure 4 Only a computer device having components 1310 - 1330 is shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0131] In this embodiment, the dynamic priority-based multimodal data transmission method stored in the memory 1310 can also be divided into one or more program modules and executed by one or more processors (processor 1320 in this embodiment) to complete the embodiment of the present application.

[0132] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0133] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present invention, the above embodiments or technical features in different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0134] In addition, to simplify the description and discussion, and in order not to obscure the present invention, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the devices may be shown in the form of block diagrams to avoid obscuring the present invention, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the present invention will be implemented (i.e., these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present invention, it will be apparent to those skilled in the art that the present invention may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0135] Although the invention has been described in conjunction with specific embodiments of the invention, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0136] The embodiments of the present invention are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal data transmission method based on dynamic priority, characterized in that: include: Extract image, speech spectrum and text feature data from the multimodal data of the collected business, respectively, and add the corresponding modality embedding vectors to them for splicing; The spliced ​​feature data is input into the cross attention mechanism, and the attention scores of the feature data of each modality are added to obtain the attention score of the multimodal fusion feature data of the business; Jointly evaluating the attention score of the multimodal fusion feature data of the service and the priority pre-set for the service to obtain a dynamic priority of the service; Prioritize the transmission of data of services with high dynamic priority.

2. The method according to claim 1, characterized in that The step of jointly evaluating the attention score of the multimodal fusion feature data of the service and the priority pre-set for the service specifically includes: The dynamic priority w of the kth service is calculated according to the following formula 1: k : Among them, α k represents the priority set in advance for the kth service, A k It represents the attention score of the multimodal fusion feature data of the k-th business, K represents the total number of businesses, and β is the set weight value.

3. The method according to claim 1, characterized in that After obtaining the dynamic priority of the service, the method further includes: Inputting the dynamic priority of the service and the channel state information of the data transmitted by the service into the pre-trained neural network model; Bandwidth resources and transmission power are allocated to the service according to the scheduling strategy output by the neural network model.

4. The method according to claim 3, characterized in that The neural network model is specifically an Actor-Critic network; and During the training process of the neural network model, the reward value is calculated according to the optimization objective function shown in Formula 2 as the reward function, and the model parameters are adjusted according to the reward value: Among them, B k represents the bandwidth resources allocated to the kth service, P k represents the transmission power allocated to the kth service, h k represents the channel gain matrix of the transmission channel of the kth service, h e represents the channel gain matrix of the channel from the kth service to the eavesdropper, x k Indicates the data of the kth service transmitted, data(x k ) represents x k The amount of data, t k It is represented as the transmission delay of the data of the kth service, represents the channel gain noise power of the transmission channel of the kth service, represents the channel gain noise power of the channel from the kth service to the eavesdropper, C s,k represents the achievable secure channel capacity for data transmission of the kth service, T out (k) represents the throughput of the kth service.

5. The method according to claim 4, characterized in that The neural network model is specifically trained according to the following method: Train the neural network model based on different ε values ​​in multiple agents; The model parameters of the neural network models trained by different agents are averaged to obtain the global model parameters; Further training of the neural network model for global model parameters; Among them, ε represents the probability of the agent choosing a random action.

6. A multimodal data transmission method based on dynamic priority, characterized in that: include: Extract image, speech spectrum and text feature data from the multimodal data of the collected business, respectively, and add the corresponding modality embedding vectors to them for splicing; The spliced ​​feature data is input into the cross attention mechanism, and the attention scores of the feature data of each modality are added to obtain the attention score of the multimodal fusion feature data of the business; Jointly evaluating the attention score of the multimodal fusion feature data of the service and the priority pre-set for the service to obtain a dynamic priority of the service; According to the dynamic priority of the service and the state of the channel for transmitting the data of the service, corresponding bandwidth resources and transmission power are scheduled for the service.

7. A multimodal data transmission system based on dynamic priority, characterized in that: include: Data collection terminal, and dispatch center; among them, The data acquisition terminal is used to extract image, voice spectrum, and text feature data from the image, voice, and text data in the multimodal data of the collected business, respectively, and then add the corresponding modal embedding vectors for splicing; input the spliced ​​feature data into the cross-attention mechanism, add the attention scores of the feature data of each modality, and obtain the attention score of the multimodal fusion feature data of the business; jointly evaluate the attention score of the multimodal fusion feature data of the business and the priority pre-set for the business to obtain the dynamic priority of the business; and preferentially transmit the data of the business with a high dynamic priority to the dispatching center.

8. A multimodal data transmission system based on dynamic priority, characterized in that: include: Data collection terminal, and dispatch center; among them, The data acquisition terminal is used to extract image, speech spectrum and text feature data from the image, speech and text data in the multimodal data of the collected business, respectively, and then add the corresponding modal embedding vectors to splice them; input the spliced ​​feature data into the cross-attention mechanism, add the attention scores of the feature data of each modality to obtain the attention score of the multimodal fusion feature data of the business; jointly evaluate the attention score of the multimodal fusion feature data of the business and the priority pre-set for the business, and after obtaining the dynamic priority of the business, send the data of the business and the dynamic priority of the business to the dispatch center; The scheduling center is used to schedule corresponding bandwidth resources and transmission power for the service according to the dynamic priority of the service and the channel state information of the data transmitted by the service.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it is used to implement the steps of the multimodal data transmission method based on dynamic priority as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the dynamic priority-based multimodal data transmission method described in any one of claims 1 to 6.

Images