Resource Scheduling Method and System for Multi-User Multi-Modal Flow Integrity Transmission Task

Abstract

The present invention discloses a resource scheduling method for multi-user multi-modal flow integrity transmission tasks, including the following steps: for multi-user multi-modal flow transmission tasks, obtain the audio-visual flow transmission rate and the haptic flow transmission delay; according to the audio-visual flow transmission rate and the haptic flow transmission delay, with the goal of maximizing the multi-user multi-modal perception utility, establish an optimization mathematical model for resource scheduling, and solve the model to obtain the resource quantity that maximizes the multi-user multi-modal perception utility under the constraints of spectrum and power resources; allocate power and spectrum resources to multiple users according to the resource quantity required by the users. The resource scheduling method for multi-user multi-modal flow integrity transmission tasks proposed by the present invention comprehensively considers the real-time requirements of haptic flow transmission and the integrity requirements of audio-visual flow transmission, and improves the multi-modal perception utility of users in virtual-real fusion services by allocating spectrum and power resources.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communication, and more specifically, relates to a resource scheduling method for multi-user multi-modal flow integrity transmission tasks. Background Art

[0002] The emergence and rapid expansion of the metaverse have promoted the rapid development of virtual-real fusion services mainly based on technologies such as virtual reality, augmented reality, and mixed reality. In user-centered virtual-real fusion interactions, multi-sensory virtual-real fusion services based on vision, hearing, and touch can provide an immersive experience for multiple users. Due to the significant differences in the transmission characteristics and requirements of different sensory signals in immersive virtual-real fusion services, how to effectively transmit multi-user multi-modal flows still requires further research.

[0003] In the transmission of multi-modal flows, the transmission of audio-visual flows has the characteristics of enhanced mobile broadband communication, while the transmission of tactile flows has the characteristics of ultra-reliable low-latency communication. The flexible frame structure and resource puncturing scheme in 5G New Radio are effective transmission methods to meet the performance requirements of multi-modal services in cellular networks. Traditional wireless resource optimization aims mainly at successfully transmitting data packets, while the transmission of multi-modal flows should consider the integrity transmission of audio-visual flows under the condition of meeting the transmission requirements of tactile flows.

[0004] From the perspective of network transmission, each audio-visual frame consists of dozens of Internet Protocol (IP) data packets. Without considering hierarchical or slicing schemes, the audio-visual frame can be decoded at the receiving end only when all relevant IP data packets are successfully received. If the receiver does not receive one or more IP data packets, the corresponding audio-visual frame cannot be decoded, thus affecting the user's immersive experience. For multi-user multi-sensory virtual-real fusion services, the complete transmission of audio-visual frames and the real-time transmission of tactile flows are both important for the user's immersive multi-sensory experience. How to perform resource scheduling under the real-time constraint of user tactile flow transmission to meet the integrity transmission requirements of user audio-visual flows is a challenging problem. Summary of the Invention

[0005] Object of the Invention: Aiming at the deficiencies of the prior art, the present invention first provides a resource scheduling method for multi-user multi-modal flow integrity transmission tasks, which optimizes the multi-modal perception utility of multiple users through spectrum and power resource scheduling, and meets the integrity transmission requirements of multi-user audio-visual flows under the condition of considering the real-time transmission of multi-user tactile flows.

[0006] Another object of the present invention is to provide a resource scheduling system for multi-user multi-modal flow integrity transmission tasks to implement the above-mentioned resource scheduling method for multi-user multi-modal flow integrity transmission tasks.

[0007] For the above purposes, on the one hand, the present invention provides a resource scheduling method for multi-user multi-modal flow integrity transmission tasks, which is characterized by including the following steps:

[0008] S1. For multi-user multi-modal flow transmission tasks, obtain the audio-visual flow transmission rate and the haptic flow transmission delay;

[0009] S2. According to the audio-visual flow transmission rate and the haptic flow transmission delay, with the goal of maximizing the multi-user multi-modal perception utility, establish an optimization mathematical model for resource scheduling, and solve this model to obtain the resource quantity that maximizes the multi-user multi-modal perception utility under the constraints of spectrum and power resources;

[0010] S3. Allocate power and spectrum resources to multiple users according to the resource quantity required by the users.

[0011] As a preferred technical solution, step S1, for multi-user multi-modal flow transmission tasks, obtains the audio-visual flow transmission rate and the haptic flow transmission delay; it includes:

[0012] S11. For multi-user multi-modal flow transmission tasks, establish a transmission model for the haptic flow and the audio-visual flow;

[0013] S12. Calculate the audio-visual flow transmission rate and the haptic flow transmission delay.

[0014] Preferably, step S11, for multi-user multi-modal flow transmission tasks, establishes a transmission model for the haptic flow and the audio-visual flow, specifically including:

[0015] For the audio-visual flow, the duration of the user's audio-visual frame is seconds, and the time reserved for transmitting each audio-visual frame is time slots , and each time slot is further divided into mini-slots , the audio-visual flow is transmitted based on time slots, while the haptic flow is transmitted based on mini-slots;

[0016] The base station provides services to users within its coverage area, and the set of base stations is , the set of users is , each base station occupies a number of spectrum resource blocks, and some spectrum resource blocks are reused among base stations, and the set of spectrum resource blocks is represented as ;

[0017] Step S12 calculates the audio-visual flow transmission rate and the haptic flow transmission delay, specifically including:

[0018] The audio-visual flow transmission rate obtained by the user served by the base station in the time slot is expressed as:

[0019]

[0020] Among them, is the bandwidth of the spectrum resource block, is the time slot in which the user has the signal-to-interference-plus-noise ratio on the spectrum resource block , indicates whether the user occupies the spectrum resource block in the time slot , indicates that the user occupies the spectrum resource block in the time slot , indicates that the user does not occupy the spectrum resource block in the time slot ;

[0021] The tactile flow transmission rate obtained by the users served by the base station in the mini-slot is expressed as:

[0022]

[0023] Among them, is the signal-to-interference-plus-noise ratio of the user in the mini-slot on the spectrum resource block , indicates whether the user occupies the spectrum resource block in the mini-slot , indicates that the user occupies the spectrum resource block in the mini-slot , indicates that the user does not occupy the spectrum resource block in the mini-slot , is the channel dispersion and there is , is the block length of the tactile flow, is the inverse function of the function and there is ;

[0024] The size of the tactile data packet of the user in the mini-slot is expressed as , then the user in the mini-slot The tactile transmission delay within is expressed as:

[0025]

[0026] For the user The packet arrival rate of the tactile stream is expressed as , and the duration of the micro-slot is expressed as , then for the user within the time slot the transmission delay of all tactile data packets reached is expressed as:

[0027]

[0028] The tactile transmission performance of the user is represented by the tactile sensitivity coefficient. For the user within the time slot the tactile sensitivity coefficient is expressed as:

[0029]

[0030] Wherein, represents the maximum value of the tactile stream transmission delay, represents the minimum value of the tactile transmission delay.

[0031] Preferably, step S2 establishes an optimized mathematical model for resource scheduling with the goal of maximizing the multi-user multi-modal perception utility according to the audiovisual stream transmission rate and the tactile stream transmission delay, and solves this model to obtain the resource quantity that can maximize the multi-user multi-modal perception utility under the constraints of spectrum and power resources; including:

[0032] S21. Construct a user multi-modal perception utility function based on the multi-attribute utility theory, with the spectrum resource allocation strategy and the power allocation strategy as the optimization objects, with maximizing the multi-user multi-modal perception utility as the objective function, and with the user audiovisual frame size, spectrum resources, and power resource limitations as the constraint objects, to construct a resource scheduling model for the integrity transmission of multi-modal streams;

[0033] S22. Solve the resource scheduling model to obtain the resource quantity that maximizes the multi-user multi-modal perception utility under the constraints of spectrum and power resources.

[0034] Preferably, in step S21, the method for constructing the resource scheduling model for the integrity transmission of multi-modal streams is as follows:

[0035] S211. Construct a user multi-modal perception utility function based on the multi-attribute utility theory. The calculation expression of the multi-modal perception utility obtained by the user within the time slot is:

[0036]

[0037] Among them, is the transmission rate of the audiovisual stream for the user within the time slot ; is the transmission time of the audiovisual stream for the user within the time slot ; is the size of the audiovisual block requested by the user ; is the minimum value of the audiovisual block ; is the maximum value of the haptic stream transmission delay ; is the minimum value of the haptic stream transmission delay ;

[0038] S212. Establish a resource scheduling model representation optimization problem for the integrity transmission of multimodal streams:

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] Among them, is the set of base stations, is the number of base stations; is the set of users, is the number of users; is the set of spectrum resource blocks, is the total number of spectrum resource blocks; represents the number of spectrum resource blocks occupied by the base station ; is the set of time slots, is the total number of time slots prepared for transmitting each segment of the audiovisual frame; is the set of micro time slots included within the time slot ; is the number of micro time slots included in each time slot; is the spectrum resource scheduling strategy for the audiovisual stream, represents the spectrum resource block Allocated to the user within a time slot ; otherwise ; ; Represents the spectrum resource scheduling strategy for the tactile stream Represents the spectrum resource block Allocated to the user within a mini - time slot ; otherwise ; ; Represents the power allocation strategy of the user. The total transmit power of each base station within each time slot cannot exceed ; Represents the duration of each time slot; Represents the user Requested size of the audio - visual frame. Each audio - visual frame is divided into audio - visual blocks;

[0047] Step S22 solves the resource scheduling model to obtain the resource quantity that maximizes the multi - user multi - modal perception utility under spectrum and power resource constraints. The specific method is as follows:

[0048] First, based on the Lyapunov theory, the optimization problem in step S212 is decomposed into a multi - modal perception utility maximization problem for each time slot; then, the spectrum resources are allocated based on a one - to - many matching algorithm, the power resources are allocated based on the alternating direction multiplier method, and finally, a stable spectrum resource allocation strategy and power resource allocation strategy are obtained by combining the spectrum resource and power resource allocation algorithms.

[0049] Preferably, step S3 allocates power and spectrum resources to multiple users according to the resource quantity required by the users;

[0050] Among them, the transmission of the tactile stream adopts a resource puncturing method, that is, the transmission of the tactile stream occupies the spectrum resources already allocated to the audio - visual stream.

[0051] Preferably, when the current audio - visual frame of the user has been transmitted, the occupied spectrum and power resources are allocated to the users whose audio - visual frames have not been completely transmitted; at this time, the transmission of the tactile stream of the user occupies the spectrum of the audio - visual frame transmission of other users.

[0052] In the second aspect of the present invention, a resource scheduling system for multi - user multi - modal stream integrity transmission tasks is provided, including:

[0053] A transmission task acquisition module, used to obtain the audio - visual stream transmission rate and the transmission delay of the tactile stream for the multi - user multi - modal stream transmission task;

[0054] A transmission strategy optimization module, which is used to establish an optimized mathematical model for resource scheduling aiming at maximizing the multi-user multi-modal perception utility according to the audio-visual stream transmission rate and the tactile stream transmission delay, and solve the resource scheduling model to obtain the resource quantity that maximizes the multi-user multi-modal perception utility under the constraints of spectrum and power resources;

[0055] A transmission task allocation module, which is used to allocate power and spectrum resources to multiple users according to the resource quantity required by the users.

[0056] The present invention also provides a non-transitory computer-readable storage medium, on which computer instructions are stored, and the computer instructions enable a computer to execute the above-mentioned resource scheduling method for multi-user multi-modal stream integrity transmission tasks.

[0057] The present invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus;

[0058] Wherein, the processor, the communication interface, and the memory complete mutual communication through the communication bus;

[0059] The processor is used to call the logical instructions in the memory to execute the above-mentioned resource scheduling method for multi-user multi-modal stream integrity transmission tasks.

[0060] Beneficial effects: The resource scheduling method for multi-user multi-modal stream integrity transmission tasks proposed by the present invention comprehensively considers the real-time requirements of tactile stream transmission and the integrity requirements of audio-visual stream transmission, and improves the multi-modal perception utility of users in the virtual-real fusion service by allocating spectrum and power resources. Specifically, the present invention takes maximizing the multi-modal perception utility of multiple users as the objective function, and takes the user audio-visual frame size, spectrum resource, and power resource limitations as constraints to construct a resource scheduling model for multi-modal stream integrity transmission tasks. Based on the Lyapunov theory, the many-to-one matching theory, and the ADMM algorithm, the resource scheduling optimization model is solved to obtain the spectrum and power resource allocation strategies for multiple users under resource constraints. Using the obtained resource allocation strategies, the resource utilization rate can be further improved, and the multi-user immersive service experience can be enhanced. Description of the Drawings

[0061] Figure 1 It is a schematic diagram of a multi-cell multi-modal stream downlink transmission system in an embodiment of the present invention;

[0062] Figure 2 It is a flowchart of the resource scheduling method for multi-user multi-modal stream integrity transmission tasks provided in Embodiment 1 of the present invention;

[0063] Figure 3 It is a schematic diagram of the multi-modal stream video resource scheduling model provided in Embodiment 1 of the present invention;

[0064] Figure 4 It is a graph showing the relationship between the average multi-modal perception utility provided by the embodiments of the present invention and the number of users and the number of spectrum resource blocks.

[0065] Figure 5 It is a graph showing the relationship between the average multi-modal perception utility under different schemes provided by the embodiments of the present invention and the number of audio-visual blocks in each audio-visual frame. Detailed implementation manners

[0066] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. They should not be construed as limiting the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention. In the description of the present invention, it should be understood that the terms used are only for the purpose of description and cannot be construed as indicating or implying relative importance.

[0067] Next, with reference to Figures 1 - 5 Describe the resource scheduling method and system for the multi-user multi-modal stream integrity transmission task provided by the present invention.

[0068] Embodiment 1: This embodiment provides a resource scheduling method for the multi-user multi-modal stream integrity transmission task. This method is used for the transmission task based on the transmission system as shown in Figure 1 The system includes a user layer and a base station layer.

[0069] The user layer consists of several users waiting for services. The users request multi-modal content, including audio-visual streams and tactile streams, from the base station through the wireless uplink.

[0070] The base station layer consists of several base stations that provide multi-modal content for users. The base stations cache the multi-modal content requested by the users and send the audio-visual streams and tactile streams to the users through the wireless downlink.

[0071] Inter-cell interference is caused by multiple base stations sharing the same spectrum resource blocks.

[0072] This embodiment proposes a resource scheduling method for the multi-cell multi-user multi-modal stream integrity transmission task, which maximizes the multi-modal perception utility of users by using limited spectrum and power resources and improves the immersive experience of users. As shown in Figure 2 Generally speaking, it includes the steps:

[0073] S1. For the multi-user multi-modal stream transmission task, obtain the audio-visual stream transmission rate and the tactile stream transmission delay;

[0074] S2. Based on the audiovisual stream transmission rate and the haptic stream transmission delay, establish an optimized mathematical model for resource scheduling with the goal of maximizing the multi-user multi-modal perception utility, and solve this model to obtain the resource quantities that maximize the multi-user multi-modal perception utility under spectrum and power resource constraints;

[0075] S3. Allocate power and spectrum resources to multiple users according to the resource quantities required by the users.

[0076] Furthermore, step S1 obtains the audiovisual stream transmission rate and the haptic stream transmission delay for the multi-user multi-modal stream transmission task; it includes:

[0077] S11. For the multi-user multi-modal stream transmission task, establish a transmission model for the haptic stream and the audiovisual stream:

[0078] The transmission resource scheduling model for the multi-modal stream is as Figure 3 shown. The multi-modal stream mainly includes the audiovisual stream and the haptic stream. Due to the small size and burstiness of the haptic data packets, the transmission of the haptic stream can adopt a resource puncturing scheme, that is, the suddenly arriving haptic stream can occupy the transmission resources of the audiovisual stream;

[0079] For the audiovisual stream, the duration of the user's audiovisual frame is seconds, and the time reserved for transmitting each audiovisual frame is time slots . Each time slot is further divided into mini-slots . The audiovisual stream is transmitted based on time slots, while the haptic stream is transmitted based on mini-slots;

[0080] The base station provides services to the users within its coverage area. The set of base stations is , and the set of users is . Each base station occupies a number of spectrum resource blocks, and some spectrum resource blocks are reused among the base stations. The set of spectrum resource blocks is represented as ;

[0081] S12. Calculate the audiovisual stream transmission rate and the haptic stream transmission delay:

[0082] The audiovisual stream transmission rate obtained by the user served by the base station within the time slot is expressed as:

[0083]

[0084] where is the bandwidth of the spectrum resource block, and is the time slot and the user The signal-to-interference-plus-noise ratio on the spectrum resource block indicates whether the user occupies the spectrum resource block within the time slot ; indicates that the user occupies the spectrum resource block within the time slot ; indicates that the user does not occupy the spectrum resource block within the time slot ;

[0085] The tactile flow transmission rate obtained by the user served by the base station within the mini-slot is expressed as:

[0086]

[0087] where is the signal-to-interference-plus-noise ratio of the user on the spectrum resource block within the mini-slot ; indicates whether the user occupies the spectrum resource block within the mini-slot ; indicates that the user occupies the spectrum resource block within the mini-slot ; indicates that the user does not occupy the spectrum resource block within the mini-slot ; is the channel dispersion and has ; is the block length of the tactile flow, is the inverse function of the function and has ;

[0088] The size of the tactile data packet of the user within the mini-slot is expressed as , then the tactile transmission delay of the user within the mini-slot is expressed as:

[0089]

[0090] The packet arrival rate of the tactile flow of the user is expressed as ​, the duration of a micro-slot is expressed as , then the user in the time slot The transmission delay of all haptic data packets reached within is expressed as:

[0091]

[0092] The haptic transmission performance of the user is represented by the haptic sensitivity coefficient. The user in the time slot The haptic sensitivity coefficient within is expressed as:

[0093]

[0094] Among them, represents the maximum value of the haptic stream transmission delay, represents the minimum value of the haptic transmission delay.

[0095] Furthermore, step S2 establishes an optimized mathematical model for resource scheduling with the goal of maximizing the multi-user multi-modal perception utility based on the audiovisual stream transmission rate and the haptic stream transmission delay, and solves this model to obtain the resource quantity that can maximize the multi-user multi-modal perception utility under the constraints of spectrum and power resources; including:

[0096] S21. Construct a user multi-modal perception utility function based on the multi-attribute utility theory, use the spectrum resource allocation strategy and the power allocation strategy as the optimization objects, use the maximization of the multi-user multi-modal perception utility as the objective function, and use the user audiovisual frame size, spectrum resource, and power resource limitations as the constraint objects to construct a resource scheduling model for the integrity transmission of multi-modal streams; The specific method is:

[0097] S211. Construct a user multi-modal perception utility function based on the multi-attribute utility theory. Given the haptic sensitivity coefficient , the user in the time slot The conditional utility of the audiovisual experience within is expressed as:

[0098]

[0099] Among them, is the transmission time of the audiovisual stream of the user in the time slot , is the size of the audiovisual block requested by the user , is the minimum value of the audiovisual block;

[0100] S212. The calculation expression of the multi-modal perception utility obtained by the user in the time slot is:

[0101]

[0102] wherein, is the transmission rate of the audiovisual stream for the user within the time slot ; is the transmission time of the audiovisual stream for the user within the time slot ; is the size of the audiovisual block requested by the user ; is the minimum value of the audiovisual block is the maximum value of the transmission delay of the haptic stream is the minimum value of the transmission delay of the haptic stream is the transmission delay of the haptic stream for the user within the time slot ;

[0103] S213. Establish a resource scheduling model for the integrity transmission of multimodal streams to represent the optimization problem:

[0104]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111] wherein, is the set of base stations, is the number of base stations; is the set of users, is the number of users; is the set of spectrum resource blocks, is the total number of spectrum resource blocks; represents the number of spectrum resource blocks occupied by the base station ; is the set of time slots, is the total number of time slots prepared for transmitting each segment of the audiovisual frame; is the set of micro time slots included within the time slot ; is the number of micro time slots included within each time slot. For the spectrum resource scheduling strategy of the audiovisual stream, denotes the spectrum resource block allocated to the user within the time slot , otherwise ; ; For the spectrum resource scheduling strategy of the tactile stream, denotes the spectrum resource block allocated to the user within the micro time slot , otherwise ; ; Denotes the power allocation strategy of the user, and the total transmission power of each base station within each time slot cannot exceed ; Denotes the duration of each time slot; Denotes the user requested audiovisual frame size, and each audiovisual frame is divided into audiovisual blocks;

[0112] S22. Solve the resource scheduling model to obtain the resource quantity that maximizes the multi-user multi-modal perception utility under the constraints of spectrum and power resources;

[0113] The difficulty in solving the above maximization problem lies in that the integrity transmission of the audiovisual stream couples the spectrum and power resource allocation strategies of different time slots. In addition, the maximization problem of the multi-modal perception utility for each time slot is a complex non-convex integer non-linear programming problem.

[0114] S221. To solve the long-term resource allocation coupling problem in the integrity transmission of the audiovisual stream, a virtual audiovisual stream data volume remaining queue is constructed for the user based on the Lyapunov theory to guide the allocation of spectrum and power resources. The user virtual audiovisual stream data volume remaining queue has an initial value of 0 and is updated as follows in each time slot:

[0115]

[0116] In each time slot, only the current transmission data volume and channel state need to be obtained to formulate the resource allocation strategy. The optimization problem for each time slot is expressed as:

[0117]

[0118]

[0119]

[0120]

[0121]

[0122] Among them, is a control parameter for dynamically adjusting between maximizing the multi-modal perception utility and minimizing the transmission amount of residual data, , , and there is

[0123]

[0124] where is the maximum transmit power of the base station, is the power spectral density of the noise, is within the time slot from the base station to the user channel gain.

[0125] S222. To solve the optimization problem of each time slot, first design a two-time-scale spectrum resource allocation algorithm based on the one-to-many matching theory to optimize the spectrum resource allocation strategy and .

[0126] Set the preference list of users as , and the preference function of users is:

[0127]

[0128] The preference list of spectrum resource blocks is , and the preference function of spectrum resource blocks is .

[0129] The two-time-scale spectrum resource allocation algorithm based on the one-to-many matching theory in step S222 includes the following steps:

[0130] a. Given the initial system values: matching between users and resource blocks , the set of users with audio-visual stream transmission tasks , the set of users whose audio-visual stream tasks have ended , the preference list of users , the preference list of spectrum resource blocks ;

[0131] b. Perform user matching for each spectrum resource block occupied by the base station ;

[0132] c. For the spectrum resource block occupied by the base station , based on the existing matching results of users and resource blocks and the preference list of users Update the spectrum resource block 's preference list and delete the users existing in the set from ;

[0133] d. When the most preferred user of the spectrum resource block is not the currently matched user , if the spectrum resource block allocated to the user has not reached the upper limit, the spectrum resource block is allocated to the user , otherwise the user queries for a spectrum resource block with a preference value lower than that of in the spectrum resource blocks it occupies and accepts and rejects . If the preference value of the spectrum resource block at the user is the lowest, the user rejects the spectrum resource block ;

[0134] e. Until a stable matching result is found for the spectrum resource block , the loop ends;

[0135] f. For the users belonging to the set , since the audio-visual stream transmission task ends, no spectrum resources and power resources are allocated, and their haptic transmission tasks occupy the spectrum and power resources of the users in ;

[0136] g. Until a stable matching result is found for the users and spectrum resource blocks of each base station, the loop ends.

[0137] S223. To solve the power optimization problem, the Alternating Direction Method of Multipliers (ADMM) is used to minimize the remaining queue length of the audio-visual data volume. The power optimization problem is expressed as

[0138]

[0139]

[0140] where represents the number of users served by the base station .

[0141] To obtain the optimal power allocation scheme, the power allocation variables of each base station are divided into two parts. The base station The power allocation strategy is expressed as and . During time slot , the total length of the data queues of the users served by the base station is expressed as . The problem of minimizing the queue length of the base station is expressed as:

[0142]

[0143]

[0144] where , , , , and are unit vectors. For the scaled ADMM algorithm, the augmented Lagrangian related to the above queue length minimization problem is expressed as:

[0145]

[0146] where is the penalty parameter, is the Lagrange multiplier, is the scaled dual variable.

[0147] The iterations of the primal and dual variables are expressed as:

[0148]

[0149]

[0150]

[0151] where is the number of iterations.

[0152] The primal residual is , the dual residual is , and the termination criterion for the ADMM-based power allocation algorithm is and , where is the predefined threshold for the primal condition, is the predefined threshold for the dual condition.

[0153] The ADMM-based multi-modal transmission power allocation algorithm is expressed as:

[0154] a. The spectrum resource allocation strategy for a given user and ;

[0155] b. For each base station, when the following conditions are met and are satisfied, update , , , and ;

[0156] c. , until the loop ends.

[0157] S224. Combine the two-time-scale spectrum resource allocation algorithm based on the one-to-many matching theory and the multi-modal transmission power allocation algorithm based on ADMM. For each time slot during the transmission of each video frame, repeatedly execute the spectrum resource allocation algorithm and the transmission power allocation algorithm until a stable spectrum resource and power resource allocation strategy is obtained.

[0158] Furthermore, in step S3, when allocating power and spectrum resources to multiple users according to the resource quantity required by the users, the transmission of the tactile stream adopts the resource puncturing method, that is, the transmission of the tactile stream occupies the spectrum resources already allocated to the audio-visual stream.

[0159] When the current audio-visual frame of the user has been completely transmitted, the spectrum and power resources it occupies are allocated to the users whose audio-visual frames have not been completely transmitted; at this time, the transmission of the tactile stream of the user occupies the spectrum of the audio-visual frame transmission of other users.

[0160] Embodiment 2: To apply the above resource scheduling method for the multi-user multi-modal stream integrity transmission task, this embodiment provides a resource scheduling system for the multi-user multi-modal stream integrity transmission task.

[0161] A transmission task acquisition module, which is used to acquire the audio-visual stream transmission rate and the transmission delay of the tactile stream for the multi-user multi-modal stream transmission task;

[0162] A transmission strategy optimization module, which is used to establish an optimization mathematical model for resource scheduling with the goal of maximizing the multi-user multi-modal perception utility according to the audio-visual stream transmission rate and the tactile stream transmission delay, and solve the resource scheduling model to obtain the resource quantity that maximizes the multi-user multi-modal perception utility under the constraints of spectrum and power resources;

[0163] A transmission task allocation module, which is used to allocate power and spectrum resources to multiple users according to the resource quantity required by the users.

[0164] The working processes and working principles of each module are the same as those in Embodiment 1, and will not be repeated in this embodiment.

[0165] Example 3: This example provides a non-transitory computer-readable storage medium with computer instructions stored thereon. These computer instructions cause a computer to execute a resource scheduling method for a multi-user multi-modal stream integrity transmission task. The method includes the following steps:

[0166] S1. For a multi-user multi-modal stream transmission task, obtain the audio-visual stream transmission rate and the tactile stream transmission delay;

[0167] S2. Based on the audio-visual stream transmission rate and the tactile stream transmission delay, with the goal of maximizing the multi-user multi-modal perception utility, establish an optimization mathematical model for resource scheduling, and solve this model to obtain the resource quantity that maximizes the multi-user multi-modal perception utility under spectrum and power resource constraints;

[0168] S3. Allocate power and spectrum resources to multiple users according to the resource quantity required by the users.

[0169] Example 4: This example provides an electronic device, which may include: a processor, a communications interface, a memory 630, and a communication bus. Among them, the processor, the communications interface, and the memory communicate with each other through the communication bus. The processor can call the logical instructions in the memory to execute a resource scheduling method for a multi-user multi-modal stream integrity transmission task. The method includes the following steps:

[0170] S1. For a multi-user multi-modal stream transmission task, obtain the audio-visual stream transmission rate and the tactile stream transmission delay;

[0171] S2. Based on the audio-visual stream transmission rate and the tactile stream transmission delay, with the goal of maximizing the multi-user multi-modal perception utility, establish an optimization mathematical model for resource scheduling, and solve this model to obtain the resource quantity that maximizes the multi-user multi-modal perception utility under spectrum and power resource constraints;

[0172] S3. Allocate power and spectrum resources to multiple users according to the resource quantity required by the users.

[0173] In addition, when the logical instructions in the above-mentioned memory are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0174] Embodiment 5: This embodiment provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a resource scheduling method for a multi-user multi-modal flow integrity transmission task. The method includes the following steps:

[0175] S1. For a multi-user multi-modal flow transmission task, obtain the audio-visual flow transmission rate and the tactile flow transmission delay;

[0176] S2. According to the audio-visual flow transmission rate and the tactile flow transmission delay, with the goal of maximizing the multi-user multi-modal perception utility, establish an optimization mathematical model for resource scheduling, and solve this model to obtain the resource quantity that maximizes the multi-user multi-modal perception utility under the constraints of spectrum and power resources;

[0177] S3. Allocate power and spectrum resources to multiple users according to the resource quantity required by the users.

[0178] Based on the above various embodiments, in order to verify the resource scheduling method and system for the multi-user multi-modal flow integrity transmission task of the present invention, a simulation experiment is carried out to verify the effect of resource scheduling in the transmission task of the system as shown in Figure 1 the figure.

[0179] The simulation parameters are set as follows: The total number of base stations is 3, the total number of users is 30, the frame rate of the user audio-visual flow is 60 frames per second, the set of user audio-visual frame sizes is , the size of each pixel is 12 bits, and the duration of the time slot is 1 millisecond, and the duration of a micro-slot is 1 / 7 millisecond. The proposed resource scheduling method for multi-user multi-modal flow integrity transmission tasks is compared with non-integrity transmission schemes, single-slot transmission schemes, and average resource allocation schemes. In the non-integrity transmission scheme, the integrity of user audio-visual stream transmission is not considered. In the single-slot transmission scheme, resources are scheduled only based on time slots without considering micro-slots. In the average resource allocation transmission, the base station evenly allocates spectrum and power resources to users.

[0180] As Figure 4 shown, the relationships between the total number of users, the total number of spectrum resource blocks, and the average multi-modal perception utility of users are presented respectively. The average multi-modal perception utility of users decreases as the total number of users increases because the increase in the number of users simultaneously increases the reuse rate of inter-cell spectrum resource blocks, resulting in increased inter-cell interference, thus reducing the multi-modal perception utility. The average multi-modal perception utility of users increases as the number of spectrum resource blocks increases. The increase in the number of spectrum resource blocks promotes the improvement of multi-modal flow transmission performance, thereby enhancing the multi-modal perception utility. Therefore, the service requirements of user multi-modal services can be met by adjusting the number of spectrum resource blocks and the number of users served by the base station.

[0181] As Figure 5 shown, under all schemes, the average multi-modal perception utility of users increases as the number of audio-visual blocks included in each audio-visual frame increases because the increase in the number of audio-visual blocks included in the audio-visual frame can reduce the impact of integrity transmission requirements on audio-visual stream transmission. The resource scheduling scheme proposed in the present invention is superior to other benchmark algorithms in terms of the average multi-modal perception utility of users because the present invention simultaneously considers the real-time transmission requirements of tactile streams and the integrity transmission characteristics of audio-visual streams, further improving the utilization rate of spectrum resources and power resources and promoting the efficient transmission of multi-modal streams, thereby enhancing the multi-modal perception utility of users. Therefore, when considering the integrity transmission of audio-visual streams, the number of audio-visual blocks included in the audio-visual frame should be appropriately set to ensure the multi-modal immersive service experience of users with a relatively low audio-visual frame processing complexity.

[0182] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as a limitation of the present invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the present invention defined by the appended claims.

Claims

1. A resource scheduling method for multi-user multi-modal stream integrity transmission tasks, characterized in that: The steps include: S1. For multi-user multi-modal streaming tasks, obtain the audio-visual streaming transmission rate and the tactile streaming transmission delay; S2. According to the audiovisual stream transmission rate and the tactile stream transmission delay, with the goal of maximizing the multi-user multimodal perception utility, an optimization mathematical model for resource scheduling is established, and the model is solved to obtain the number of resources that maximize the multi-user multimodal perception utility under the constraints of spectrum and power resources; including: S21. Based on the multi-attribute utility theory, a user multimodal perception utility function is constructed. The spectrum resource allocation strategy and the power allocation strategy are used as optimization objects. The maximization of multimodal perception utility of multiple users is used as the objective function. The user audiovisual frame size, spectrum resources and power resource restrictions are used as constraints. A resource scheduling model for the integrity transmission of multimodal streams is constructed. The resource scheduling model for the integrity transmission of multimodal streams is constructed, which specifically includes: S211. Constructing user multimodal perceived utility function based on multi-attribute utility theory. In time slot The calculation expression of the multimodal perceived utility obtained is: ； in, For users In time slot The transmission rate of the internal audio and video stream, For users In time slot The transmission time of the internal audio-visual stream, For users The size of the requested audio-visual chunk, is the minimum value of the audio-visual block, is the maximum value of the tactile stream transmission delay, is the minimum value of the tactile stream transmission delay, For users In time slot Transmission delay of the internal tactile stream; S212. Establish a resource scheduling model for multimodal stream integrity transmission to represent the optimization problem: ； ； ； ； ； ； ； in, is the set of base stations, is the number of base stations; is a collection of users, is the number of users; is a collection of spectrum resource blocks, is the total number of spectrum resource blocks; Indicates base station The number of occupied spectrum resource blocks; is the set of time slots, is the total number of time slots prepared for the transmission of each audiovisual frame; For time slot The collection of mini-slots contained in is the number of mini-slots contained in each time slot; Spectrum resource scheduling strategy for audio-visual streams, Indicates spectrum resource block In time slot Assigned to users ,otherwise ; represents the spectrum resource scheduling strategy of tactile flow, Indicates spectrum resource block In micro-slot Assigned to users ,otherwise ; Indicates the user's power allocation strategy. The total transmission power of each base station in each time slot cannot exceed ; Indicates the duration of each time slot; Indicates user The requested audio-visual frame size. Each audio-visual frame is divided into audio-visual blocks; S22, solving the resource scheduling model to obtain the number of resources that maximizes the multi-user multi-modal sensing utility under the constraints of spectrum and power resources, specifically including: Firstly, based on Lyapunov theory, the optimization problem of step S212 is decomposed into a multimodal sensing utility maximization problem for each time slot; then, spectrum resources are allocated based on a one-to-many matching algorithm, and power resources are allocated based on an alternating direction multiplier method; finally, a stable spectrum resource allocation strategy and power resource allocation strategy are obtained by combining the spectrum resource and power resource allocation algorithms; S3. Allocate power and spectrum resources to multiple users according to the number of resources required by the users.

2. The resource scheduling method for multi-user multi-modal stream integrity transmission task according to claim 1 is characterized in that: Step S1 obtains the audiovisual stream transmission rate and the tactile stream transmission delay for a multi-user multimodal stream transmission task; including: S11. For multi-user multi-modal stream transmission tasks, a transmission model of tactile stream and audio-visual stream is established; S12. Calculate the transmission rate of the audiovisual stream and the transmission delay of the tactile stream.

3. The resource scheduling method for multi-user multi-modal stream integrity transmission task according to claim 2 is characterized in that: Step S11 establishes a transmission model of tactile stream and audio-visual stream for the multi-user multi-modal stream transmission task, specifically including: For audio-visual streams, the duration of the user's audio-visual frame is seconds, the time reserved for each audio-visual frame transmission is Time slots , each time slot is divided into Mini-slots ,The audio-visual stream is transmitted on a time-slot basis, while the tactile stream is transmitted on a micro-time-slot basis; The base station provides services to users within the coverage area. The set of base stations is , the user set is , each base station occupies a number of spectrum resource blocks, and some spectrum resource blocks are reused between base stations. The set of spectrum resource blocks is expressed as ; Step S12 calculates the audiovisual stream transmission rate and the tactile stream transmission delay, specifically including: Base Station Users of the service In time slot The audiovisual stream transmission rate obtained within is expressed as: ； in, is the bandwidth of the spectrum resource block, For time slot Internal User In spectrum resource blocks The signal-to-interference-noise ratio on Indicates user In time slot Whether the spectrum resource block is occupied Indicates that the user is in the time slot Occupied spectrum resource blocks Indicates that the user is in the time slot Unoccupied spectrum resource blocks ; Base Station Users of the service In micro-slot The tactile flow transmission rate obtained within is expressed as: ； in, Mini-slot Internal User In spectrum resource blocks The signal-to-interference-noise ratio on Indicates user In micro-slot Whether the spectrum resource block is occupied , Indicates user In micro-slot Occupied spectrum resource blocks , Indicates user In micro-slot Unoccupied spectrum resource blocks , is the channel dispersion and , is the block length of the tactile stream, is a function The inverse function of ; The user In micro-slot The haptic data packet size is expressed as , then the user In micro-slot The tactile transmission delay within is expressed as: ； The user The packet arrival rate of the tactile flow is expressed as , the duration of the mini-slot is expressed as , then the user In time slot The transmission delay of all tactile data packets arriving within is expressed as: ； The user's tactile transmission performance is expressed by the tactile sensitivity coefficient. In time slot The tactile sensitivity coefficient is expressed as: ； in, Indicates the maximum transmission delay of tactile stream, Indicates the minimum tactile transmission delay.

4. The resource scheduling method for multi-user multi-modal stream integrity transmission task according to claim 1, characterized in that: Step S3 allocates power and spectrum resources to multiple users according to the number of resources required by the users; The transmission of the tactile stream adopts a resource puncture method, that is, the transmission of the tactile stream occupies the spectrum resources that have been allocated to the audio-visual stream.

5. The resource scheduling method for multi-user multi-modal stream integrity transmission task according to claim 4 is characterized in that: In the user When the current audiovisual frame has been transmitted, the spectrum and power resources it occupies are allocated to the user whose audiovisual frame has not been transmitted yet; at this time, the user The tactile streaming transmission occupies the spectrum of other users' audio-visual frame transmission.

6. A resource scheduling system for multi-user multi-modal stream integrity transmission tasks, characterized in that: include: A transmission task acquisition module, used to obtain the transmission rate of the audiovisual stream and the transmission delay of the tactile stream for a multi-user multimodal stream transmission task; The transmission strategy optimization module is used to establish an optimization mathematical model for resource scheduling based on the audio-visual stream transmission rate and the tactile stream transmission delay, with the goal of maximizing the multi-user multi-modal perception utility, and solve the optimization mathematical model for resource scheduling to obtain the number of resources that maximize the multi-user multi-modal perception utility under the constraints of spectrum and power resources; the transmission strategy optimization module is specifically used to implement the following steps: S21. Based on the multi-attribute utility theory, a user multimodal perception utility function is constructed. The spectrum resource allocation strategy and the power allocation strategy are used as optimization objects. The maximization of multimodal perception utility of multiple users is used as the objective function. The user audiovisual frame size, spectrum resources and power resource restrictions are used as constraints. A resource scheduling model for the integrity transmission of multimodal streams is constructed. The resource scheduling model for the integrity transmission of multimodal streams is constructed, which specifically includes: S211. Constructing user multimodal perceived utility function based on multi-attribute utility theory. In time slot The calculation expression of the multimodal perceived utility obtained is: ； in, For users In time slot The transmission rate of the internal audio and video stream, For users In time slot The transmission time of the internal audio-visual stream, For users The size of the requested audio-visual chunk, is the minimum value of the audio-visual block, is the maximum value of the tactile stream transmission delay, is the minimum value of the tactile stream transmission delay, For users In time slot Transmission delay of the internal tactile stream; S212. Establish a resource scheduling model for multimodal stream integrity transmission to represent the optimization problem: ； ； ； ； ； ； ； in, is the set of base stations, is the number of base stations; is a collection of users, is the number of users; is a collection of spectrum resource blocks, is the total number of spectrum resource blocks; Indicates base station The number of occupied spectrum resource blocks; is the set of time slots, is the total number of time slots prepared for the transmission of each audiovisual frame; For time slot The collection of mini-slots contained in is the number of mini-slots contained in each time slot; Spectrum resource scheduling strategy for audio-visual streams, Indicates spectrum resource block In time slot Assigned to users ,otherwise ; represents the spectrum resource scheduling strategy of tactile flow, Indicates spectrum resource block In micro-slot Assigned to users ,otherwise ; Indicates the user's power allocation strategy. The total transmission power of each base station in each time slot cannot exceed ; Indicates the duration of each time slot; Indicates user The requested audio-visual frame size. Each audio-visual frame is divided into audio-visual blocks; S22, solving the resource scheduling model to obtain the number of resources that maximizes the multi-user multi-modal sensing utility under the constraints of spectrum and power resources, specifically including: Firstly, based on Lyapunov theory, the optimization problem of step S212 is decomposed into a multimodal sensing utility maximization problem for each time slot; then, spectrum resources are allocated based on a one-to-many matching algorithm, and power resources are allocated based on an alternating direction multiplier method; finally, a stable spectrum resource allocation strategy and power resource allocation strategy are obtained by combining the spectrum resource and power resource allocation algorithms; The transmission task allocation module is used to allocate power and spectrum resources to multiple users according to the number of resources required by the users.

7. A non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions enable a computer to execute the resource scheduling method for a multi-user multi-modal stream integrity transmission task as described in any one of claims 1 to 5.

8. An electronic device comprising a processor, a communication interface, a memory and a communication bus; in, The processor, the communication interface, and the memory communicate with each other via a communication bus; The processor is used to call the logic instructions in the memory to execute the resource scheduling method for the multi-user multi-modal stream integrity transmission task described in any one of claims 1 to 5.

Images