Satellite service combination selection method and system based on reinforcement learning multi-agent collaboration

Abstract

The application discloses a satellite service combination selection method and system based on reinforcement learning multi-agent cooperation, comprising the following steps: determining the residual resource values of each sample satellite corresponding to the sample satellites respectively, and the sample service item resource demand values corresponding to each sample service item in the sample service item set respectively; inputting the sample service item set, the sample satellite residual resource values and the sample service item resource demand values into a first multi-agent reinforcement learning network model to obtain first joint action values output by the first multi-agent reinforcement learning network model; calculating loss values of the first multi-agent reinforcement learning network model based on the sample service item resource demand values and the first joint action values; and adjusting first parameters of the first multi-agent reinforcement learning network model based on the loss values to obtain a target multi-agent reinforcement learning network model.

Description

Technical Field

[0001] This application belongs to the field of machine learning technology, specifically relating to a satellite service combination selection method and system based on reinforcement learning multi-agent collaboration. Background Technology

[0002] Currently, the satellite network field is developing rapidly. Due to its coverage capabilities in remote areas, the application scenarios of satellite networks are becoming increasingly widespread, such as communication, weather forecasting, and navigation. Generally, each satellite can handle multiple services. When a user sends a command to a satellite, services on different satellites are invoked to fulfill the request. After receiving this information, the user equipment calculates the appropriate combination of satellite services across satellites. Subsequently, the equipment transmits the calculation results back to the access satellite, which coordinates with other satellites in the constellation to allocate services to each satellite according to the determined satellite service combination, so that each satellite can execute the service. The process of determining the satellite service combination usually requires a certain amount of computing resources. Since each satellite provides limited resources, it is necessary to allocate the resources provided by the satellites reasonably so that the satellites can execute as many services as possible with as few resources as possible.

[0003] In related technologies, heuristic algorithms, such as simulated annealing or genetic algorithms, can be used for optimal selection. First, a satellite service combination is determined by combining satellites and service items. Then, simulated annealing or genetic algorithms are used to select the optimal satellite service combination.

[0004] However, the satellite service combination obtained by this method is often not the optimal combination in reality, and the utilization rate of satellite resources is low. Summary of the Invention

[0005] This application aims to provide a model training method, a satellite service combination selection method based on reinforcement learning multi-agent collaboration, a system, an electronic device, and a readable storage medium to solve the problem of low satellite resource utilization in the prior art.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a model training method, the method comprising:

[0008] Determine the remaining resource value of each sample satellite and the resource requirement value of each sample service item in the sample service item set.

[0009] The sample service item set, the remaining resource value of the sample satellite, and the resource requirement value of the sample service item are input into the first multi-agent reinforcement learning network model to obtain the first joint action value output by the first multi-agent reinforcement learning network model.

[0010] Based on the resource requirement values ​​of each of the sample service items and the values ​​of each of the first joint actions, the loss value of the first multi-agent reinforcement learning network model is calculated.

[0011] Based on the loss value, the first parameter of the first multi-agent reinforcement learning network model is adjusted to obtain the target multi-agent reinforcement learning network model; the target multi-agent reinforcement learning network model is used to determine the target joint action value between the target satellite and the target service item.

[0012] Optionally, the first multi-agent reinforcement learning network model includes an encoding module and a joint action value calculation module. The step of inputting the sample service item set, the sample satellite remaining resource value, and the sample service item resource requirement value into the first multi-agent reinforcement learning network model to obtain the first joint action value output by the first multi-agent reinforcement learning network model includes:

[0013] Input the set of sample service items, the remaining resource value of the sample satellite, and the resource requirement value of the sample service item into the encoding module to obtain the first sample remaining resource observation value and the first sample resource requirement observation value output by the encoding module;

[0014] The remaining resource value of the sample satellite and the resource requirement value of the sample service item are concatenated to obtain the state vector corresponding to each sample service item;

[0015] The state vector, the observation value of the remaining resources of the first sample, and the observation value of the resource demand of the first sample are input into the joint action value calculation module to obtain the first joint action value output by the joint action value calculation module.

[0016] Optionally, the step of inputting the sample service item set, the sample satellite remaining resource value, and the sample service item resource requirement value into the encoding module to obtain the sample remaining resource observation value and sample resource requirement observation value output by the encoding module includes:

[0017] The sample service item set is concatenated with the sample satellite remaining resource value to obtain the first concatenation vector;

[0018] The sample service item set and the sample service item resource requirement value are concatenated to obtain the second concatenation vector;

[0019] The first concatenated vector and the second concatenated vector are respectively input into the encoding module to obtain the first sample remaining resource observation value and the first sample resource demand observation value output by the encoding module.

[0020] Optionally, the step of inputting the first concatenated vector and the second concatenated vector into the encoding module to obtain the first sample remaining resource observation value and the first sample resource demand observation value output by the encoding module respectively includes:

[0021] Based on the remaining resource values ​​of the sample satellites and the resource requirements of the sample service items, a mask matrix corresponding to each sample satellite and each sample service item is determined; wherein, the row number of the mask matrix corresponds to each sample satellite, the column number of the mask matrix corresponds to each sample service item, and the first element value of the mask matrix indicates whether the sample service item corresponding to the first element value is allowed to run on the sample satellite corresponding to the first element value.

[0022] The first concatenated vector and the second concatenated vector are respectively input into the encoding module to obtain the second sample remaining resource observation value and the second sample resource demand observation value output by the encoding module respectively;

[0023] The mask matrix is ​​input into the encoding module to obtain the mask matrix encoding output by the encoding module;

[0024] The mask matrix encoding and the second sample remaining resource observation value are concatenated to obtain the first sample remaining resource observation value. The mask matrix encoding and the second sample resource demand observation value are concatenated to obtain the first sample resource demand observation value.

[0025] Optionally, adjusting the first parameters of the first multi-agent reinforcement learning network model based on the loss value to obtain the target multi-agent reinforcement learning network model includes:

[0026] Based on the loss value, the first parameter of the first multi-agent reinforcement learning network model is adjusted to obtain the first multi-agent reinforcement learning network model with updated parameters.

[0027] Repeat the steps of determining the remaining resource value of each sample satellite and the resource requirement value of each sample service item in the sample service item set.

[0028] Compare the loss value with a first threshold, and accumulate the first number of times the loss value is less than the first threshold;

[0029] If the first number of attempts is greater than or equal to the second threshold, training of the first multi-agent reinforcement learning network model is stopped, and the target multi-agent reinforcement learning network model is obtained.

[0030] Secondly, embodiments of this application provide a satellite service combination selection method based on reinforcement learning multi-agent cooperation, the method comprising:

[0031] Determine the remaining resource value of the target satellite and the resource requirement value of each target service item included in the target service item set;

[0032] The target service item set, the remaining resource value, and the resource demand value are input into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model; wherein, the target multi-agent reinforcement learning network model is obtained based on the above training method;

[0033] Based on the resource requirement value and the target joint action value, the target satellite and the target service item are combined to obtain the target satellite service combination.

[0034] Optionally, the step of inputting the target service item set, the remaining resource value, and the resource demand value into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model includes:

[0035] The remaining resource value is concatenated with the target service item set to obtain the first concatenation vector;

[0036] The resource requirement value is concatenated with the target service item set to obtain a second concatenation vector;

[0037] The first concatenated vector and the second concatenated vector are input into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model.

[0038] Optionally, the step of combining the target satellite and the target service item based on the resource requirement value and the target joint action value to obtain a target satellite service combination includes:

[0039] Determine the first selection probability of the target service item corresponding to the resource requirement value;

[0040] Determine the second selection probability of the target satellite corresponding to the target joint action value;

[0041] Arrange the first selection probability and the second selection probability in descending order to obtain the first selection probability sequence and the second selection probability sequence;

[0042] The target satellite and target service item corresponding to the sequence values ​​at the same position in the first selection probability sequence and the second selection probability sequence are combined to obtain the target satellite service combination.

[0043] Thirdly, embodiments of this application provide a model training system, the system comprising:

[0044] The determination module is used to determine the remaining resource value of each sample satellite and the resource requirement value of each sample service item in the sample service item set.

[0045] The input module is used to input the sample service item set, the sample satellite remaining resource value and the sample service item resource requirement value into the first multi-agent reinforcement learning network model to obtain the first joint action value output by the first multi-agent reinforcement learning network model;

[0046] The calculation module is used to calculate the loss value of the first multi-agent reinforcement learning network model based on the resource requirement value of each of the sample service items and the value of each of the first joint actions.

[0047] An adjustment module is used to adjust the first parameter of the first multi-agent reinforcement learning network model based on the loss value to obtain a target multi-agent reinforcement learning network model; the target multi-agent reinforcement learning network model is used to determine the target joint action value between the target satellite and the target service item.

[0048] Optionally, the first multi-agent reinforcement learning network model includes an encoding module and a joint action value calculation module, and the input module includes:

[0049] The first input submodule is used to input the sample service item set, the sample satellite remaining resource value and the sample service item resource requirement value into the encoding module to obtain the first sample remaining resource observation value and the first sample resource requirement observation value output by the encoding module.

[0050] The splicing submodule is used to splice the remaining resource value of the sample satellite and the resource requirement value of the sample service item to obtain the state vector corresponding to each sample service item;

[0051] The second input submodule is used to input the state vector, the observation value of the remaining resources of the first sample, and the observation value of the resource demand of the first sample into the joint action value calculation module to obtain the first joint action value output by the joint action value calculation module.

[0052] Optionally, the first input submodule includes:

[0053] The first splicing unit is used to splice the sample service item set with the sample satellite remaining resource value to obtain the first splicing vector;

[0054] The second splicing unit is used to splice the sample service item set with the sample service item resource requirement value to obtain a second splicing vector;

[0055] The input unit is used to input the first concatenation vector and the second concatenation vector into the encoding module respectively, so as to obtain the first sample remaining resource observation value and the first sample resource demand observation value output by the encoding module respectively.

[0056] Optionally, the input unit includes:

[0057] A determining subunit is used to determine a mask matrix corresponding to each sample satellite and each sample service item based on the remaining resource value of the sample satellite and the resource requirement value of the sample service item; wherein, the row number of the mask matrix corresponds to each sample satellite, the column number of the mask matrix corresponds to each sample service item, and the first element value of the mask matrix indicates whether the sample service item corresponding to the first element value is allowed to run on the sample satellite corresponding to the first element value.

[0058] The first input subunit is used to input the first concatenation vector and the second concatenation vector into the encoding module respectively, so as to obtain the second sample remaining resource observation value and the second sample resource demand observation value output by the encoding module respectively;

[0059] The second input subunit is used to input the mask matrix into the encoding module to obtain the mask matrix encoding output by the encoding module;

[0060] The splicing subunit is used to splice the mask matrix encoding and the second sample remaining resource observation value to obtain the first sample remaining resource observation value, and to splice the mask matrix encoding and the second sample resource demand observation value to obtain the first sample resource demand observation value.

[0061] Optionally, the adjustment module includes:

[0062] The adjustment submodule is used to adjust the first parameter of the first multi-agent reinforcement learning network model based on the loss value, so as to obtain the first multi-agent reinforcement learning network model with updated parameters.

[0063] The execution submodule is used to re-execute the steps of determining the remaining resource value of each sample satellite and the resource requirement value of each sample service item in the sample service item set.

[0064] The comparison and accumulation submodule is used to compare the loss value with the first threshold and accumulate the first number of times the loss value is less than the first threshold;

[0065] The stopping submodule is used to stop training the first multi-agent reinforcement learning network model when the first number of times is greater than or equal to the second threshold, so as to obtain the target multi-agent reinforcement learning network model.

[0066] Fourthly, embodiments of this application provide a satellite service combination selection system based on reinforcement learning multi-agent cooperation, the system comprising:

[0067] The determination module is used to determine the remaining resource value of the target satellite and the resource requirement value of each target service item included in the target service item set;

[0068] The input module is used to input the target service item set, the remaining resource value, and the resource demand value into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model; wherein, the target multi-agent reinforcement learning network model is obtained based on any of the model training methods described above;

[0069] The combination module is used to combine the target satellite and the target service item based on the resource requirement value and the target joint action value to obtain the target satellite service combination.

[0070] Optionally, the input module includes:

[0071] The first splicing submodule is used to splice the remaining resource value with the target service item set to obtain a first splicing vector;

[0072] The second splicing submodule is used to splice the resource requirement value with the target service item set to obtain a second splicing vector;

[0073] The input submodule is used to input the first concatenated vector and the second concatenated vector into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model.

[0074] Optionally, the combined module includes:

[0075] The first determining submodule is used to determine the first selection probability of the target service item corresponding to the resource requirement value;

[0076] The second determining submodule is used to determine the second selection probability of the target satellite corresponding to the target joint action value;

[0077] The permutation submodule is used to arrange each of the first selection probabilities and each of the second selection probabilities in descending order to obtain a first selection probability sequence and a second selection probability sequence;

[0078] The combination submodule is used to combine the target satellites and target service items corresponding to the sequence values ​​at the same position in the first selection probability sequence and the second selection probability sequence to obtain the target satellite service combination.

[0079] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the model training method described above, or the satellite service combination selection method based on reinforcement learning multi-agent collaboration.

[0080] Sixthly, embodiments of this application provide a readable storage medium storing a program or instructions, which, when executed by a processor, implement the model training method described above, or the satellite service combination selection method based on reinforcement learning multi-agent collaboration.

[0081] Compared to existing technologies, this application determines the remaining resource value of each sample satellite and the resource requirement value of each sample service item in the sample service item set. The sample service item set, the remaining resource value of the sample satellite, and the resource requirement value of the sample service item are input into a first multi-agent reinforcement learning network model to obtain the first joint action value output by the first multi-agent reinforcement learning network model. Based on the resource requirement value of each sample service item and each first joint action value, the loss value of the first multi-agent reinforcement learning network model is calculated. Based on the loss value, the first parameter of the first multi-agent reinforcement learning network model is adjusted to obtain the target multi-agent reinforcement learning network model. The target multi-agent reinforcement learning network model is used to determine the target joint action value between the target satellite and the target service item, resulting in a more accurate target multi-agent reinforcement learning network model with more accurate output joint action values. When using the target joint action value output by the target multi-agent reinforcement learning network model to match and combine target satellites and target service items to obtain target satellite service combinations, it ensures that all target service items are matched with available target satellites, reduces the satellite resources occupied by the target service items, and improves the utilization rate of satellite resources.

[0082] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0083] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0084] Figure 1 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0085] Figure 2 A flowchart illustrating a satellite service combination selection method based on reinforcement learning multi-agent collaboration, provided for an embodiment of this application;

[0086] Figure 3 Histogram comparing the experimental performance of a target multi-agent reinforcement learning network model provided in this application embodiment;

[0087] Figure 4 A schematic diagram illustrating a satellite service combination method provided in an embodiment of this application;

[0088] Figure 5 A logic block diagram of a model training system provided in an embodiment of this application;

[0089] Figure 6 A logical block diagram of a satellite service composition selection system based on reinforcement learning multi-agent cooperation provided in this application embodiment;

[0090] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0091] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0092] Reference Figure 1 , Figure 1 This is a flowchart illustrating a satellite service combination selection method and system based on reinforcement learning multi-agent cooperation, as provided in an embodiment of this application. The method may include:

[0093] Step 101: Determine the remaining resource value of each sample satellite and the resource requirement value of each sample service item in the sample service item set.

[0094] In the embodiments of this application, a satellite constellation may contain multiple satellites for collaboratively executing a set of service items consisting of multiple service items. To train the first multi-agent reinforcement learning network model, existing satellite constellation data can be sampled to obtain a training dataset, such as the TianSuan constellation dataset or the Simulation-24 dataset. The training dataset may include sample satellites, the remaining resource values ​​of the sample satellites at different times, and the set of sample service items at different times. The satellite resources required by each sample service item in the set of sample service items are usually determined, and can be experimentally determined when planning the service items that satellites can execute. Based on this, the remaining resource values ​​of each sample satellite can be determined using the training dataset, and the resource requirements of each sample service item in the set of sample service items can be determined by searching a database. Furthermore, when establishing the training dataset, a correspondence between the resource requirements of sample service items and the sample service items can also be established, thereby determining the resource requirements of each sample service item in the set of sample service items in the training dataset.

[0095] Step 102: Input the sample service item set, the sample satellite remaining resource value, and the sample service item resource requirement value into the first multi-agent reinforcement learning network model to obtain the first joint action value output by the first multi-agent reinforcement learning network model.

[0096] In the embodiments of this application, the sample service item set, the remaining resource value of the sample satellite, and the resource requirement value of the sample service item can be input into a first multi-agent reinforcement learning network model to obtain a first joint action value output by the first multi-agent reinforcement learning network model. The first multi-agent reinforcement learning network model can be a multi-agent reinforcement learning model with a certain neural network structure, such as QMIX, MADDPG, etc. Each first joint action value corresponds to a sample satellite's remaining resource value and a sample service item's resource requirement value. Since there is a one-to-one correspondence between the sample satellite's remaining resource value and the sample satellite, and a one-to-one correspondence between the sample service item's resource requirement value and the sample service item, each first joint action value corresponds to one sample satellite and one sample service item. Furthermore, the sample satellites and sample service items corresponding to different joint action values ​​may not be completely identical.

[0097] Optionally, step 102 may include the following sub-steps:

[0098] Sub-step 1021: Input the sample service item set, the sample satellite remaining resource value, and the sample service item resource requirement value into the encoding module to obtain the first sample remaining resource observation value and the first sample resource requirement observation value output by the encoding module.

[0099] In the embodiments of this application, the first multi-agent reinforcement learning network model may include an encoding module and a joint action value calculation module. The sample service item set, the sample satellite remaining resource value, and the sample service item resource requirement value can be input into the encoding module according to a preset format, so as to obtain the first sample remaining resource observation value and the first sample resource requirement observation value output by the encoding module.

[0100] Optionally, sub-step 1021 may include the following sub-steps:

[0101] Sub-step A1: Concatenate the sample service item set with the sample satellite remaining resource value to obtain the first concatenation vector.

[0102] In the embodiments of this application, the sample service item set and the remaining resource values ​​of the sample satellites can be concatenated to obtain a first concatenated vector. During concatenation, the remaining resource value vector formed by the remaining resource values ​​of the sample satellites can be concatenated at the beginning of the sample service item set. During concatenation, the sample service item set can be used as a vector for concatenation.

[0103] For example, if the sample service item set is The remaining resource value vector formed by the remaining resource values ​​of the sample satellites is Then the first concatenated vector obtained after concatenation is , where the symbol " " is a concatenation character. The label for the sample satellite. Indicates sample satellite The remaining resource value of the sample satellite, The label for the sample service item.

[0104] Sub-step A2 involves concatenating the sample service item set with the sample service item resource requirement value to obtain a second concatenated vector.

[0105] In embodiments of this application, a second concatenated vector can be obtained by concatenating a set of sample service items with the resource requirement values ​​of the sample service items. During concatenation, a resource requirement value vector formed by the resource requirement values ​​of the sample service items can be appended to the beginning of the set of sample service items. Alternatively, the set of sample service items can be used as a vector for concatenation.

[0106] Continuing with the example above, if the sample service item set is The resource requirement value vector formed by the resource requirement values ​​of the sample service items is Then the second concatenated vector obtained after concatenation is .in, The label for the sample service item. Service items for samples The resource requirements for sample service items.

[0107] Sub-step A3: Input the first concatenation vector and the second concatenation vector into the encoding module respectively to obtain the first sample remaining resource observation value and the first sample resource demand observation value output by the encoding module respectively.

[0108] In embodiments of this application, the first concatenation vector and the second concatenation vector can be input into the encoding module of the first multi-agent reinforcement learning network model, thereby obtaining the first sample remaining resource observation value and the first sample resource demand observation value output by the encoding module. The first sample remaining resource observation value can correspond to the first concatenation vector, and the first sample resource demand observation value can correspond to the second concatenation vector.

[0109] Using the example above, the remaining resource observations for the first sample can be represented as: ,in The remaining resource observations for the first sample. k Indicates the number of training sessions. The encoding function represents the encoding module; the first sample resource requirement observation can be expressed as... ,in This represents the resource demand observation for the first sample.

[0110] Optionally, sub-step A3 may include the following sub-steps:

[0111] Sub-step a1: Based on the remaining resource value of the sample satellite and the resource requirement value of the sample service item, determine the mask matrix corresponding to each sample satellite and each sample service item; wherein, the row number of the mask matrix corresponds to each sample satellite, the column number of the mask matrix corresponds to each sample service item, and the first element value of the mask matrix indicates whether the sample service item corresponding to the first element value is allowed to run on the sample satellite corresponding to the first element value.

[0112] In the embodiments of this application, a mask matrix corresponding to each sample satellite and each sample service item can be determined based on the remaining resource value of the sample satellite and the resource requirement value of the sample service item. The row numbers of the mask matrix correspond to each sample satellite, and the column numbers correspond to each sample service item. The first element value in the mask matrix indicates whether the sample service item corresponding to the first element value is allowed to run on the sample satellite corresponding to the first element value. Specifically, each element value in the mask matrix can correspond to one sample satellite and one sample service item. When the resource requirement value of the sample service item is greater than the remaining resource value of the sample satellite, the element value can be 0, indicating that the sample service item is not allowed to run on that sample satellite; when the resource requirement value is less than or equal to the remaining resource value of the sample satellite, the element value can be 1, indicating that the sample service item is allowed to run on that sample satellite.

[0113] Sub-step a2: Input the first concatenation vector and the second concatenation vector into the encoding module respectively to obtain the second sample remaining resource observation value and the second sample resource demand observation value output by the encoding module respectively.

[0114] In the embodiments of this application, the implementation of this step can be referred to the embodiment of sub-step A3, and will not be repeated here.

[0115] Following the example under sub-step A3, in the embodiment of this step, the remaining resource observation value of the second sample can be The second sample resource demand observation can be .

[0116] Sub-step a3: Input the mask matrix into the encoding module to obtain the mask matrix encoding output by the encoding module.

[0117] In the embodiments of this application, the mask matrix can be input into the encoding module to obtain the mask matrix encoding output by the encoding module.

[0118] For example, if the mask matrix is Then the mask matrix encoding can be expressed as ,in, This refers to the encoding function in the encoding module.

[0119] Sub-step a4: Concatenate the mask matrix encoding and the second sample remaining resource observation value to obtain the first sample remaining resource observation value; concatenate the mask matrix encoding and the second sample resource demand observation value to obtain the first sample resource demand observation value.

[0120] In embodiments of this application, the mask matrix encoding can be concatenated with the second sample remaining resource observation value and the second sample resource demand observation value, respectively, to obtain the first sample remaining resource observation value and the first sample resource demand observation value. The concatenation can be in the form of addition.

[0121] Using the example above, the remaining resource observations for the first sample can be represented as: The first sample resource demand observation can be expressed as .

[0122] In the embodiments of this application, a mask matrix corresponding to each sample satellite and each sample service item is determined based on the remaining resource value of the sample satellite and the resource requirement value of the sample service item. The row numbers of the mask matrix correspond to each sample satellite, the column numbers correspond to each sample service item, and the first element value of the mask matrix indicates whether the sample service item corresponding to the first element value is allowed to run on the sample satellite corresponding to the first element value. The first concatenation vector and the second concatenation vector are respectively input into the encoding module to obtain the second sample remaining resource observation value and the second sample resource requirement observation value output by the encoding module. The mask matrix is ​​input into the encoding module to obtain the mask matrix encoding output by the encoding module. The mask matrix encoding and the second sample remaining resource observation value are concatenated to obtain the first sample remaining resource observation value. The mask matrix encoding and the second sample resource requirement observation value are also concatenated to obtain the first sample resource requirement observation value. The mask matrix can provide reference values ​​to the first multi-agent reinforcement learning network model, which to some extent improves the accuracy of the first sample remaining resource observation value and the first sample resource requirement observation value, thereby improving the single-training effect of the first multi-agent reinforcement learning network model.

[0123] In the embodiments of this application, a first concatenation vector is obtained by concatenating the sample service item set with the sample satellite remaining resource value, and a second concatenation vector is obtained by concatenating the sample service item set with the sample service item resource demand value. The first and second concatenation vectors are respectively input into the encoding module to obtain the first sample remaining resource observation value and the first sample resource demand observation value output by the encoding module. This can establish the correlation between the sample service item set, the sample satellite remaining resource value, and the sample service item resource demand value in a vector concatenation manner, which improves the training effect of the first multi-agent reinforcement learning network model to a certain extent.

[0124] Sub-step 1022: The remaining resource value of the sample satellite and the resource demand value of the sample service item are concatenated to obtain the state vector corresponding to each sample service item.

[0125] In the embodiments of this application, the remaining resource value of the sample satellite and the resource requirement value of the sample service item can be concatenated to obtain the state vector corresponding to the resource requirement value of each sample service item. Since there is a one-to-one correspondence between the resource requirement value of the sample service item and the sample service item, the state vector also has a corresponding relationship with the sample service item.

[0126] Using the example above, the state vector can be represented as ,in, The label for the sample service item. The label for the sample satellite. Indicates sample satellite The remaining resource value of the sample satellite, Indicates sample service items The resource requirements for sample service items.

[0127] Sub-step 1023: Input the state vector, the observation value of the remaining resources of the first sample, and the observation value of the resource demand of the first sample into the joint action value calculation module to obtain the first joint action value output by the joint action value calculation module.

[0128] In the embodiments of this application, the state vector, the observation value of the remaining resources of the first sample, and the observation value of the resource demand of the first sample can be input into the joint action value calculation module of the first multi-agent reinforcement learning network model, thereby obtaining the first joint action value output by the joint action value calculation module. The calculation method of the first joint action value is shown in Equation 1 below:

[0129] (Equation 1)

[0130] In Equation 1, , The Q-value function represents the resource demand observations of the first sample. The Q-value function represents the remaining resource observations in the first sample. This represents the joint action of the two agents during the k-th training iteration. and It is an activation function. It is a multilayer perceptron, and the activation function of the multilayer perceptron is... .

[0131] In the embodiments of this application, by inputting the sample service item set, the sample satellite remaining resource value, and the sample service item resource requirement value into the encoding module, the first sample remaining resource observation value and the first sample resource requirement observation value output by the encoding module are obtained. The sample satellite remaining resource value and the sample service item resource requirement value are concatenated to obtain the state vector corresponding to each sample service item. The state vector, the first sample remaining resource observation value, and the first sample resource requirement observation value are input into the joint action value calculation module to obtain the first joint action value output by the joint action value calculation module. The sample service item set, the sample satellite remaining resource value, and the sample service item resource requirement value can be processed by the encoding module and the joint action value calculation module of the first multi-agent reinforcement learning network model, thereby improving the model's processing efficiency for the above data.

[0132] Step 103: Calculate the loss value of the first multi-agent reinforcement learning network model based on the resource requirement value of each sample service item and the first joint action value.

[0133] In the embodiments of this application, the loss value of the first multi-agent reinforcement learning network model can be calculated based on the resource requirement value of each sample service item and the value of each first joint action. The method for calculating the loss value is shown in Equation 2 below:

[0134] (Equation 2)

[0135] In Equation 2, Indicates the loss value. Indicates sample service items k The resource requirement value for sample service items (used as a reward value). The discount factor is a hyperparameter in the first multi-agent reinforcement learning network model. "" represents the set of parameters that need to be learned in the multilayer perceptron of the first multi-agent reinforcement learning network model.

[0136] Step 104: Based on the loss value, adjust the first parameter of the first multi-agent reinforcement learning network model to obtain the target multi-agent reinforcement learning network model; the target multi-agent reinforcement learning network model is used to determine the target joint action value between the target satellite and the target service item.

[0137] In the embodiments of this application, the first parameters of the first multi-agent reinforcement learning network model can be adjusted based on the loss value of the first multi-agent reinforcement learning network model. The adjustment direction is to gradually reduce the loss value until it no longer changes, thereby obtaining the target multi-agent reinforcement learning network model. The first parameters include, but are not limited to, the encoding function, discount factor, and hyperparameters in the multilayer perceptron of the first multi-agent reinforcement learning network model. If the training effect is not ideal, the activation function in the multilayer perceptron can also be adjusted.

[0138] Optionally, step 104 may include the following sub-steps:

[0139] Sub-step 1041: Based on the loss value, adjust the first parameter of the first multi-agent reinforcement learning network model to obtain the first multi-agent reinforcement learning network model with updated parameters.

[0140] In the embodiments of this application, the first parameter of the first multi-agent reinforcement learning network model is adjusted according to the loss value. Under normal circumstances, the adjustment will not be successful on the first attempt. Therefore, after the first parameter is adjusted once, the first multi-agent reinforcement learning network model with updated parameters can be obtained.

[0141] Sub-step 1042 involves re-executing the steps of determining the remaining resource values ​​of each sample satellite and the resource requirements of each sample service item in the sample service item set.

[0142] In the embodiments of this application, after the first parameter in the first multi-agent reinforcement learning network model is adjusted once, the steps of determining the remaining resource value of each sample satellite corresponding to each sample satellite and the resource requirement value of each sample service item corresponding to each sample service item in the sample service item set can be re-executed to conduct the next training of the first multi-agent reinforcement learning network model.

[0143] Sub-step 1043: Compare the loss value with the first threshold, and accumulate the first number of times the loss value is less than the first threshold.

[0144] In the embodiments of this application, a first threshold corresponding to the loss value can be set. After each training to obtain the loss value, the size of the loss value and the first threshold can be compared and the comparison result can be obtained. If the loss value of this training is less than the first threshold, the first number of times the loss value is less than the first threshold can be accumulated.

[0145] Sub-step 1044: If the first number of times is greater than or equal to the second threshold, stop training the first multi-agent reinforcement learning network model and obtain the target multi-agent reinforcement learning network model.

[0146] In the embodiments of this application, if the first number of times is greater than or equal to the second threshold, the training of the first multi-agent reinforcement learning network model can be stopped, so that the current first multi-agent reinforcement learning network model can be used as the target multi-agent reinforcement learning network model.

[0147] In one possible embodiment, a second number of consecutive occurrences of the loss value being less than the first threshold can also be accumulated. That is, when the loss value is less than the first threshold, the second count begins to accumulate. During the accumulation of the second count, if the loss value is greater than or equal to the first threshold in a certain training iteration, the second count is set to 0. If the second count is greater than or equal to a third threshold, training of the first multi-agent reinforcement learning network model can be stopped, and the current first multi-agent reinforcement learning network model can be used as the target multi-agent reinforcement learning network model.

[0148] In the embodiments of this application, by adjusting the first parameters of the first multi-agent reinforcement learning network model based on the loss value, a parameter-updated first multi-agent reinforcement learning network model is obtained. The steps of determining the remaining resource values ​​of each sample satellite and the resource requirements of each sample service item in the sample service item set are re-executed. The loss value and the first threshold are compared, and the first number of times the loss value is less than the first threshold is accumulated. If the first number of times is greater than or equal to the second threshold, the training of the first multi-agent reinforcement learning network model is stopped, and the target multi-agent reinforcement learning network model is obtained. The first multi-agent reinforcement learning network model can be trained repeatedly, and training can be stopped under certain conditions to obtain the target multi-agent reinforcement learning network model, thereby improving the usability and output accuracy of the target multi-agent reinforcement learning network model.

[0149] In the embodiments of this application, by determining the remaining resource value of each sample satellite and the resource requirement value of each sample service item in the sample service item set, the sample service item set, the remaining resource value of the sample satellite, and the resource requirement value of the sample service item are input into a first multi-agent reinforcement learning network model to obtain a first joint action value output by the first multi-agent reinforcement learning network model. Based on the resource requirement value of each sample service item and each first joint action value, the loss value of the first multi-agent reinforcement learning network model is calculated. Based on the loss value, the first parameter of the first multi-agent reinforcement learning network model is adjusted to obtain a target multi-agent reinforcement learning network model. The target multi-agent reinforcement learning network model is used to determine the target joint action value between the target satellite and the target service item. A target multi-agent reinforcement learning network model with more accurate output joint action values ​​can be obtained. When the target satellite and target service item are matched and combined using the target joint action value output by the target multi-agent reinforcement learning network model to obtain the target satellite service combination, the target service item can be matched with an available target satellite, and the satellite resources occupied by the target service item can be reduced, thereby improving the utilization rate of satellite resources.

[0150] Reference Figure 2 , Figure 2 This is a flowchart illustrating another satellite service combination selection method and system based on reinforcement learning multi-agent cooperation provided in this application embodiment. The method may include:

[0151] Step 201: Determine the remaining resource value of the target satellite and the resource requirement value of each target service item included in the target service item set.

[0152] In the embodiments of this application, the occupancy of service operation resources in the target satellite can be statistically analyzed, thereby determining the remaining resource value of the target satellite. Additionally, the resource requirements of each target service item included in the target service item set can be determined experimentally.

[0153] Step 202: Input the target service item set, the remaining resource value, and the resource demand value into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model; wherein, the target multi-agent reinforcement learning network model is obtained based on any of the model training methods described above.

[0154] In the embodiments of this application, the target service item set, remaining resource value, and resource requirement value can be input into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model. The target multi-agent reinforcement learning network model can be obtained based on any of the model training methods described above. The processing of the target service item set, remaining resource value, and resource requirement value by the target multi-agent reinforcement learning network model is similar to the processing of the sample service item set, sample satellite remaining resource value, and sample service item resource requirement value during the training phase of the model. Refer to step 102 and its sub-steps for implementation details, which will not be repeated here.

[0155] Optionally, step 202 may include the following sub-steps:

[0156] Sub-step 2021: Concatenate the remaining resource value with the target service item set to obtain the first concatenation vector.

[0157] In the embodiments of this application, the remaining resource value and the target service item set can be concatenated to obtain a first concatenated vector. The method for concatenating the remaining resource value and the target service item set is similar to the method for concatenating the sample service item set and the sample satellite remaining resource value; please refer to the embodiment of step A1, which will not be repeated here.

[0158] Sub-step 2022: Concatenate the resource requirement value with the target service item set to obtain the second concatenation vector.

[0159] In the embodiments of this application, the resource requirement value and the target service item set can be concatenated to obtain a second concatenated vector. The method for concatenating the resource requirement value and the target service item set is similar to the method for concatenating the sample service item set and the sample service item resource requirement value, and can be referred to the embodiment of step A2, which will not be repeated here.

[0160] Sub-step 2023: Input the first concatenation vector and the second concatenation vector into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model.

[0161] In the embodiments of this application, the first concatenated vector and the second concatenated vector can be input into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model. The processing of the first concatenated vector and the second concatenated vector by the target multi-agent reinforcement learning network model is similar to the processing of the first concatenated vector and the second concatenated vector by the first multi-agent reinforcement learning network model during the training phase, and can be referred to the embodiment of step A3, which will not be repeated here.

[0162] It should be noted that if a mask matrix is ​​used during the training phase of the first multi-agent reinforcement learning network model to improve its training performance, then during the usage phase of the target multi-agent reinforcement learning network model, the mask matrix composed of the target satellite and the target service item can be input together with the first and second concatenation vectors into the target multi-agent reinforcement learning network model to improve the accuracy of the target joint action value.

[0163] In the embodiments of this application, a first concatenated vector is obtained by concatenating the remaining resource value with the target service item set, and a second concatenated vector is obtained by concatenating the resource demand value with the target service item set. The first and second concatenated vectors are then input into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model. This method can transform the input data into an input vector that is more compatible with the target multi-agent reinforcement learning network model by concatenating vectors, thereby improving the output accuracy of the target multi-agent reinforcement learning network model.

[0164] Step 203: Based on the resource requirement value and the target joint action value, combine the target satellite and the target service item to obtain the target satellite service combination.

[0165] In the embodiments of this application, each target joint action value can correspond to a target service item and a target satellite. Therefore, the target satellite and target service item can be combined according to the resource requirement value and the target joint action value to obtain a target satellite service combination. Specifically, firstly, the target joint action value corresponding to each resource requirement value is determined, and then the target satellite service combination is determined according to the target joint action value corresponding to each resource requirement value.

[0166] Table 1 below lists a comparison of experimental data using the multi-agent reinforcement learning network model and hierarchical deep reinforcement learning provided in the embodiments of this application.

[0167] Table 1

[0168]

[0169] In Table 1, CMAR refers to Cooperative Multi-Agent Reinforcement Learning, and HDRL refers to Hierarchical Deep Reinforcement Learning. Evaluation metrics include Gap and Testing Error, where Gap refers to the omissions in the satellite service suite, and Testing Error refers to the error rate during model experiments. It can be seen that the CMAR multi-agent reinforcement learning model provided in this application's embodiments shows a certain degree of reduction in both omissions and error rate compared to HDRL. Furthermore, Figure 3 This is a histogram comparing the experimental performance of a target multi-agent reinforcement learning network model provided in this embodiment of the application. In the figure, the horizontal axis represents the three datasets used in the experiment, and the vertical axis represents the process time. It can be seen that the multi-agent reinforcement learning model CMAR provided in this embodiment of the application also has a certain degree of reduction in experimental time compared with hierarchical deep reinforcement learning HDRL.

[0170] Optionally, step 203 may include the following sub-steps:

[0171] Sub-step 2031: Determine the first selection probability of the target service item corresponding to the resource demand value.

[0172] In the embodiments of this application, a first selection probability of the target service item corresponding to the resource requirement value can be determined. The formula for calculating the first selection probability is shown in Equation 3 below:

[0173] (Equation 3)

[0174] In Equation 3, It means and The Q-value function, which can be output during the calculation of the target joint action value, is used as a natural constant on the right-hand side of Equation 3. The index. Indicates the target service item. This represents the observed resource demand value corresponding to the current resource demand value. It can be understood that the larger the resource demand value, the greater the probability of it being the first choice.

[0175] Sub-step 2032: Determine the second selection probability of the target satellite corresponding to the target joint action value.

[0176] In the embodiments of this application, a second selection probability of the target satellite corresponding to the determined joint action value can be calculated. The formula for calculating the second selection probability is shown in Equation 4 below:

[0177] (Equation 4)

[0178] In Equation 4, It means and The Q-value function between these parameters can be output during the calculation of the target joint action value, and is treated as a natural constant on the right-hand side of Equation 4. The index. Indicates the target satellite, This represents the remaining resource observation value of the target satellite associated with the target joint action value. It can be understood that the larger the remaining resource value, the greater the corresponding second selection probability.

[0179] Sub-step 2033: Arrange each of the first selection probabilities and each of the second selection probabilities in descending order to obtain the first selection probability sequence and the second selection probability sequence.

[0180] In the embodiments of this application, the first selection probability and the second selection probability can be arranged in descending order to obtain the first selection probability sequence and the second selection probability sequence.

[0181] Sub-step 2034: Combine the target satellites and target service items corresponding to the sequence values ​​at the same position in the first selection probability sequence and the second selection probability sequence to obtain the target satellite service combination.

[0182] In embodiments of this application, target satellites and target service items corresponding to sequence values ​​at the same position in the first and second selection probability sequences can be combined to obtain target satellite service combinations. For example, the sequence value at the first position in the first selection probability sequence can correspond to a resource demand value, and this resource demand value is the largest resource demand value. The sequence value at the first position in the second selection probability sequence can correspond to a remaining resource value, and this remaining resource value is the largest remaining resource value. Thus, the target service item corresponding to the resource demand value and the target satellite corresponding to the remaining resource value can be combined to obtain a target satellite service combination. By traversing the first and second selection probability sequences according to the above method, target satellite service combinations corresponding to all target service items can be obtained.

[0183] In the embodiments of this application, by determining the first selection probability of the target service item corresponding to the resource demand value and the second selection probability of the target satellite corresponding to the target joint action value, the first selection probabilities and the second selection probabilities are arranged in descending order to obtain a first selection probability sequence and a second selection probability sequence. The target satellites and target service items corresponding to the sequence values ​​at the same position in the first selection probability sequence and the second selection probability sequence are combined to obtain a target satellite service combination. The target satellites and target service items can be selected and combined according to the selection probabilities corresponding to the target service items and target satellites respectively to obtain a target satellite service combination. This makes the compatibility between the target service items and target satellites corresponding to the obtained target satellite service combination better, and further improves the availability and accuracy of the target satellite service combination.

[0184] In the embodiments of this application, by determining the remaining resource value of the target satellite and the resource requirement value of each target service item included in the target service item set, the target service item set, the remaining resource value, and the resource requirement value are input into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model. The target multi-agent reinforcement learning network model is obtained based on any of the training methods described above. Based on the resource requirement value and the target joint action value, the target satellite and the target service item are combined to obtain a target satellite service combination. The target multi-agent reinforcement learning network model can obtain the target joint action value between the target service item and the target satellite, and the target satellite and target service item are combined using the target joint action value to obtain the target satellite service combination, thereby improving the availability and accuracy of the target satellite service combination to a certain extent.

[0185] Reference Figure 4 , Figure 4 This is a schematic diagram of a satellite service composition method provided in an embodiment of this application. In the diagram, satellites represent the remaining resource values ​​corresponding to satellites, services represent the resource demand values ​​corresponding to service items, and request represents the set of service items. In one step of the model's processing, satellites, services, and request are first input into the encoder, and a mask matrix composed of satellites and service items is also input. Also input into the encoder, and calculate the Q-value function separately. and After that, according to and Select satellites and services to combine, as shown in the image. In the sequence, 0.3 is the largest value, so the service item corresponding to 0.3 is selected first. , The largest value in the sequence is 0.5, so the satellite corresponding to 0.5 is selected first. and will and By combining these components, the first satellite service combination is obtained, and the output is also provided. and The corresponding Q-value functions and , and according to and Output the first target joint action value After that, you can add satellites, services, requests, and the mask matrix. The first satellite service combination identified is input into the model for further processing.

[0186] Reference Figure 5 , Figure 5 This is a logical block diagram of a model training system provided in an embodiment of this application. The system 500 may include:

[0187] The determination module 501 is used to determine the remaining resource value of each sample satellite and the resource requirement value of each sample service item in the sample service item set.

[0188] Input module 502 is used to input the sample service item set, the sample satellite remaining resource value and the sample service item resource demand value into the first multi-agent reinforcement learning network model to obtain the first joint action value output by the first multi-agent reinforcement learning network model;

[0189] The calculation module 503 is used to calculate the loss value of the first multi-agent reinforcement learning network model based on the resource requirement value of each of the sample service items and the value of each of the first joint actions.

[0190] The adjustment module 504 is used to adjust the first parameter of the first multi-agent reinforcement learning network model based on the loss value to obtain the target multi-agent reinforcement learning network model; the target multi-agent reinforcement learning network model is used to determine the target joint action value between the target satellite and the target service item.

[0191] Optionally, the first multi-agent reinforcement learning network model includes an encoding module and a joint action value calculation module, and the input module 502 includes:

[0192] The first input submodule is used to input the sample service item set, the sample satellite remaining resource value and the sample service item resource requirement value into the encoding module to obtain the first sample remaining resource observation value and the first sample resource requirement observation value output by the encoding module.

[0193] The splicing submodule is used to splice the remaining resource value of the sample satellite and the resource requirement value of the sample service item to obtain the state vector corresponding to each sample service item;

[0194] The second input submodule is used to input the state vector, the observation value of the remaining resources of the first sample, and the observation value of the resource demand of the first sample into the joint action value calculation module to obtain the first joint action value output by the joint action value calculation module.

[0195] Optionally, the first input submodule includes:

[0196] The first splicing unit is used to splice the sample service item set with the sample satellite remaining resource value to obtain the first splicing vector;

[0197] The second splicing unit is used to splice the sample service item set with the sample service item resource requirement value to obtain a second splicing vector;

[0198] The input unit is used to input the first concatenation vector and the second concatenation vector into the encoding module respectively, so as to obtain the first sample remaining resource observation value and the first sample resource demand observation value output by the encoding module respectively.

[0199] Optionally, the input unit includes:

[0200] A determining subunit is used to determine a mask matrix corresponding to each sample satellite and each sample service item based on the remaining resource value of the sample satellite and the resource requirement value of the sample service item; wherein, the row number of the mask matrix corresponds to each sample satellite, the column number of the mask matrix corresponds to each sample service item, and the first element value of the mask matrix indicates whether the sample service item corresponding to the first element value is allowed to run on the sample satellite corresponding to the first element value.

[0201] The first input subunit is used to input the first concatenation vector and the second concatenation vector into the encoding module respectively, so as to obtain the second sample remaining resource observation value and the second sample resource demand observation value output by the encoding module respectively;

[0202] The second input subunit is used to input the mask matrix into the encoding module to obtain the mask matrix encoding output by the encoding module;

[0203] The splicing subunit is used to splice the mask matrix encoding and the second sample remaining resource observation value to obtain the first sample remaining resource observation value, and to splice the mask matrix encoding and the second sample resource demand observation value to obtain the first sample resource demand observation value.

[0204] Optionally, the adjustment module 504 includes:

[0205] The adjustment submodule is used to adjust the first parameter of the first multi-agent reinforcement learning network model based on the loss value, so as to obtain the first multi-agent reinforcement learning network model with updated parameters.

[0206] The execution submodule is used to re-execute the steps of determining the remaining resource value of each sample satellite and the resource requirement value of each sample service item in the sample service item set.

[0207] The comparison and accumulation submodule is used to compare the loss value with the first threshold and accumulate the first number of times the loss value is less than the first threshold;

[0208] The stopping submodule is used to stop training the first multi-agent reinforcement learning network model when the first number of times is greater than or equal to the second threshold, so as to obtain the target multi-agent reinforcement learning network model.

[0209] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0210] The model training system described above has the same advantages over the prior art as the model training method described in the foregoing embodiments, and will not be repeated here.

[0211] Reference Figure 6 , Figure 6 This is a logical block diagram of a satellite service combination selection system based on reinforcement learning multi-agent cooperation provided in an embodiment of this application. The system 600 may include:

[0212] The determination module 601 is used to determine the remaining resource value of the target satellite and the resource requirement value of each target service item included in the target service item set;

[0213] Input module 602 is used to input the target service item set, the remaining resource value, and the resource demand value into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model; wherein, the target multi-agent reinforcement learning network model is obtained based on any of the model training methods described above;

[0214] The combination module 603 is used to combine the target satellite and the target service item based on the resource demand value and the target joint action value to obtain the target satellite service combination.

[0215] Optionally, the input module 602 includes:

[0216] The first splicing submodule is used to splice the remaining resource value with the target service item set to obtain a first splicing vector;

[0217] The second splicing submodule is used to splice the resource requirement value with the target service item set to obtain a second splicing vector;

[0218] The input submodule is used to input the first concatenated vector and the second concatenated vector into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model.

[0219] Optionally, the combination module 603 includes:

[0220] The first determining submodule is used to determine the first selection probability of the target service item corresponding to the resource requirement value;

[0221] The second determining submodule is used to determine the second selection probability of the target satellite corresponding to the target joint action value;

[0222] The permutation submodule is used to arrange each of the first selection probabilities and each of the second selection probabilities in descending order to obtain a first selection probability sequence and a second selection probability sequence;

[0223] The combination submodule is used to combine the target satellites and target service items corresponding to the sequence values ​​at the same position in the first selection probability sequence and the second selection probability sequence to obtain the target satellite service combination.

[0224] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0225] The satellite service composition selection system based on reinforcement learning multi-agent cooperation described above has the same advantages over the prior art as the satellite service composition selection method based on reinforcement learning multi-agent cooperation described in the foregoing embodiments, and will not be repeated here.

[0226] This application provides an electronic device, see [link to relevant documentation] Figure 7 The electronic device 70 includes: a processor 701, a memory 702, and a computer program 7021 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 7021, it implements the model training method of the foregoing embodiments, or the satellite service combination selection method based on reinforcement learning multi-agent collaboration.

[0227] This application provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the model training method described above, or the satellite service combination selection method based on reinforcement learning multi-agent collaboration.

[0228] This application also provides a computer program product that, when run on an electronic device, enables the processor to implement the model training method disclosed in this application, or the steps in the satellite service combination selection method based on reinforcement learning multi-agent collaboration.

[0229] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0230] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0231] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0232] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0233] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0234] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0235] The above provides a detailed description of the satellite service combination selection method and system based on reinforcement learning multi-agent collaboration provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model training method, characterized in that, The model is a first multi-agent reinforcement learning network model, including an encoding module and a joint action value calculation module, and the method includes: Determine the remaining resource value of each sample satellite and the resource requirement value of each sample service item in the sample service item set. The sample service item set is concatenated with the sample satellite remaining resource value to obtain the first concatenation vector; The sample service item set and the sample service item resource requirement value are concatenated to obtain the second concatenation vector; Based on the remaining resource values ​​of the sample satellites and the resource requirements of the sample service items, a mask matrix is ​​determined for each sample satellite and each sample service item; wherein, the row number of the mask matrix corresponds to each sample satellite, the column number of the mask matrix corresponds to each sample service item, and the element value of the mask matrix indicates whether the sample service item corresponding to the element value is allowed to run on the sample satellite corresponding to the element value. The first concatenation vector, the second concatenation vector, and the mask matrix are input into the encoding module of the first multi-agent reinforcement learning network model to obtain the first sample remaining resource observation value and the first sample resource demand observation value output by the encoding module respectively. The remaining resource value of the sample satellite and the resource requirement value of the sample service item are concatenated to obtain the state vector corresponding to each sample service item; The state vector, the observation value of the remaining resources of the first sample, and the observation value of the resource demand of the first sample are input into the joint action value calculation module to obtain the first joint action value output by the joint action value calculation module. Based on the resource requirement values ​​of each of the sample service items and the values ​​of each of the first joint actions, the loss value of the first multi-agent reinforcement learning network model is calculated. Based on the loss value, the first parameter of the first multi-agent reinforcement learning network model is adjusted to obtain the target multi-agent reinforcement learning network model; the target multi-agent reinforcement learning network model is used to determine the target joint action value between the target satellite and the target service item.

2. The method of claim 1, wherein, The step of inputting the first concatenation vector, the second concatenation vector, and the mask matrix into the encoding module of the first multi-agent reinforcement learning network model to obtain the first sample remaining resource observation value and the first sample resource demand observation value output by the encoding module respectively includes: The first concatenated vector and the second concatenated vector are respectively input into the encoding module to obtain the second sample remaining resource observation value and the second sample resource demand observation value output by the encoding module respectively; The mask matrix is ​​input into the encoding module to obtain the mask matrix encoding output by the encoding module; The mask matrix encoding and the second sample remaining resource observation value are concatenated to obtain the first sample remaining resource observation value. The mask matrix encoding and the second sample resource demand observation value are concatenated to obtain the first sample resource demand observation value.

3. The method of claim 1, wherein, The step of adjusting the first parameters of the first multi-agent reinforcement learning network model based on the loss value to obtain the target multi-agent reinforcement learning network model includes: Based on the loss value, the first parameter of the first multi-agent reinforcement learning network model is adjusted to obtain the first multi-agent reinforcement learning network model with updated parameters. Repeat the steps of determining the remaining resource value of each sample satellite and the resource requirement value of each sample service item in the sample service item set. Compare the loss value with a first threshold, and accumulate the first number of times the loss value is less than the first threshold; If the first number of attempts is greater than or equal to the second threshold, training of the first multi-agent reinforcement learning network model is stopped, and the target multi-agent reinforcement learning network model is obtained.

4. A satellite service combination selection method based on reinforcement learning multi-agent collaboration, characterized in that, The method includes: Determine the remaining resource value of the target satellite and the resource requirement value of each target service item included in the target service item set; The target service item set, the remaining resource value, and the resource demand value are input into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model; wherein, the target multi-agent reinforcement learning network model is obtained based on the training method described in any one of claims 1 to 3; Based on the resource requirement value and the target joint action value, the target satellite and the target service item are combined to obtain the target satellite service combination.

5. The method of claim 4, wherein, The step of inputting the target service item set, the remaining resource value, and the resource demand value into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model includes: The remaining resource value is concatenated with the target service item set to obtain the first concatenation vector; The resource requirement value is concatenated with the target service item set to obtain a second concatenation vector; The first concatenated vector and the second concatenated vector are input into the target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model.

6. The method of claim 5, wherein, The process of combining the target satellite and the target service item based on the resource demand value and the target joint action value to obtain a target satellite service combination includes: Determine the first selection probability of the target service item corresponding to the resource requirement value; Determine the second selection probability of the target satellite corresponding to the target joint action value; Arrange the first selection probability and the second selection probability in descending order to obtain the first selection probability sequence and the second selection probability sequence; The target satellite and target service item corresponding to the sequence values ​​at the same position in the first selection probability sequence and the second selection probability sequence are combined to obtain the target satellite service combination.

7. A model training system, comprising: The model is a first multi-agent reinforcement learning network model, including an encoding module and a joint action value calculation module. The system includes: The determination module is used to determine the remaining resource value of each sample satellite and the resource requirement value of each sample service item in the sample service item set. The input module concatenates the set of sample service items with the remaining resource value of the sample satellite to obtain a first concatenation vector; it then concatenates the set of sample service items with the resource requirement value of the sample service item to obtain a second concatenation vector; based on the remaining resource value of the sample satellite and the resource requirement value of the sample service item, it determines a mask matrix corresponding to each sample satellite and each sample service item; wherein, the row number of the mask matrix corresponds to each sample satellite, the column number of the mask matrix corresponds to each sample service item, and the element value of the mask matrix indicates whether the sample service item corresponding to the element value is allowed on the sample satellite corresponding to the element value. Run the process; input the first concatenation vector, the second concatenation vector, and the mask matrix into the encoding module of the first multi-agent reinforcement learning network model to obtain the first sample remaining resource observation value and the first sample resource demand observation value output by the encoding module respectively; concatenate the sample satellite remaining resource value and the sample service item resource demand value to obtain the state vector corresponding to each sample service item; input the state vector, the first sample remaining resource observation value, and the first sample resource demand observation value into the joint action value calculation module of the first multi-agent reinforcement learning network model to obtain the first joint action value output by the joint action value calculation module; The calculation module is used to calculate the loss value of the first multi-agent reinforcement learning network model based on the resource requirement value of each of the sample service items and the value of each of the first joint actions. An adjustment module is used to adjust the first parameter of the first multi-agent reinforcement learning network model based on the loss value to obtain a target multi-agent reinforcement learning network model; the target multi-agent reinforcement learning network model is used to determine the target joint action value between the target satellite and the target service item. 8.A satellite service combination selection system based on reinforcement learning multi-agent collaboration, characterized in that, The system includes: The determination module is used to determine the remaining resource value of the target satellite and the resource requirement value of each target service item included in the target service item set; An input module is used to input the target service item set, the remaining resource value, and the resource demand value into a target multi-agent reinforcement learning network model to obtain the target joint action value output by the target multi-agent reinforcement learning network model; wherein, the target multi-agent reinforcement learning network model is obtained based on the training method described in any one of claims 1 to 3; The combination module is used to combine the target satellite and the target service item based on the resource requirement value and the target joint action value to obtain the target satellite service combination.

Images