Multi-task scheduling method and device, electronic equipment and readable storage medium

Abstract

The invention provides a multi-task scheduling method and device, electronic equipment and a readable storage medium. The method comprises the following steps: inputting the current state information of each vehicle, the information of each destination and the information of an article into an optimized multi-agent reinforcement learning algorithm, and obtaining each distribution task output by the optimized multi-agent reinforcement learning algorithm and a target vehicle of the distribution task, the feedback value obtained by executing the delivery task by the target vehicle is greater than the feedback values obtained by executing the delivery task by other vehicles; the optimized multi-agent reinforcement learning algorithm is obtained by optimizing historical feedback values of the target vehicles, and the feedback values comprise communication connectivity and reward values. By calculating the communication connectivity of each vehicle and the reward value for executing the distribution task, the optimal target vehicle for executing the distribution task is determined, so that task scheduling and distribution are better performed, each vehicle undertakes the distribution task matched with the vehicle, and the completion of each distribution task is ensured.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a multi-task scheduling method, device, electronic device and readable storage medium. Background Art

[0002] At present, the task of delivering goods is often carried out through the Internet of Vehicles in cities. However, the increase in the number of vehicles and the increase in tasks makes it difficult to allocate tasks well. Summary of the invention

[0003] The embodiments of the present application provide a multi-task scheduling method, device, electronic device and readable storage medium, which can solve the problem that the increase in the number of vehicles and the increase in tasks make it impossible to allocate tasks well.

[0004] In a first aspect, an embodiment of the present application provides a multi-task scheduling method, which is applied to a server, comprising:

[0005] Obtain the current status information of each vehicle, the information of each destination and the information of items;

[0006] Inputting the current state information of each of the vehicles, the information of each of the destinations, and the information of the items into the optimized multi-agent reinforcement learning algorithm, obtaining each delivery task output by the optimized multi-agent reinforcement learning algorithm and the target vehicle of the delivery task, wherein the feedback value obtained by the target vehicle when executing the delivery task is greater than the feedback value obtained by other vehicles when executing the delivery task;

[0007] Sending a corresponding delivery task to the target vehicle;

[0008] Among them, the optimized multi-agent reinforcement learning algorithm is obtained after optimization using the historical feedback values ​​of each target vehicle, and the optimized multi-agent reinforcement learning algorithm is used to determine each delivery task based on the information of each destination and the information of the item; for each delivery task, the communication connectivity of each vehicle and the reward value for executing the delivery task are determined based on the current status information of each vehicle, and the target vehicle is determined based on the communication connectivity and reward value of each vehicle; and each delivery task and the target vehicle of the delivery task are output, and the feedback value includes the communication connectivity and the reward value.

[0009] In one embodiment, the communication connectivity of the vehicle is determined by the optimized multi-agent reinforcement learning algorithm according to the bandwidth, driving speed, first data transmission rate of the vehicle and the driving speed and first data transmission rate of the other vehicles when the vehicle communicates with other vehicles;

[0010] When the delivery task is an independent task, the reward value of the vehicle for executing the delivery task is determined by the optimized multi-agent reinforcement learning algorithm according to the number of vehicles communicating with the server, the data processing capacity of the vehicle, and the evaluation value of the vehicle for executing the delivery task;

[0011] When the delivery task is a collaborative task, the reward value of the vehicle for executing the delivery task is determined by the optimized multi-agent reinforcement learning algorithm according to the data processing capability of the vehicle and the evaluation value of the vehicle for executing the delivery task;

[0012] The data processing capacity of the vehicle is determined by the optimized multi-agent reinforcement learning algorithm according to the amount of data processed per unit time and the CPU frequency and driving distance of the vehicle within a preset time period;

[0013] The current status information includes the bandwidth, the driving speed, the first data transmission rate, the CPU frequency, the amount of processed data per unit time, and the driving distance.

[0014] In one embodiment, the optimized multi-agent reinforcement learning algorithm is used to determine the importance of the delivery task according to the overtaking probability and overtaking risk level of the vehicle; determine the urgency of the delivery task according to the task completion time and environmental impact of the vehicle; determine the dependency of the delivery task according to the basic value and the number of levels of the delivery task; and determine the evaluation value of the delivery task according to the importance, urgency and dependency of the delivery task;

[0015] The basic value is the number of preceding delivery tasks of the delivery task, and the number of levels is determined according to the distance between the destination of the delivery task and the location of the item on a delivery route.

[0016] In one embodiment, the optimized multi-agent reinforcement learning algorithm is also used to determine the energy consumption, data processing capability, and second data transmission rate of each vehicle for each delivery task according to the current state information of each vehicle; and determine the target vehicle according to the communication connectivity, feedback value, energy consumption, data processing capability, and second data transmission rate of each vehicle, wherein the energy consumption of the target vehicle is less than that of other vehicles, and the data processing capability and second data transmission rate of the target vehicle are both greater than the data processing capability and second data transmission rate of other vehicles;

[0017] Correspondingly, the optimized multi-agent reinforcement learning algorithm is obtained after optimization using the historical feedback values, energy consumption, data processing capabilities, and second data transmission rate of each of the target vehicles.

[0018] In one embodiment, the energy consumption is determined by the optimized multi-agent reinforcement learning algorithm according to the bandwidth of the vehicle, the CPU frequency within a preset time period, and the driving distance; the second data transmission rate is determined by the optimized multi-agent reinforcement learning algorithm according to the bandwidth of the vehicle, the antenna height, the Gaussian white noise power, the communication distance between the vehicle and the server, the communication path loss index, and the communication transmission power;

[0019] The current status information includes the antenna height, the Gaussian white noise power, the communication distance, the communication path loss index, and the communication transmission power.

[0020] In one embodiment, the method further comprises:

[0021] Sending a feedback value of the corresponding delivery task to the target vehicle.

[0022] In one embodiment, the optimized multi-agent reinforcement learning algorithm is also used to determine the reliability of the vehicle in performing the delivery task based on the current state information of the vehicle. If the reliability is greater than a preset level, the reward value for the vehicle to perform the delivery task is determined based on the current state information of the vehicle.

[0023] In a second aspect, an embodiment of the present application provides a multi-task scheduling device, including:

[0024] A communication module is used to obtain the current status information of each vehicle, the information of each destination and the information of the item;

[0025] It is also used to send corresponding delivery tasks to target vehicles;

[0026] A task scheduling module is used to input the current state information of each vehicle, the information of each destination and the information of the item into the optimized multi-agent reinforcement learning algorithm, obtain each delivery task output by the optimized multi-agent reinforcement learning algorithm and the target vehicle of the delivery task, and the feedback value obtained by the target vehicle when executing the delivery task is greater than the feedback value obtained by other vehicles when executing the delivery task;

[0027] Among them, the optimized multi-agent reinforcement learning algorithm is obtained after optimization using the historical feedback values ​​of each target vehicle, and the optimized multi-agent reinforcement learning algorithm is used to determine each delivery task based on the information of each destination and the information of the item; for each delivery task, the communication connectivity of each vehicle and the reward value for executing the delivery task are determined based on the current status information of each vehicle, and the target vehicle is determined based on the communication connectivity and reward value of each vehicle; and each delivery task and the target vehicle of the delivery task are output, and the feedback value includes the communication connectivity and the reward value.

[0028] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method as described in any one of the above-mentioned first aspects when executing the computer program.

[0029] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described in any one of the above-mentioned first aspects is implemented.

[0030] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes any one of the methods described in the first aspect.

[0031] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0032] The embodiment of the present application includes inputting the current state information of each vehicle, the information of each destination and the information of the item into the optimized multi-agent reinforcement learning algorithm, obtaining each delivery task and the target vehicle of the delivery task output by the optimized multi-agent reinforcement learning algorithm, and the feedback value obtained by the target vehicle performing the delivery task is greater than the feedback value obtained by other vehicles performing the delivery task; sending the corresponding delivery task to the target vehicle; wherein the optimized multi-agent reinforcement learning algorithm is obtained after optimization using the historical feedback value of each target vehicle, and the optimized multi-agent reinforcement learning algorithm is used to determine each delivery task according to the information of each destination and the information of the item; for each delivery task, determining the communication connectivity of each vehicle and the reward value for performing the delivery task according to the current state information of each vehicle, and determining the target vehicle according to the communication connectivity and the reward value of each vehicle; and outputting each delivery task and the target vehicle of the delivery task, the feedback value includes the communication connectivity and the reward value, and by calculating the communication connectivity of each vehicle and the reward value for performing the delivery task, the optimal target vehicle for performing the delivery task is determined, so that the task scheduling and allocation can be better performed, so that each vehicle can undertake the delivery task that is suitable for it, and the completion of each delivery task is guaranteed.

[0033] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 This is a first flow chart of a multi-task scheduling method provided by an embodiment of the present application;

[0036] Figure 2 This is a second flow chart of the multi-task scheduling method provided by an embodiment of the present application;

[0037] Figure 3 is a structural diagram of a multi-task scheduling device provided by an embodiment of the present application;

[0038] Figure 4 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0039] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0040] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0041] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0042] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0043] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0044] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0045] Figure 1 This is a first flow chart of a multi-task scheduling method provided by an embodiment of the present application. Figure 1 As shown, the method, applied to a server, comprises:

[0046] S11: Acquire the current status information of each vehicle, the information of each destination, and the information of the object.

[0047] In the application, the server communicates with each vehicle through the base station. The vehicle communicates with the server through the base station in real time and reports its own status information. The status information can be collected by the device installed in the vehicle or by the terminal device.

[0048] When materials are needed at various locations, the corresponding destination and destination conditions, i.e., the destination information, are generated. Correspondingly, the location where the materials are placed and the storage conditions of the materials, i.e., the information of the items, also needs to be known.

[0049] The server obtains the current status information of each vehicle, the information of each destination and the information of the object. The current status information includes the situation of the environment where the vehicle is located and the vehicle itself information.

[0050] S12: Input the current status information of each vehicle, the information of each destination and the information of the items into the optimized multi-agent reinforcement learning algorithm to obtain the delivery tasks and the target vehicles of the delivery tasks output by the optimized multi-agent reinforcement learning algorithm.

[0051] Among them, the feedback value obtained by the target vehicle when performing the delivery task is greater than the feedback value obtained by other vehicles when performing the delivery task.

[0052] The optimized multi-agent reinforcement learning algorithm is obtained after optimization using the historical feedback values ​​of each target vehicle. The optimized multi-agent reinforcement learning algorithm is used to determine each delivery task based on the information of each destination and the information of the item; for each delivery task, the communication connectivity of each vehicle and the reward value for executing the delivery task are determined based on the current state information of each vehicle, and the target vehicle is determined based on the communication connectivity and reward value of each vehicle; and each delivery task and the target vehicle for the delivery task are output, and the feedback value includes the communication connectivity and the reward value.

[0053] In the application, after each distribution task is assigned, the feedback value of each target vehicle can be obtained. At this time, the historical feedback value of each target vehicle can be obtained. The historical feedback value is used to fine-tune the multi-agent reinforcement learning algorithm being used to optimize the multi-agent reinforcement learning algorithm, so that the generation and distribution of distribution tasks are more accurate.

[0054] In a possible implementation, the current state information includes bandwidth, driving speed, first data transmission rate, CPU frequency, amount of processed data per unit time, and driving distance.

[0055] The communication connectivity of a vehicle is determined by the optimized multi-agent reinforcement learning algorithm based on the bandwidth, driving speed, first data transmission rate of the vehicle and the driving speed and first data transmission rate of other vehicles when the vehicle communicates with other vehicles.

[0056] In the application, when a vehicle enters the communication range of other vehicles, the two parties will communicate with each other, and at the same time collect their own bandwidth, driving speed, and first data transmission rate, as well as collect the driving speed and first data transmission rate of other vehicles through communication.

[0057] In the process of determining the target vehicle for the delivery task, the optimized multi-agent reinforcement learning algorithm determines the communication connectivity of the vehicle based on the vehicle's bandwidth, driving speed, first data transmission rate and the driving speed and first data transmission rate of other vehicles.

[0058] The calculation formula for communication connectivity is: com=(B·V1·r1) / (V2·r2); B is the bandwidth of the vehicle, V1 is the driving speed of the vehicle, r1 is the first data transmission rate of the vehicle; V2 is the driving speed of other vehicles, and r2 is the first data transmission rate of other vehicles.

[0059] It can be understood that communication connectivity can be used to determine which vehicles with good communication capabilities will take on delivery tasks, thereby improving vehicle reliability.

[0060] When the delivery task is an independent task, the reward value of the vehicle performing the delivery task is determined by the optimized multi-agent reinforcement learning algorithm based on the number of vehicles communicating with the server, the vehicle's data processing capability, and the evaluation value of the vehicle performing the delivery task.

[0061] In the application, the number of vehicles communicating with the server is the number of vehicles connected to the base station.

[0062] When the delivery task is a task that a vehicle completes independently, the optimized multi-agent reinforcement learning algorithm determines the reward value for the vehicle to perform the delivery task based on the number of vehicles connected to the base station, the vehicle's data processing capability, and the evaluation value of the vehicle's performance of the delivery task.

[0063] The reward value is calculated as follows: is the reward value determined by the base station according to the number of connected vehicles, D is the data processing capability and TV is the evaluation value.

[0064] It can be understood that the reward value for a vehicle performing a delivery task is determined based on the number of vehicles connected to the base station, that is, the reward value is adjusted based on the number of vehicles to undertake the delivery task. When the number of vehicles increases or decreases, the number of vehicles can be considered to better schedule and allocate tasks.

[0065] Among them, the vehicle's data processing capability is determined by the optimized multi-agent reinforcement learning algorithm based on the amount of data processed per unit time and the vehicle's CPU frequency and driving distance within a preset time period.

[0066] The calculation formula for data processing capacity is: f is the CPU frequency, Δt is the preset time period, d1 is the driving distance, and ad is the amount of processed data per unit time.

[0067] When the delivery task is a collaborative task, the reward value of the vehicle performing the delivery task is determined by the optimized multi-agent reinforcement learning algorithm based on the vehicle's data processing capabilities and the evaluation value of the vehicle performing the delivery task;

[0068] In the application, when the delivery task is a collaborative task and the collaborative task belongs to the same destination as some delivery tasks, the optimized multi-agent reinforcement learning algorithm determines the reward value based on the vehicle's data processing capability and the evaluation value of the vehicle's execution of the delivery task.

[0069] The calculation formula of the reward value is: Red2 = D·TV.

[0070] In one possible implementation, the evaluation value of the delivery task is determined based on the situation of the delivery task and the current state information of the vehicle to obtain a more accurate evaluation value.

[0071] The optimized multi-agent reinforcement learning algorithm is used to determine the importance of the delivery task according to the vehicle's overtaking probability and overtaking risk level; determine the urgency of the delivery task according to the vehicle's task completion time and the degree of environmental impact; determine the dependency of the delivery task according to the basic value and number of levels of the delivery task; and determine the evaluation value of the delivery task according to the importance, urgency and dependency of the delivery task.

[0072] In the application, the optimized multi-agent reinforcement learning algorithm determines the vehicle's overtaking probability and overtaking risk level based on the environmental information in the vehicle's current state information, and then determines the importance of the delivery task based on the vehicle's overtaking probability and overtaking risk level. Among them, the overtaking risk level is the risk level caused by overtaking. Environmental information includes the vehicle conditions and road conditions outside the vehicle.

[0073] The calculation formula for importance is: IM = TP·RD; TP is the overtaking probability and RD is the overtaking risk level.

[0074] The optimized multi-agent reinforcement learning algorithm determines the task completion time based on the vehicle's historical task execution, the vehicle's current state information, and the degree of environmental impact based on the vehicle's environmental information. Then, the urgency of the delivery task is determined based on the vehicle's task completion time and the degree of environmental impact. The degree of environmental impact refers to the degree of impact of the environment on driving.

[0075] The calculation formula for the degree of urgency is: UR = TD·AD; TD is the task completion time, and AD is the degree of environmental impact.

[0076] The optimized multi-agent reinforcement learning algorithm determines the basic value and number of levels of the delivery task according to the situation of the delivery task, and then determines the degree of dependency of the delivery task based on the basic value and number of levels of the delivery task. Among them, the basic value is the number of delivery tasks preceding the delivery task, and the number of levels is determined based on the distance between the destination of the delivery task and the location of the item on a delivery route. Generally speaking, for multiple destinations on a delivery route, the greater the distance between the destination of the delivery task and the location of the item, the greater the number of task levels; the smaller the distance between the destination of the delivery task and the location of the item, the smaller the number of task levels.

[0077] The calculation formula for the degree of dependence is: DC = FT·AF; FT is the basic value and AF is the number of levels.

[0078] For example, when a delivery task can only be executed after two other delivery tasks are completed, the number of the delivery tasks before the delivery task is two, and the basic value can be set to 2. And the distance between the destination of the delivery task and the location of the item is the second smallest, and the number of task levels is set to 2.

[0079] The optimized multi-agent reinforcement learning algorithm determines the evaluation value of the delivery task according to its importance, urgency and dependency.

[0080] The calculation formula for the assessment value is TV=IM·UR·DC; IM is the importance, UR is the urgency, and DC is the dependence.

[0081] By determining the importance of the delivery task based on the vehicle's overtaking probability and overtaking risk level; determining the urgency of the delivery task based on the vehicle's task completion time and the degree of environmental impact; determining the dependency of the delivery task based on the basic value and number of levels of the delivery task; and determining the evaluation value of the delivery task based on the importance, urgency and dependency of the delivery task, a more accurate evaluation value can be obtained.

[0082] S13: Send the corresponding delivery task to the target vehicle.

[0083] In the application, the base station sends the corresponding delivery tasks to the target vehicles, and each target vehicle performs the delivery of items according to the delivery tasks.

[0084] It can be understood that the reward value is determined by the assessed value of the delivery task, so that the vehicle that completes the task with the greatest value receives a larger reward value to ensure that the delivery task can be completed.

[0085] And the reward value is determined by the data processing capability, so that vehicles with strong data processing capabilities can obtain larger reward values, to ensure that the delivery task can be completed, and also enable the vehicle to complete the delivery task efficiently.

[0086] By maintaining communication between the server and the vehicle, the multi-agent reinforcement learning algorithm can be centrally trained on the server and decentralized executed on the vehicle side.

[0087] For example, in dynamic emergency mission rescue, the current status information of each vehicle, the information of each disaster-stricken point and the information of rescue materials are obtained; the current status information of each vehicle, the information of each disaster-stricken point and the information of rescue materials are input into the optimized multi-agent reinforcement learning algorithm, and the distribution tasks and the target vehicles of the distribution tasks output by the optimized multi-agent reinforcement learning algorithm are obtained, and the feedback value obtained by the target vehicle when performing the distribution task is greater than the feedback value obtained by other vehicles when performing the distribution task; better scheduling and allocation of multiple different quantities of materials are achieved to minimize the damage to the disaster-stricken points.

[0088] This embodiment includes inputting the current state information of each vehicle, the information of each destination and the information of the item into the optimized multi-agent reinforcement learning algorithm, obtaining each delivery task and the target vehicle of the delivery task output by the optimized multi-agent reinforcement learning algorithm, and the feedback value obtained by the target vehicle performing the delivery task is greater than the feedback value obtained by other vehicles performing the delivery task; sending the corresponding delivery task to the target vehicle; wherein the optimized multi-agent reinforcement learning algorithm is obtained after optimization using the historical feedback value of each target vehicle, and the optimized multi-agent reinforcement learning algorithm is used to determine each delivery task according to the information of each destination and the information of the item; for each delivery task, determining the communication connectivity of each vehicle and the reward value for performing the delivery task according to the current state information of each vehicle, and determining the target vehicle according to the communication connectivity and the reward value of each vehicle; and outputting each delivery task and the target vehicle of the delivery task, the feedback value includes the communication connectivity and the reward value, and by calculating the communication connectivity of each vehicle and the reward value for performing the delivery task, the optimal target vehicle for performing the delivery task is determined, so that the task scheduling and allocation can be better performed, so that each vehicle can undertake the delivery task that is suitable for it, and the completion of each delivery task is guaranteed.

[0089] In one embodiment, the optimized multi-agent reinforcement learning algorithm is also used to determine the reliability of the vehicle in performing the delivery task based on the vehicle's current state information. If the reliability is greater than a preset level, the reward value for the vehicle in performing the delivery task is determined based on the vehicle's current state information.

[0090] In the application, the optimized multi-agent reinforcement learning algorithm determines the probability of task incompleteness based on the current state information, and then determines the reliability based on the task completion time and the probability of task incompleteness.

[0091] The calculation formula for reliability is: g=TD(1-a); TD is the task completion time, and a is the probability of task incompletion.

[0092] This embodiment determines the reliability of the vehicle in performing the delivery task based on the vehicle's current status information. If the reliability is greater than a preset level, the reward value for the vehicle to perform the delivery task is determined based on the vehicle's current status information. The vehicle can only undertake the delivery task when it is determined that the vehicle is capable of undertaking the delivery task, thereby ensuring the completion of the delivery task.

[0093] In one embodiment, Figure 2 1 is a second flow chart of the multi-task scheduling method provided by an embodiment of the present application. Figure 2 As shown, the method further includes:

[0094] S14: Sending the feedback value of the corresponding delivery task to the target vehicle.

[0095] In the application, the feedback value of the corresponding delivery task is sent to the target vehicle through the base station, so that the target vehicle receives the feedback value of the delivery task through V2I (Vehicle-to-Infrastructure, vehicle-to-infrastructure communication), and the feedback value of the delivery task includes the feedback value of the independent task and the feedback value of the collaborative task. When the delivery task of the target vehicle is a collaborative task, it can also receive the feedback value of the delivery task sent by other vehicles through V2V (Vehicle to Vehicle technology, vehicle-to-vehicle communication technology), and the delivery tasks of other vehicles and the delivery tasks of the target vehicle belong to the same destination.

[0096] Receive feedback from independent tasks via V2I Receive feedback from other vehicles via V2V

[0097] In a possible implementation, feedback values ​​may be obtained from target vehicles to obtain historical feedback values ​​of each target vehicle. After the historical feedback values ​​stored in the server are corrected based on the historical feedback values ​​of each target vehicle, the multi-agent reinforcement learning algorithm is optimized.

[0098] This embodiment sends the feedback value of the corresponding delivery task to the target vehicle, so that the target vehicle stores the feedback value of the delivery task, and the server obtains the accurate feedback value of the target vehicle, so that the server can use the feedback value of the target vehicle later.

[0099] In one embodiment, the optimized multi-agent reinforcement learning algorithm is also used to determine the energy consumption, data processing capability, and second data transmission rate of each vehicle for each delivery task based on the current status information of each vehicle; and to determine the target vehicle based on the communication connectivity, feedback value, energy consumption, data processing capability, and second data transmission rate of each vehicle, wherein the energy consumption of the target vehicle is less than that of other vehicles, and the data processing capability and data transmission rate of the target vehicle are both greater than those of other vehicles.

[0100] In a possible implementation, the current status information includes antenna height, Gaussian white noise power, communication distance, communication path loss index, and communication transmission power.

[0101] The energy consumption is determined by the optimized multi-agent reinforcement learning algorithm based on the vehicle’s bandwidth, CPU frequency, and driving distance within a preset time period.

[0102] The energy consumption is calculated as follows: B is the bandwidth, f is the CPU frequency, d1 is the driving distance, and Δt is the preset time period.

[0103] The second data transmission rate is determined by the multi-agent reinforcement learning algorithm based on the vehicle's bandwidth, antenna height, Gaussian white noise power, the communication distance between the vehicle and the server, the communication path loss index, and the communication transmission power.

[0104] The calculation formula for the second data transmission rate is: B is the bandwidth, p is the communication transmission power, h is the antenna height, w is the Gaussian white noise power, d2 is the communication distance, and θ is the communication path loss index.

[0105] Specifically, p is the communication transmission power for communicating with the base station, and d2 is the communication distance for communicating with the base station.

[0106] Correspondingly, the optimized multi-agent reinforcement learning algorithm is obtained by optimizing the historical feedback values, energy consumption, data processing capabilities, and data transmission rate of each target vehicle.

[0107] This embodiment determines the target vehicle based on the communication connectivity, feedback value, energy consumption, data processing capability, and second data transmission rate of each vehicle, so that the vehicle that completes the delivery task to obtain the maximum reward value while consuming as little energy as possible can be selected, so as to better schedule and allocate the delivery tasks.

[0108] It should be understood that the order of execution of the steps in the above embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, the data collection in the above embodiments is compliant, and its use or implementation does not involve any infringement on the public interest.

[0109] Corresponding to the method described in the above embodiment, for the convenience of explanation, only the part related to the embodiment of the present application is shown.

[0110] In one embodiment, Figure 3 Schematic diagram of the structure of a multi-task scheduling device provided by an embodiment of the present application. Figure 3 As shown, the device comprises:

[0111] The communication module 10 is used to obtain the current status information of each vehicle, the information of each destination and the information of the item;

[0112] It is also used to send corresponding delivery tasks to target vehicles;

[0113] The task scheduling module 11 is used to input the current state information of each vehicle, the information of each destination and the information of the item into the optimized multi-agent reinforcement learning algorithm, obtain the delivery tasks and the target vehicles of the delivery tasks output by the optimized multi-agent reinforcement learning algorithm, and the feedback value obtained by the target vehicle when performing the delivery task is greater than the feedback value obtained by other vehicles when performing the delivery task;

[0114] Among them, the optimized multi-agent reinforcement learning algorithm is obtained after optimization using the historical feedback values ​​of each target vehicle. The optimized multi-agent reinforcement learning algorithm is used to determine each delivery task based on the information of each destination and the information of the item; for each delivery task, the communication connectivity of each vehicle and the reward value for executing the delivery task are determined according to the current state information of each vehicle, and the target vehicle is determined according to the communication connectivity and reward value of each vehicle; and each delivery task and the target vehicle of the delivery task are output, and the feedback value includes the communication connectivity and reward value.

[0115] Figure 4 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 4 As shown, the electronic device 2 of this embodiment includes: at least one processor 20 ( Figure 4 Only one is shown in the figure), a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, wherein the processor 20 implements the steps of any of the above-mentioned method embodiments when executing the computer program 22.

[0116] The electronic device 2 may be a cloud server computing device. The electronic device 2 may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will appreciate that Figure 4 It is only an example of the electronic device 2 and does not constitute a limitation on the electronic device 2. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0117] The processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0118] In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, such as a hard disk or memory of the electronic device 2. In other embodiments, the memory 21 may also be an external storage device of the electronic device 2, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 2. Further, the memory 21 may also include both an internal storage unit of the electronic device 2 and an external storage device. The memory 21 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 21 may also be used to temporarily store data that has been output or is to be output.

[0119] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0120] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0121] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0122] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some cases, the computer-readable medium cannot be an electric carrier signal and a telecommunication signal.

[0124] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0125] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0126] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be in the form of telecommunications, mechanical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A multi-task scheduling method, characterized in that: Applicable to servers, including: Obtain the current status information of each vehicle, the information of each destination and the information of items; Inputting the current state information of each of the vehicles, the information of each of the destinations, and the information of the items into the optimized multi-agent reinforcement learning algorithm, obtaining each delivery task output by the optimized multi-agent reinforcement learning algorithm and the target vehicle of the delivery task, wherein the feedback value obtained by the target vehicle when executing the delivery task is greater than the feedback value obtained by other vehicles when executing the delivery task; Sending a corresponding delivery task to the target vehicle; Among them, the optimized multi-agent reinforcement learning algorithm is obtained after optimization using the historical feedback values ​​of each target vehicle, and the optimized multi-agent reinforcement learning algorithm is used to determine each delivery task based on the information of each destination and the information of the item; for each delivery task, the communication connectivity of each vehicle and the reward value for executing the delivery task are determined based on the current status information of each vehicle, and the target vehicle is determined based on the communication connectivity and reward value of each vehicle; and each delivery task and the target vehicle of the delivery task are output, and the feedback value includes the communication connectivity and the reward value.

2. The method according to claim 1, characterized in that: The communication connectivity of the vehicle is determined by the optimized multi-agent reinforcement learning algorithm according to the bandwidth, driving speed, first data transmission rate of the vehicle and the driving speed and first data transmission rate of the other vehicles when the vehicle communicates with other vehicles; When the delivery task is an independent task, the reward value of the vehicle for executing the delivery task is determined by the optimized multi-agent reinforcement learning algorithm according to the number of vehicles communicating with the server, the data processing capacity of the vehicle, and the evaluation value of the vehicle for executing the delivery task; When the delivery task is a collaborative task, the reward value of the vehicle for executing the delivery task is determined by the optimized multi-agent reinforcement learning algorithm according to the data processing capability of the vehicle and the evaluation value of the vehicle for executing the delivery task; The data processing capacity of the vehicle is determined by the optimized multi-agent reinforcement learning algorithm according to the amount of data processed per unit time and the CPU frequency and driving distance of the vehicle within a preset time period; The current status information includes the bandwidth, the driving speed, the first data transmission rate, the CPU frequency, the amount of processed data per unit time, and the driving distance.

3. The method according to claim 2, characterized in that: The optimized multi-agent reinforcement learning algorithm is used to determine the importance of the delivery task according to the overtaking probability and overtaking risk level of the vehicle; and to determine the urgency of the delivery task according to the task completion time and environmental impact of the vehicle; Determining the dependency degree of the delivery task according to the basic value and the number of levels of the delivery task; and determining the evaluation value of the delivery task according to the importance, urgency and dependency of the delivery task; The basic value is the number of preceding delivery tasks of the delivery task, and the number of levels is determined according to the distance between the destination of the delivery task and the location of the item on a delivery route.

4. The method according to claim 1, characterized in that: The optimized multi-agent reinforcement learning algorithm is also used to determine the energy consumption, data processing capability, and second data transmission rate of each vehicle for each delivery task according to the current state information of each vehicle; and determine the target vehicle according to the communication connectivity, feedback value, energy consumption, data processing capability, and second data transmission rate of each vehicle, wherein the energy consumption of the target vehicle is less than that of other vehicles, and the data processing capability and second data transmission rate of the target vehicle are greater than the data processing capability and second data transmission rate of other vehicles; Correspondingly, the optimized multi-agent reinforcement learning algorithm is obtained after optimization using the historical feedback values, energy consumption, data processing capabilities, and second data transmission rate of each of the target vehicles.

5. The method according to claim 4, characterized in that: The energy consumption is determined by the optimized multi-agent reinforcement learning algorithm according to the bandwidth of the vehicle, the CPU frequency within a preset time period, and the driving distance; the second data transmission rate is determined by the optimized multi-agent reinforcement learning algorithm according to the bandwidth of the vehicle, the antenna height, the Gaussian white noise power, the communication distance between the vehicle and the server, the communication path loss index, and the communication transmission power; The current status information includes the antenna height, the Gaussian white noise power, the communication distance, the communication path loss index, and the communication transmission power.

6. The method according to any one of claims 1 to 5, characterized in that: Also includes: Sending a feedback value of the corresponding delivery task to the target vehicle.

7. The method according to claim 1, characterized in that: The optimized multi-agent reinforcement learning algorithm is also used to determine the reliability of the vehicle in performing the delivery task based on the current state information of the vehicle. If the reliability is greater than a preset level, the reward value for the vehicle to perform the delivery task is determined based on the current state information of the vehicle.

8. A multi-task scheduling device, characterized in that: include: A communication module is used to obtain the current status information of each vehicle, the information of each destination and the information of the items; It is also used to send corresponding delivery tasks to target vehicles; A task scheduling module is used to input the current state information of each vehicle, the information of each destination and the information of the item into the optimized multi-agent reinforcement learning algorithm, obtain each delivery task output by the optimized multi-agent reinforcement learning algorithm and the target vehicle of the delivery task, and the feedback value obtained by the target vehicle when executing the delivery task is greater than the feedback value obtained by other vehicles when executing the delivery task; The optimized multi-agent reinforcement learning algorithm is obtained after optimization using the historical feedback value of each target vehicle, and the optimized multi-agent reinforcement learning algorithm is used to determine each delivery task according to the information of each destination and the information of the item; for each delivery task, the communication connectivity of each vehicle and the reward value for executing the delivery task are determined according to the current state information of each vehicle, and the target vehicle is determined according to the communication connectivity and the reward value of each vehicle; And output each delivery task and the target vehicle of the delivery task, the feedback value includes the communication connectivity and the reward value.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images