A task offloading method, device, equipment and medium based on reinforcement learning

Abstract

The present application discloses a task unloading method, device, equipment and medium based on reinforcement learning, the method comprising: forming a sample with preset state data, preset task unloading instructions and preset reward value, forming a training set with multiple samples; determining a trained actor network and a trained critic network based on the training set; forming current state data with current resource utilization, current task data volume, second driving position and second driving speed; obtaining the current task unloading instruction output by the trained actor network based on the current state data; generating a model according to the current reward value and the sending coefficient, generating the current sending coefficient, and sending the current task unloading instruction to the current vehicle when the current sending coefficient is greater than the preset sending coefficient. The present application is conducive to improving the efficiency of task unloading.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a task offloading method, device, equipment and medium based on reinforcement learning. Background Art

[0002] In recent years, with the rapid development of the Internet of Vehicles, task offloading has also been widely used. Whether it is accurate decision-making for autonomous driving, instant feedback on real-time road conditions, or accurate monitoring of vehicle health status, the application of task offloading has made it smoother, more efficient, and more intelligent.

[0003] However, the sending process of the current task unloading instruction in the prior art is cumbersome, which is not conducive to improving the sending efficiency of the current task unloading instruction. The reason is that in the process of task unloading, the user needs to consider various information of task unloading and decide whether to send the current task unloading instruction. The whole process takes a lot of time. Therefore, the sending process of the current task unloading instruction in the prior art is cumbersome, which is not conducive to improving the sending efficiency of the current task unloading instruction. Summary of the invention

[0004] The embodiments of the present application provide a task offloading method, apparatus, device and medium based on reinforcement learning to solve the technical problem that the sending process of the current task offloading instruction in the above-mentioned prior art is cumbersome and not conducive to improving the sending efficiency of the current task offloading instruction.

[0005] In a first aspect, an embodiment of the present application provides a task offloading method based on reinforcement learning, which is applied to an edge server, wherein the edge server stores an actor network and a critic network, and the task offloading method includes:

[0006] Obtaining a preset resource utilization rate, a preset task data volume, and a preset first driving position and a first driving speed of a vehicle, and combining the preset resource utilization rate, the preset task data volume, the first driving position, and the first driving speed to form preset state data, wherein the preset task data volume is the data volume of a preset task to be processed;

[0007] The preset state data, the preset task uninstallation instruction corresponding to the preset state data, and the preset reward value corresponding to the preset state data are combined into a sample, and a plurality of the samples are combined into a training set;

[0008] Based on the training set, determining the trained actor network and the trained critic network;

[0009] Acquire the current resource utilization rate, the current task data volume, the second driving position and the second driving speed of the current vehicle, and combine the current resource utilization rate, the current task data volume, the second driving position and the second driving speed to form current state data, wherein the current task data volume is the data volume of the current task to be processed;

[0010] Inputting the current state data into the trained actor network, and obtaining the current task offloading instruction output by the trained actor network based on the current state data;

[0011] Inputting the current state data and the current task unloading instruction into the trained critic network, and obtaining a current reward value output by the trained critic network based on the current state data and the current task unloading instruction;

[0012] A current sending coefficient is generated according to the current reward value and a preset sending coefficient generation model, and when the current sending coefficient is greater than the preset sending coefficient, the current task unloading instruction is sent to the current vehicle.

[0013] Exemplarily, the generating a current sending coefficient according to the current reward value and a preset sending coefficient generating model, and when the current sending coefficient is greater than a preset sending coefficient, sending the current task unloading instruction to the current vehicle, comprises:

[0014] Obtaining a current signal-to-noise ratio and a current signal strength of the current vehicle;

[0015] Generate a current transmission coefficient according to a ratio of the current reward value to the first reward value, a ratio of the current signal-to-noise ratio to a preset signal-to-noise ratio, a ratio of the current signal strength to a preset signal strength, and a preset transmission coefficient generation model;

[0016] When the current sending coefficient is greater than the preset sending coefficient, the current task unloading instruction is sent to the current vehicle.

[0017] Among them, the sending coefficient generation model is a model for generating the current sending coefficient.

[0018] Exemplarily, the transmission coefficient generation model is:

[0019] T = M1 × T1 + M2 × T2 + M3 × T3;

[0020] Wherein, T is the current sending coefficient, M1 is the first weighting coefficient, M2 is the second weighting coefficient, M3 is the third weighting coefficient, and the sum of M1, M2 and M3 is 1;

[0021] The T1 is a ratio of the current reward value to the first reward value, the T2 is a ratio of the current signal-to-noise ratio to a preset signal-to-noise ratio, and the T3 is a ratio of the current signal strength to a preset signal strength.

[0022] In a possible implementation manner of the first aspect, the obtaining of a preset resource utilization rate and a preset task data volume, obtaining a preset first driving position and a first driving speed of a vehicle, and combining the preset resource utilization rate, the preset task data volume, the first driving position, and the first driving speed into preset state data includes:

[0023] Obtain a file path, and import the first file and the second file through the file path;

[0024] In the first file, the preset resource utilization rate and the preset task data volume are obtained, and in the second file, the first driving position and the first driving speed of the preset vehicle are obtained;

[0025] The preset resource utilization rate, the preset task data volume, the first driving position and the first driving speed are combined into the preset state data.

[0026] In a possible implementation manner of the first aspect, determining the trained actor network and the trained critic network based on the training set includes:

[0027] Inputting the preset state data in the training set into the actor network, and obtaining the prediction task offloading instruction output by the actor network based on the preset state data;

[0028] Inputting the preset state data and the prediction task offloading instruction into the trained critic network, and obtaining the predicted reward value output by the critic network based on the preset state data and the prediction task offloading instruction;

[0029] Using a preset loss function, generating a loss value between the preset reward value and the predicted reward value, and determining whether the loss value is less than a preset value;

[0030] When the loss value is less than the preset value, the training of the actor network and the critic network is stopped, and the trained actor network and the trained critic network are saved.

[0031] In a possible implementation manner of the first aspect, the acquiring a current resource utilization rate and a current task data volume, acquiring a second driving position and a second driving speed of the current vehicle, and combining the current resource utilization rate, the current task data volume, the second driving position, and the second driving speed to form current state data includes:

[0032] Obtaining the current resource utilization and the current task data volume;

[0033] Connecting the current vehicle via the Internet of Vehicles, and receiving the current vehicle position and the current vehicle speed sent by the current vehicle;

[0034] The current reward value, the current task data volume, the current vehicle position, and the current vehicle speed are combined into the current state data, and the current task data volume is the data volume of the current task to be processed.

[0035] In a possible implementation manner of the first aspect, inputting the current state data and the current task offloading instruction into the trained critic network, and obtaining a current reward value output by the trained critic network based on the current state data and the current task offloading instruction includes:

[0036] Importing a third file, and obtaining the trained critic network from the third file;

[0037] The current state data and the current task offloading instruction are input into the trained critic network, and a current reward value output by the trained critic network based on the current state data and the current task offloading instruction is obtained.

[0038] In a possible implementation manner of the first aspect, after generating a current sending coefficient according to the current reward value and a preset sending coefficient generation model, and sending the current task unloading instruction to the current vehicle when the current sending coefficient is greater than the preset sending coefficient, the task unloading method includes:

[0039] Receiving a computing task sent by the current vehicle according to the current task unloading instruction;

[0040] The computing task is processed to obtain a processing result, and the processing result is sent to the current vehicle.

[0041] In a possible implementation manner of the first aspect, after generating a current sending coefficient according to the current reward value and a preset sending coefficient generation model, and sending the current task unloading instruction to the current vehicle when the current sending coefficient is greater than the preset sending coefficient, the task unloading method includes:

[0042] The vehicle identification of the current vehicle is obtained, and the vehicle identification and the current task unloading instruction are recorded and stored.

[0043] In a second aspect, an embodiment of the present application provides a task offloading device based on reinforcement learning, which is applied to an edge server, wherein the edge server stores an actor network and a critic network, including:

[0044] A first acquisition module is used to acquire a preset resource utilization rate, a preset task data volume, and a preset first driving position and a first driving speed of a vehicle, and to combine the preset resource utilization rate, the preset task data volume, the first driving position, and the first driving speed into preset state data, wherein the preset task data volume is a preset data volume of a task to be processed;

[0045] a composition module, used to combine the preset state data, the preset task uninstallation instruction corresponding to the preset state data, and the preset reward value corresponding to the preset state data into a sample, and combine a plurality of the samples into a training set;

[0046] A determination module, used for determining the trained actor network and the trained critic network based on the training set;

[0047] A second acquisition module is used to acquire the current resource utilization rate, the current task data volume, the second driving position and the second driving speed of the current vehicle, and to combine the current resource utilization rate, the current task data volume, the second driving position and the second driving speed into current state data, wherein the current task data volume is the data volume of the current task to be processed;

[0048] a third acquisition module, configured to input the current state data into the trained actor network, and acquire the current task unloading instruction output by the trained actor network based on the current state data;

[0049] A fourth acquisition module, used for inputting the current state data and the current task unloading instruction into the trained critic network, and acquiring a current reward value output by the trained critic network based on the current state data and the current task unloading instruction;

[0050] The sending module is used to generate a current sending coefficient according to the current reward value and a preset sending coefficient generation model, and when the current sending coefficient is greater than the preset sending coefficient, send the current task unloading instruction to the current vehicle.

[0051] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the task offloading method in the first aspect when executing the computer program.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the task offloading method in the first aspect is implemented.

[0053] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the task offloading method in the first aspect.

[0054] The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, a current sending coefficient is generated according to the current reward value and a preset sending coefficient generation model. When the current sending coefficient is greater than the preset sending coefficient, the current task unloading instruction is sent to the current vehicle. This process does not require human intervention, which saves time and is therefore beneficial to improving the efficiency of sending the current task unloading instruction. On the other hand, by using the sending coefficient generation model, no human intervention is required, which greatly reduces human delays and is therefore beneficial to timely sending the current task unloading instruction to the current vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 An application scenario diagram of the task offloading method provided in an embodiment of the present application;

[0057] Figure 2 It is a flowchart of a task offloading method provided in an embodiment of the present application;

[0058] Figure 3 A flowchart of sending processing results provided in an embodiment of the present application;

[0059] Figure 4A schematic block diagram of a task offloading device provided in an embodiment of the present application;

[0060] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0062] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0063] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0064] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0065] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0066] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0067] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0068] In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0069] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0070] The task offloading method provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific types of electronic devices.

[0071] For example, the electronic device may be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a vehicle networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, a wireless modem card, a TV set top box (STB), a customer premises equipment (CPE) and / or other devices for communicating on a wireless system and a next generation communication system, for example, a mobile terminal in a 5G network or a mobile terminal in a future evolved Public Land Mobile Network (PLMN) network, etc.

[0072] See also Figure 1 , Figure 1 The application scenario diagram of the task offloading method provided in the embodiment of the present application is described in detail as follows:

[0073] The edge server is connected to the on-board device of the current vehicle.

[0074] The edge server obtains the current vehicle position and the current vehicle speed sent by the vehicle-mounted device.

[0075] The edge server combines the current reward value, the current task data volume, the current vehicle position, and the current vehicle speed into the current state data, where the current task data volume is the data volume of the current task to be processed.

[0076] In an embodiment of the present application, the edge server is connected to the vehicle-mounted device to obtain the current vehicle position and current vehicle speed sent by the vehicle-mounted device.

[0077] See also Figure 2 , Figure 2 It is a flow chart of a task offloading method provided in an embodiment of the present application. The method can be applied to an edge server, and the edge server stores an actor network and a critic network.

[0078] like Figure 2 As shown, the task offloading method provided in the embodiment of the present application includes the following steps, which are described in detail as follows:

[0079] S201, obtaining a preset resource utilization rate, a preset task data volume, and a preset first driving position and a first driving speed of a vehicle, and combining the preset resource utilization rate, the preset task data volume, the first driving position, and the first driving speed into preset state data, wherein the preset task data volume is the data volume of the preset task to be processed;

[0080] The step of obtaining a preset resource utilization rate and a preset task data volume, obtaining a preset first driving position and a first driving speed of a vehicle, and combining the preset resource utilization rate, the preset task data volume, the first driving position and the first driving speed into preset state data includes:

[0081] Obtain a file path, and import the first file and the second file through the file path;

[0082] In the first file, the preset resource utilization rate and the preset task data volume are obtained, and in the second file, the first driving position and the first driving speed of the preset vehicle are obtained;

[0083] The preset resource utilization rate, the preset task data volume, the first driving position and the first driving speed are combined into the preset state data.

[0084] For the sake of illustration, an example is given below:

[0085] For example, the preset resource utilization is 50%, which means that the edge server is running and the CPU and memory of the edge server are being effectively utilized.

[0086] For example, if the preset task data volume is 10 GB, it means that the preset task data volume that has not been processed by the edge server is 10 GB.

[0087] The first driving position of the preset vehicle is: longitude: 112.9823°; latitude: 28.1941°;

[0088] The first driving speed of the preset vehicle: Driving speed: 60 km / h, which means that the preset vehicle is currently traveling at a speed of 60 km per hour.

[0089] Combining these two pieces of information, it is described as: the vehicle is located at longitude 112.9823°, latitude 28.1941°, and is traveling at a speed of 60 km / h.

[0090] S202, combining the preset state data, the preset task uninstallation instruction corresponding to the preset state data, and the preset reward value corresponding to the preset state data into a sample, and combining a plurality of the samples into a training set;

[0091] The preset state data, the preset task unloading instruction corresponding to the preset state data, and the preset reward value corresponding to the preset state data are combined into a sample, and a preset number of samples are combined into a training set.

[0092] The preset quantity is set by the user or by the system default and is not limited here.

[0093] For the sake of illustration, an example is given below:

[0094] For example, the preset state data, the preset task unloading instruction corresponding to the preset state data, and the preset reward value corresponding to the preset state data are combined into a sample, and 10,000 samples are combined into a training set.

[0095] For example, the preset state data, the preset task unloading instruction corresponding to the preset state data, and the preset reward value corresponding to the preset state data are combined into a sample, and 20,000 samples are combined into a training set.

[0096] For example, the preset state data, the preset task unloading instruction corresponding to the preset state data, and the preset reward value corresponding to the preset state data are combined into a sample, and 50,000 samples are combined into a training set.

[0097] The training set is used to train the actor network and the critic network. Through the training set, the actor network and the critic network can learn patterns, features and relationships in the data, so as to make predictions on unseen data.

[0098] S203, determining the trained actor network and the trained critic network based on the training set;

[0099] Among them, Actor network and Critic network are two important networks in deep reinforcement learning. Actor network is also called policy network, and Critic network is also called value network.

[0100] Wherein, determining the trained actor network and the trained critic network based on the training set includes:

[0101] Inputting the preset state data in the training set into the actor network, and obtaining the prediction task offloading instruction output by the actor network based on the preset state data;

[0102] Inputting the preset state data and the prediction task offloading instruction into the trained critic network, and obtaining the predicted reward value output by the critic network based on the preset state data and the prediction task offloading instruction;

[0103] Using a preset loss function, generating a loss value between the preset reward value and the predicted reward value, and determining whether the loss value is less than a preset value;

[0104] When the loss value is less than the preset value, the training of the actor network and the critic network is stopped, and the trained actor network and the trained critic network are saved.

[0105] The loss function includes one of a mean square error function, a cross entropy loss function, and a logarithmic loss function, or a combination thereof.

[0106] S204, obtaining the current resource utilization rate, the current task data volume, the second driving position and the second driving speed of the current vehicle, and combining the current resource utilization rate, the current task data volume, the second driving position and the second driving speed to form current state data, wherein the current task data volume is the data volume of the current task to be processed;

[0107] The obtaining of the current resource utilization rate, the current task data volume, the second driving position and the second driving speed of the current vehicle, and the current resource utilization rate, the current task data volume, the second driving position and the second driving speed forming the current state data include:

[0108] Obtaining the current resource utilization and the current task data volume;

[0109] Connecting the current vehicle via the Internet of Vehicles, and receiving the current vehicle position and the current vehicle speed sent by the current vehicle;

[0110] The current reward value, the current task data volume, the current vehicle position, and the current vehicle speed are combined into the current state data, and the current task data volume is the data volume of the current task to be processed.

[0111] For the sake of illustration, an example is given below:

[0112] For example, if the current resource utilization is 60%, it means that the edge server is running and the CPU and memory of the edge server are being effectively used.

[0113] For example, if the current task data volume is 5 GB, it means that the current task data volume that has not been processed by the edge server is 5 GB.

[0114] The second driving position of the current vehicle: longitude: 111.2833°; latitude: 27.8119°;

[0115] The second driving speed of the current vehicle: driving speed: 80 kilometers per hour, which means that the current vehicle is currently traveling at a speed of 80 kilometers per hour.

[0116] Combining these two pieces of information, it is described as: the current vehicle is located at longitude 111.2833°, latitude 27.8119°, and is traveling at a speed of 80 km / h.

[0117] S205, inputting the current state data into the trained actor network, and obtaining the current task offloading instruction output by the trained actor network based on the current state data;

[0118] The step of inputting the current state data and the current task unloading instruction into the trained critic network, and obtaining the current reward value output by the trained critic network based on the current state data and the current task unloading instruction, comprises:

[0119] Import a third file, and obtain the trained critic network in the third file; input the current state data and the current task unloading instruction into the trained critic network, and obtain the current reward value output by the trained critic network based on the current state data and the current task unloading instruction.

[0120] S206, inputting the current state data and the current task unloading instruction into the trained critic network, and obtaining a current reward value output by the trained critic network based on the current state data and the current task unloading instruction;

[0121] S207, generating a current sending coefficient according to the current reward value and a preset sending coefficient generation model, and sending the current task unloading instruction to the current vehicle when the current sending coefficient is greater than the preset sending coefficient.

[0122] Exemplarily, the generating a current sending coefficient according to the current reward value and a preset sending coefficient generating model, and when the current sending coefficient is greater than a preset sending coefficient, sending the current task unloading instruction to the current vehicle, comprises:

[0123] Obtaining a current signal-to-noise ratio and a current signal strength of the current vehicle;

[0124] Generate a current transmission coefficient according to a ratio of the current reward value to the first reward value, a ratio of the current signal-to-noise ratio to a preset signal-to-noise ratio, a ratio of the current signal strength to a preset signal strength, and a preset transmission coefficient generation model;

[0125] When the current sending coefficient is greater than the preset sending coefficient, the current task unloading instruction is sent to the current vehicle.

[0126] Among them, the sending coefficient generation model is a model for generating the current sending coefficient.

[0127] Exemplarily, the transmission coefficient generation model is:

[0128] T = M1 × T1 + M2 × T2 + M3 × T3;

[0129] Wherein, T is the current sending coefficient, M1 is the first weighting coefficient, M2 is the second weighting coefficient, M3 is the third weighting coefficient, and the sum of M1, M2 and M3 is 1;

[0130] The T1 is a ratio of the current reward value to the first reward value, the T2 is a ratio of the current signal-to-noise ratio to a preset signal-to-noise ratio, and the T3 is a ratio of the current signal strength to a preset signal strength.

[0131] Exemplarily, the higher the current transmission coefficient, the greater the current reward value, the current signal-to-noise ratio, and the current signal strength; the lower the current transmission coefficient, the smaller the current reward value, the current signal-to-noise ratio, and the current signal strength.

[0132] The preset sending coefficient and the first reward value are set by the user or by the system by default, and are not limited here.

[0133] The M1, M2, and M3 are set by the user or by the system by default and are not limited here.

[0134] For the sake of illustration, an example is given below:

[0135] For example, the M1 is 0.3, the M2 is 0.3, the M3 is 0.4, the T1 is 1.2, the T2 is 1.1, and the T3 is 1.3;

[0136] T=0.3×1.2+0.3×1.1+0.4×1.1=1.13;

[0137] For example, the M1 is 0.5, the M2 is 0.2, the M3 is 0.3, the T1 is 1.2, the T2 is 1.1, and the T3 is 1.3;

[0138] T=0.5×1.2+0.2×1.1+0.3×1.1=1.15.

[0139] For example, the M1 is 0.6, the M2 is 0.2, the M3 is 0.2, the T1 is 1.6, the T2 is 1.2, and the T3 is 1.3;

[0140] T=0.6×1.6+0.2×1.2+0.2×1.3=1.46.

[0141] The preset sending coefficient is set by the user or by the system default and is not restricted here.

[0142] For the sake of illustration, an example is given below:

[0143] For example, the preset transmission coefficient is 1.1;

[0144] When the current sending coefficient is 1.13, the current sending coefficient is greater than the preset sending coefficient. Therefore, the edge server sends the current task unloading instruction to the current vehicle.

[0145] For example, the preset transmission coefficient is 1.12;

[0146] When the current sending coefficient is 1.13, the current sending coefficient is greater than the preset sending coefficient, so the edge server sends the current task unloading instruction to the current vehicle.

[0147] For example, the preset transmission coefficient is 1.3;

[0148] When the current sending coefficient is 1.46, the current sending coefficient is greater than the preset sending coefficient, so the edge server sends the current task unloading instruction to the current vehicle.

[0149] Among them, when the current sending coefficient is greater than the preset sending coefficient, it means that the current sending coefficient is better than expected, which is conducive to ensuring the reliability of sending the current task offloading instruction.

[0150] Wherein, after generating the current sending coefficient according to the current reward value and the preset sending coefficient generation model, and sending the current task unloading instruction to the current vehicle when the current sending coefficient is greater than the preset sending coefficient, the task unloading method includes:

[0151] Step A, obtaining the vehicle identification of the current vehicle, and recording and storing the vehicle identification and the current task unloading instruction.

[0152] Wherein, the vehicle identification and the current task unloading instruction are recorded and stored.

[0153] Among them, the record of uninstallation instructions is crucial for the maintenance and monitoring of edge servers. By monitoring uninstallation events, users can promptly discover potential problems of edge servers, such as storage device failures and network problems.

[0154] The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, a current sending coefficient is generated according to the current reward value and a preset sending coefficient generation model. When the current sending coefficient is greater than the preset sending coefficient, the current task unloading instruction is sent to the current vehicle. This process does not require human intervention, which saves time and is therefore beneficial to improving the efficiency of sending the current task unloading instruction. On the other hand, by using the sending coefficient generation model, no human intervention is required, which greatly reduces human delays and is therefore beneficial to timely sending the current task unloading instruction to the current vehicle.

[0155] See also Figure 3 , Figure 3 The flowchart of sending the processing result provided in the embodiment of the present application is described in detail as follows:

[0156] S301, receiving a computing task sent by the current vehicle according to the current task unloading instruction;

[0157] The current task uninstall instruction includes a complete uninstall instruction, a partial uninstall instruction, or a combination thereof.

[0158] Receiving a computing task sent by the current vehicle according to the current task unloading instruction, comprising:

[0159] receiving a computing task sent by the current vehicle according to the complete unloading instruction;

[0160] Alternatively, a computing task sent by the current vehicle according to the partial unloading instruction is received.

[0161] For the sake of illustration, an example is given below:

[0162] For example, the current vehicle has computing tasks A, B, and C;

[0163] When the current computing task unloading instruction is a complete unloading instruction, receiving computing task A, computing task B, and computing task C sent by the current vehicle according to the complete unloading instruction;

[0164] When the current computing task unloading instruction is a partial unloading instruction, computing task A and computing task B sent by the current vehicle according to the partial unloading instruction are received.

[0165] S302, processing the computing task to obtain a processing result, and sending the processing result to the current vehicle.

[0166] There are two ways to process the computing task, obtain the processing result, and send the processing result to the current vehicle;

[0167] In the first mode, the edge server directly processes the computing task, obtains a processing result, and sends the processing result to the current vehicle;

[0168] In the second mode, the edge server and other edge servers process the computing task together to obtain a processing result, and send the processing result to the current vehicle.

[0169] Among them, other edge servers are edge servers other than the current edge server.

[0170] For ease of explanation, take the first method as an example, as follows:

[0171] For example, there are computing tasks A, B, and C. Computing task A is processed to obtain processing result A, computing task B is processed to obtain processing result B, computing task C is processed to obtain processing result C, and processing results A, B, and C are sent to the current vehicle.

[0172] For example, there are computing tasks A and B. Computing task A is processed to obtain processing result A. Computing task B is processed to obtain processing result B. Processing results A and B are sent to the current vehicle.

[0173] For the sake of explanation, the second method is taken as an example as follows:

[0174] For example, there is a computing task A, which is divided into a plurality of easy-to-execute first subtasks, a second subtask, and a third subtask;

[0175] The first subtask is calculated by the edge server; the second subtask and the third subtask are unloaded by the edge server to other edge servers, and the second subtask and the third subtask are calculated by other edge servers;

[0176] After calculating the first subtask, the edge server obtains the first subresult;

[0177] After calculating the second subtask and the third subtask, the other edge servers obtain the second subresult corresponding to the second subtask and the third subresult corresponding to the third subtask;

[0178] 1. The second sub-result and the third sub-result may be returned to the edge server, and the edge server returns the second sub-result and the third sub-result to the current vehicle;

[0179] 2. The second sub-result and the third sub-result may also be directly transmitted to the next edge server for vehicle path prediction through other edge servers, and the second sub-result and the third sub-result may be returned to the current vehicle through the next edge server.

[0180] Step A may be executed before or after S301, or simultaneously with S301. The specific execution order is not limited here.

[0181] In an embodiment of the present application, the computing task is processed to obtain a processing result, and the processing result is sent to the current vehicle, which not only reduces the computing pressure of the on-board equipment, but also improves the efficiency and speed of processing the computing task.

[0182] For the task offloading method described in the above embodiment, please refer to Figure 4 , Figure 4 A schematic block diagram of a task offloading device provided in an embodiment of the present application, Figure 4 The task offloading device 400 shown can be applied to Figure 1 The electronic device in the application scenario diagram shown below takes the electronic device as an example. Figure 4 The task offloading device 400 shown in the figure is described in detail. The task offloading device 400 may include a first acquisition module 401, a composition module 402, a determination module 403, a second acquisition module 404, a third acquisition module 405, a fourth acquisition module 406, and a sending module 407.

[0183] The first acquisition module 401 is used to acquire a preset resource utilization rate, a preset task data volume, and a preset first driving position and a first driving speed of the vehicle, and to combine the preset resource utilization rate, the preset task data volume, the first driving position, and the first driving speed into preset state data, wherein the preset task data volume is a preset data volume of the task to be processed;

[0184] A composition module 402 is used to combine the preset state data, the preset task uninstallation instruction corresponding to the preset state data, and the preset reward value corresponding to the preset state data into a sample, and combine multiple samples into a training set;

[0185] A determination module 403 is used to determine the trained actor network and the trained critic network based on the training set;

[0186] The second acquisition module 404 is used to acquire the current resource utilization rate, the current task data volume, the second driving position and the second driving speed of the current vehicle, and combine the current resource utilization rate, the current task data volume, the second driving position and the second driving speed to form current state data, where the current task data volume is the data volume of the current task to be processed;

[0187] A third acquisition module 405 is used to input the current state data into the trained actor network, and obtain the current task unloading instruction output by the trained actor network based on the current state data;

[0188] A fourth acquisition module 406 is used to input the current state data and the current task unloading instruction into the trained critic network, and obtain a current reward value output by the trained critic network based on the current state data and the current task unloading instruction;

[0189] The sending module 407 is used to generate a current sending coefficient according to the current reward value and a preset sending coefficient generation model, and when the current sending coefficient is greater than the preset sending coefficient, send the current task unloading instruction to the current vehicle.

[0190] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0191] The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, a current sending coefficient is generated according to the current reward value and a preset sending coefficient generation model. When the current sending coefficient is greater than the preset sending coefficient, the current task unloading instruction is sent to the current vehicle. This process does not require human intervention, which saves time and is therefore beneficial to improving the efficiency of sending the current task unloading instruction. On the other hand, by using the sending coefficient generation model, no human intervention is required, which greatly reduces human delays and is therefore beneficial to timely sending the current task unloading instruction to the current vehicle.

[0192] See also Figure 5 , Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0193] like Figure 5 As shown, Figure 5 The electronic device 2 includes: at least one processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, and when the processor 20 executes the computer program 22, the steps in any of the above-mentioned method embodiments are implemented.

[0194] The electronic device 2 may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will appreciate that Figure 5It is only an example of the electronic device 2 and does not constitute a limitation on the electronic device 2. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0195] The processor 20 is used to run the computer program 22 stored in the memory 21, and implement the following steps when executing the computer program 22:

[0196] Obtaining a preset resource utilization rate, a preset task data volume, and a preset first driving position and a first driving speed of a vehicle, and combining the preset resource utilization rate, the preset task data volume, the first driving position, and the first driving speed to form preset state data, wherein the preset task data volume is the data volume of a preset task to be processed;

[0197] The preset state data, the preset task uninstallation instruction corresponding to the preset state data, and the preset reward value corresponding to the preset state data are combined into a sample, and a plurality of the samples are combined into a training set;

[0198] Based on the training set, determining the trained actor network and the trained critic network;

[0199] Acquire the current resource utilization rate, the current task data volume, the second driving position and the second driving speed of the current vehicle, and combine the current resource utilization rate, the current task data volume, the second driving position and the second driving speed to form current state data, wherein the current task data volume is the data volume of the current task to be processed;

[0200] The current state data is input into the trained actor network, and the current task unloading instruction output by the trained actor network based on the current state data is obtained; the current state data and the current task unloading instruction are input into the trained critic network, and the current reward value output by the trained critic network based on the current state data and the current task unloading instruction is obtained; a current sending coefficient is generated according to the current reward value and a preset sending coefficient generation model, and when the current sending coefficient is greater than the preset sending coefficient, the current task unloading instruction is sent to the current vehicle.

[0201] In some embodiments, the processor 20 is configured to implement:

[0202] Obtain a file path, and import the first file and the second file through the file path;

[0203] In the first file, the preset resource utilization rate and the preset task data volume are obtained, and in the second file, the first driving position and the first driving speed of the preset vehicle are obtained;

[0204] The preset resource utilization rate, the preset task data volume, the first driving position and the first driving speed are combined into the preset state data.

[0205] In some embodiments, the processor 20 is configured to implement:

[0206] Inputting the preset state data in the training set into the actor network, and obtaining the prediction task offloading instruction output by the actor network based on the preset state data;

[0207] Inputting the preset state data and the prediction task offloading instruction into the trained critic network, and obtaining the predicted reward value output by the critic network based on the preset state data and the prediction task offloading instruction;

[0208] Using a preset loss function, generating a loss value between the preset reward value and the predicted reward value, and determining whether the loss value is less than a preset value;

[0209] When the loss value is less than the preset value, the training of the actor network and the critic network is stopped, and the trained actor network and the trained critic network are saved.

[0210] In some embodiments, the processor 20 is configured to implement:

[0211] Obtaining the current resource utilization and the current task data volume;

[0212] Connecting the current vehicle via the Internet of Vehicles, and receiving the current vehicle position and the current vehicle speed sent by the current vehicle;

[0213] The current reward value, the current task data volume, the current vehicle position, and the current vehicle speed are combined into the current state data, and the current task data volume is the data volume of the current task to be processed.

[0214] In some embodiments, the processor 20 is configured to implement:

[0215] Importing a third file, and obtaining the trained critic network from the third file;

[0216] The current state data and the current task offloading instruction are input into the trained critic network, and a current reward value output by the trained critic network based on the current state data and the current task offloading instruction is obtained.

[0217] In some embodiments, the processor 20 is configured to implement:

[0218] Receiving a computing task sent by the current vehicle according to the current task unloading instruction;

[0219] The computing task is processed to obtain a processing result, and the processing result is sent to the current vehicle.

[0220] In some embodiments, the processor 20 is configured to implement:

[0221] The vehicle identification of the current vehicle is obtained, and the vehicle identification and the current task unloading instruction are recorded and stored.

[0222] The processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0223] In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, such as a hard disk or memory of the electronic device 2. In other embodiments, the memory 21 may also be an external storage device of the electronic device 2, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 2. Further, the memory 21 may also include both an internal storage unit and an external storage device of the electronic device 2. The memory 21 is used to store an operating system, an application program, a boot loader, data, and other programs, such as the program code of the computer program. The memory 21 may also be used to temporarily store data that has been output or is to be output.

[0224] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0225] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0226] The computer-readable storage medium stores program codes, which can be called by a processor to execute the task offloading method described in the above method embodiment.

[0227] The computer-readable storage medium has a storage space for program codes.

[0228] The program code includes the code of any step in the task offloading method described in the above method embodiment.

[0229] For example, the program code is called by the processor and can execute the following steps:

[0230] Obtaining a preset resource utilization rate, a preset task data volume, and a preset first driving position and a first driving speed of a vehicle, and combining the preset resource utilization rate, the preset task data volume, the first driving position, and the first driving speed to form preset state data, wherein the preset task data volume is the data volume of a preset task to be processed;

[0231] The preset state data, the preset task uninstallation instruction corresponding to the preset state data, and the preset reward value corresponding to the preset state data are combined into a sample, and a plurality of the samples are combined into a training set;

[0232] Based on the training set, determining the trained actor network and the trained critic network;

[0233] Acquire the current resource utilization rate, the current task data volume, the second driving position and the second driving speed of the current vehicle, and combine the current resource utilization rate, the current task data volume, the second driving position and the second driving speed to form current state data, wherein the current task data volume is the data volume of the current task to be processed;

[0234] Inputting the current state data into the trained actor network, and obtaining the current task offloading instruction output by the trained actor network based on the current state data;

[0235] Inputting the current state data and the current task unloading instruction into the trained critic network, and obtaining a current reward value output by the trained critic network based on the current state data and the current task unloading instruction;

[0236] A current sending coefficient is generated according to the current reward value and a preset sending coefficient generation model, and when the current sending coefficient is greater than the preset sending coefficient, the current task unloading instruction is sent to the current vehicle.

[0237] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0238] Among them, the computer-readable storage medium can also be an external storage device of the task offloading device or the electronic device, for example, a plug-in hard disk equipped on the task offloading device or the electronic device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), a non-volatile computer-readable medium (non-transitory computer-readable storage medium), etc.

[0239] Since the computer program stored in the computer-readable storage medium can execute any one of the reinforcement learning-based task offloading methods provided in the embodiments of the present application, the computer-readable storage medium can achieve the beneficial effects that can be achieved by any one of the reinforcement learning-based task offloading methods provided in the embodiments of the present application. For details, please refer to the previous embodiments and will not be repeated here.

[0240] An embodiment of the present application provides a computer program product. When the computer program product is executed on an electronic device, the electronic device executes the above-mentioned task offloading method.

[0241] The computer program product is loaded into the electronic device and can execute the following steps:

[0242] Obtaining a preset resource utilization rate, a preset task data volume, and a preset first driving position and a first driving speed of a vehicle, and combining the preset resource utilization rate, the preset task data volume, the first driving position, and the first driving speed to form preset state data, wherein the preset task data volume is the data volume of a preset task to be processed;

[0243] The preset state data, the preset task uninstallation instruction corresponding to the preset state data, and the preset reward value corresponding to the preset state data are combined into a sample, and a plurality of the samples are combined into a training set;

[0244] Based on the training set, determining the trained actor network and the trained critic network;

[0245] Acquire the current resource utilization rate, the current task data volume, the second driving position and the second driving speed of the current vehicle, and combine the current resource utilization rate, the current task data volume, the second driving position and the second driving speed to form current state data, wherein the current task data volume is the data volume of the current task to be processed;

[0246] Inputting the current state data into the trained actor network, and obtaining the current task offloading instruction output by the trained actor network based on the current state data;

[0247] Inputting the current state data and the current task unloading instruction into the trained critic network, and obtaining a current reward value output by the trained critic network based on the current state data and the current task unloading instruction;

[0248] A current sending coefficient is generated according to the current reward value and a preset sending coefficient generation model, and when the current sending coefficient is greater than the preset sending coefficient, the current task unloading instruction is sent to the current vehicle.

[0249] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0250] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0251] Based on such understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0252] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0253] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A task offloading method based on reinforcement learning, characterized in that: Applied to an edge server, the edge server stores an actor network and a critic network, and the task offloading method includes: Obtaining a preset resource utilization rate, a preset task data volume, and a preset first driving position and a first driving speed of a vehicle, and combining the preset resource utilization rate, the preset task data volume, the first driving position, and the first driving speed to form preset state data, wherein the preset task data volume is the data volume of a preset task to be processed; The preset state data, the preset task uninstallation instruction corresponding to the preset state data, and the preset reward value corresponding to the preset state data are combined into a sample, and a plurality of the samples are combined into a training set; Based on the training set, determining the trained actor network and the trained critic network; Acquire the current resource utilization rate, the current task data volume, the second driving position and the second driving speed of the current vehicle, and combine the current resource utilization rate, the current task data volume, the second driving position and the second driving speed to form current state data, wherein the current task data volume is the data volume of the current task to be processed; Inputting the current state data into the trained actor network, and obtaining the current task offloading instruction output by the trained actor network based on the current state data; Inputting the current state data and the current task unloading instruction into the trained critic network, and obtaining a current reward value output by the trained critic network based on the current state data and the current task unloading instruction; Generate a current sending coefficient according to the current reward value and a preset sending coefficient generation model, and send the current task unloading instruction to the current vehicle when the current sending coefficient is greater than the preset sending coefficient; The generating a current sending coefficient according to the current reward value and a preset sending coefficient generating model, and when the current sending coefficient is greater than the preset sending coefficient, sending the current task unloading instruction to the current vehicle, comprises: Obtaining a current signal-to-noise ratio and a current signal strength of the current vehicle; Generate a current transmission coefficient according to a ratio of the current reward value to the first reward value, a ratio of the current signal-to-noise ratio to a preset signal-to-noise ratio, a ratio of the current signal strength to a preset signal strength, and a preset transmission coefficient generation model; When the current sending coefficient is greater than the preset sending coefficient, sending the current task unloading instruction to the current vehicle; Wherein, the sending coefficient generation model is a model for generating the current sending coefficient; The transmission coefficient generation model is: T = M1 × T1 + M2 × T2 + M3 × T3; Wherein, T is the current sending coefficient, M1 is the first weighting coefficient, M2 is the second weighting coefficient, M3 is the third weighting coefficient, and the sum of M1, M2 and M3 is 1; The T1 is a ratio of the current reward value to the first reward value, the T2 is a ratio of the current signal-to-noise ratio to a preset signal-to-noise ratio, and the T3 is a ratio of the current signal strength to a preset signal strength.

2. The task offloading method according to claim 1, characterized in that: The step of obtaining a preset resource utilization rate and a preset task data volume, obtaining a preset first driving position and a first driving speed of a vehicle, and combining the preset resource utilization rate, the preset task data volume, the first driving position and the first driving speed into preset state data includes: Obtain a file path, and import the first file and the second file through the file path; In the first file, the preset resource utilization rate and the preset task data volume are obtained, and in the second file, the first driving position and the first driving speed of the preset vehicle are obtained; The preset resource utilization rate, the preset task data volume, the first driving position and the first driving speed are combined into the preset state data.

3. The task offloading method according to claim 1, characterized in that: The step of determining the trained actor network and the trained critic network based on the training set includes: Inputting the preset state data in the training set into the actor network, and obtaining the actor network outputting a prediction task offloading instruction based on the preset state data; Inputting the preset state data and the prediction task offloading instruction into the trained critic network, and obtaining the predicted reward value output by the critic network based on the preset state data and the prediction task offloading instruction; Using a preset loss function, generating a loss value between the preset reward value and the predicted reward value, and determining whether the loss value is less than a preset value; When the loss value is less than the preset value, the training of the actor network and the critic network is stopped, and the trained actor network and the trained critic network are saved.

4. The task offloading method according to claim 1, characterized in that: The obtaining of the current resource utilization rate and the current task data volume, the obtaining of the second driving position and the second driving speed of the current vehicle, and the forming of the current state data with the current resource utilization rate, the current task data volume, the second driving position and the second driving speed include: Obtaining the current resource utilization and the current task data volume; Connecting the current vehicle via the Internet of Vehicles, and receiving the current vehicle position and the current vehicle speed sent by the current vehicle; The current reward value, the current task data volume, the current vehicle position, and the current vehicle speed are combined into the current state data, and the current task data volume is the data volume of the current task to be processed.

5. The task offloading method according to claim 1, characterized in that: The step of inputting the current state data and the current task offloading instruction into the trained critic network, and obtaining a current reward value output by the trained critic network based on the current state data and the current task offloading instruction, comprises: Import a third file, and obtain the trained critic network in the third file; input the current state data and the current task unloading instruction into the trained critic network, and obtain the current reward value output by the trained critic network based on the current state data and the current task unloading instruction.

6. The task offloading method according to claim 1, characterized in that: After generating a current sending coefficient according to the current reward value and a preset sending coefficient generation model, and sending the current task unloading instruction to the current vehicle when the current sending coefficient is greater than the preset sending coefficient, the task unloading method includes: Receiving a computing task sent by the current vehicle according to the current task unloading instruction; The computing task is processed to obtain a processing result, and the processing result is sent to the current vehicle.

7. The task offloading method according to claim 1, characterized in that: After generating a current sending coefficient according to the current reward value and a preset sending coefficient generation model, and sending the current task unloading instruction to the current vehicle when the current sending coefficient is greater than the preset sending coefficient, the task unloading method includes: The vehicle identification of the current vehicle is obtained, and the vehicle identification and the current task unloading instruction are recorded and stored.

8. A task offloading device based on reinforcement learning, characterized in that: Applied to edge servers, the edge servers store actor networks and critic networks, including: A first acquisition module is used to acquire a preset resource utilization rate, a preset task data volume, and a preset first driving position and a first driving speed of a vehicle, and to combine the preset resource utilization rate, the preset task data volume, the first driving position, and the first driving speed into preset state data, wherein the preset task data volume is a preset data volume of a task to be processed; a composition module, used to combine the preset state data, the preset task uninstallation instruction corresponding to the preset state data, and the preset reward value corresponding to the preset state data into a sample, and combine a plurality of the samples into a training set; A determination module, used for determining the trained actor network and the trained critic network based on the training set; A second acquisition module is used to acquire the current resource utilization rate, the current task data volume, the second driving position and the second driving speed of the current vehicle, and to combine the current resource utilization rate, the current task data volume, the second driving position and the second driving speed into current state data, wherein the current task data volume is the data volume of the current task to be processed; a third acquisition module, configured to input the current state data into the trained actor network, and acquire the current task unloading instruction output by the trained actor network based on the current state data; A fourth acquisition module, used for inputting the current state data and the current task unloading instruction into the trained critic network, and acquiring a current reward value output by the trained critic network based on the current state data and the current task unloading instruction; A sending module, used to generate a current sending coefficient according to the current reward value and a preset sending coefficient generation model, and when the current sending coefficient is greater than the preset sending coefficient, send the current task unloading instruction to the current vehicle; The generating a current sending coefficient according to the current reward value and a preset sending coefficient generating model, and when the current sending coefficient is greater than the preset sending coefficient, sending the current task unloading instruction to the current vehicle, comprises: Obtaining a current signal-to-noise ratio and a current signal strength of the current vehicle; Generate a current transmission coefficient according to a ratio of the current reward value to the first reward value, a ratio of the current signal-to-noise ratio to a preset signal-to-noise ratio, a ratio of the current signal strength to a preset signal strength, and a preset transmission coefficient generation model; When the current sending coefficient is greater than the preset sending coefficient, sending the current task unloading instruction to the current vehicle; Wherein, the sending coefficient generation model is a model for generating the current sending coefficient; The transmission coefficient generation model is: T = M1 × T1 + M2 × T2 + M3 × T3; Wherein, T is the current sending coefficient, M1 is the first weighting coefficient, M2 is the second weighting coefficient, M3 is the third weighting coefficient, and the sum of M1, M2 and M3 is 1; The T1 is a ratio of the current reward value to the first reward value, the T2 is a ratio of the current signal-to-noise ratio to a preset signal-to-noise ratio, and the T3 is a ratio of the current signal strength to a preset signal strength.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the task offloading method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the task offloading method according to any one of claims 1 to 7 is implemented.

Images