Model training method, spectrum resource allocation method, device and electronic equipment

By iterative training of the initial evaluation model to generate a strategy evaluation model, the optimal spectrum resources are allocated to the target interactive link in the Internet of Vehicles, which solves the problem that the existing technology is difficult to meet the communication needs of the Internet of Vehicles, and improves the utilization rate of spectrum resources.

CN117956597BActive Publication Date: 2024-12-06GUANGZHOU AUTOMOBILE GROUP CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311872037.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-29
Publication Date
2024-12-06
Estimated Expiration
2043-12-29

AI Technical Summary

Technical Problem

The existing spectrum resource allocation algorithm is difficult to meet the communication needs of high-speed moving vehicle nodes and constantly changing channel conditions in the Internet of Vehicles. Traditional data modeling methods and convolutional neural networks perform poorly in this scenario, and the reinforcement learning algorithm framework is complex. It is also difficult to converge in the environment.

Method used

By obtaining the empirical data set during the test vehicle's driving process, iteratively training the initial evaluation model to generate a strategy evaluation model, which is used to allocate the optimal spectrum resources, including subcarrier spectrum and transmission power, to improve spectrum resources, by increasing spectrum resources, by utilization rate.

Benefits of technology

Dynamic allocation of spectrum resources under high-speed motion and changing channel conditions in the Internet of Vehicles is realized, the utilization rate of spectrum resources is improved, and the communication needs of different applications are met.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117956597B_ABST
    Figure CN117956597B_ABST
Patent Text Reader

Abstract

The present application discloses a model training method, a spectrum resource allocation method, a device and an electronic device. The method obtains an empirical data set, which is generated by using an initial evaluation model in the process of a test vehicle driving through a target section; based on the empirical data set, the initial evaluation model is iteratively trained until the initial evaluation model meets the first preset condition, and the trained initial evaluation model is obtained as a strategy evaluation model. The strategy evaluation model is used to output the reward value corresponding to each spectrum resource included in the resource data set corresponding to the target interaction link based on the state data set corresponding to the target interaction link of the vehicle. The initial evaluation model is iteratively trained using the empirical data set generated by the initial evaluation model in the process of the test vehicle driving through the target section, so that the obtained strategy evaluation model can allocate the optimal spectrum resources for the target interaction link, thereby better meeting the communication needs in the Internet of Vehicles.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of vehicle control technology, and more specifically, to a model training method, spectrum resource allocation method, device and electronic equipment. Background Art

[0002] With the continuous development of the new generation of mobile communication technology, the Internet of Vehicles has attracted worldwide attention in recent years because of its potential value in promoting the development of intelligent transportation and smart cities. The Internet of Vehicles connects vehicles to mobile networks to achieve all-round connection functions such as vehicle to vehicle (V2V) and vehicle to infrastructure (V2I). Spectrum resources are a prerequisite for realizing Internet of Vehicles communication, but with the continuous expansion of the application scope of the Internet of Vehicles and the continuous improvement of the requirements for network communication performance, the existing spectrum resources and allocation algorithms can no longer meet all communication needs in the Internet of Vehicles. Summary of the invention

[0003] In view of the above problems, this application proposes a model training method, spectrum resource allocation method, device and electronic device, which can allocate optimal spectrum resources to the target interactive link through a strategy evaluation model, improve the utilization rate of spectrum resources, and better meet the communication needs in the Internet of Vehicles.

[0004] In a first aspect, an embodiment of the present application provides a model training method, the method comprising: obtaining an empirical data set, the empirical data set being generated by using an initial evaluation model during a test vehicle driving through a target road section, the empirical data set comprising a plurality of empirical data, each of the empirical data comprising current state data and expected state data corresponding to a test interactive link of the vehicle, spectrum resource data allocated to the test interactive link, and an evaluation reward value; based on the empirical data set, iteratively training the initial evaluation model until the initial evaluation model meets a first preset condition, and obtaining the trained initial evaluation model as the strategy evaluation model, the strategy evaluation model being used to output a reward value corresponding to each spectrum resource included in a resource data set corresponding to the target interactive link based on a state data set corresponding to the target interactive link of the vehicle, the state data set comprising a channel gain corresponding to the target interactive link, the resource data set comprising spectrum resources corresponding to at least one frequency band that can be allocated to the target interactive link, the spectrum resources comprising a subcarrier spectrum and a transmit power.

[0005] In a second aspect, an embodiment of the present application provides a spectrum resource allocation method, the method comprising: obtaining a state data set and a resource data set corresponding to a target interactive link of the vehicle, the state data set comprising a channel gain corresponding to the target interactive link, the resource data set comprising spectrum resources corresponding to at least one frequency band that can be allocated to the target interactive link, the spectrum resources comprising subcarrier frequency and transmission power; inputting the state data set into a pre-trained strategy evaluation model, determining a reward value corresponding to each spectrum resource in the resource data set output by the strategy evaluation model, the reward value being used to characterize the probability of the vehicle allocating each spectrum resource to the target interactive link under the operating state corresponding to the state data set, the strategy evaluation model being trained according to the method provided in the first aspect above; determining from the resource data set a target spectrum resource whose reward value meets the target numerical condition as the spectrum resource allocated to the target interactive link.

[0006] In the third aspect, the embodiment of the present application provides a model training device, which includes: a data acquisition module and a model training module. Among them, the data acquisition module is used to obtain an experience data set, which is generated by using an initial evaluation model during the process of the test vehicle driving through the target road section, and the experience data set includes multiple experience data, each of which includes the current state data and expected state data corresponding to the test interactive link of the vehicle, the spectrum resource data allocated for the test interactive link, and the evaluation reward value; the model training module is used to iteratively train the initial evaluation model based on the experience data set until the initial evaluation model meets the first preset condition, and the trained initial evaluation model is obtained as the strategy evaluation model, and the strategy evaluation model is used to output the reward value corresponding to each spectrum resource included in the resource data set corresponding to the target interactive link based on the state data set corresponding to the target interactive link of the vehicle, the state data set includes the channel gain corresponding to the target interactive link, and the resource data set includes the spectrum resources corresponding to at least one frequency band that can be allocated to the target interactive link, and the spectrum resources include subcarrier spectrum and transmission power.

[0007] In a fourth aspect, an embodiment of the present application provides a spectrum resource allocation device, the device comprising: a data acquisition module, a reward determination module and a resource allocation module, wherein the data acquisition module is used to obtain a state data set and a resource data set corresponding to a target interactive link of the vehicle, the state data set comprising a channel gain corresponding to the target interactive link, the resource data set comprising spectrum resources corresponding to at least one frequency band that can be allocated to the target interactive link, the spectrum resources comprising a subcarrier frequency and a transmission power; the reward determination module is used to input the state data set into a pre-trained strategy evaluation model to determine a reward value corresponding to each spectrum resource in the resource data set output by the strategy evaluation model, the reward value being used to characterize the probability of the vehicle allocating each spectrum resource to the target interactive link under the operating state corresponding to the state data set, and the strategy evaluation model is trained according to the method provided in the first aspect above; the resource allocation module is used to determine, from the resource data set, a target spectrum resource whose reward value meets a target numerical condition as the spectrum resource allocated to the target interactive link.

[0008] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the model training method provided in the first aspect above, or to execute the spectrum resource allocation method provided in the second aspect above.

[0009] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the model training method provided in the first aspect above, or to execute the spectrum resource allocation method provided in the second aspect above.

[0010] The solution provided by the present application is to obtain an empirical data set, which is generated by using an initial evaluation model during the process of a test vehicle driving through a target road section; based on the empirical data set, the initial evaluation model is iteratively trained until the initial evaluation model meets the first preset condition, and the trained initial evaluation model is obtained as a strategy evaluation model, which is used to output the reward value corresponding to each spectrum resource included in the resource data set corresponding to the target interaction link based on the state data set corresponding to the target interaction link of the vehicle. The initial evaluation model is iteratively trained using the empirical data set generated by the initial evaluation model during the process of the test vehicle driving through the target road section, so that the obtained strategy evaluation model can allocate the optimal spectrum resources for the target interaction link, improve the utilization rate of spectrum resources, and thus better meet the communication needs of different applications in the Internet of Vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application is shown.

[0013] Figure 2 A flow chart of a model training method provided in one embodiment of the present application is shown.

[0014] Figure 3 A flow chart of a model training method provided in another embodiment of the present application is shown.

[0015] Figure 4 A specific flow chart of step S220 in another embodiment of the present application is shown.

[0016] Figure 5 A specific flow chart of step S230 in another embodiment of the present application is shown.

[0017] Figure 6 Another specific flow chart of step S230 in another embodiment of the present application is shown.

[0018] Figure 7 A schematic flow chart of a spectrum resource allocation method provided in yet another embodiment of the present application is shown.

[0019] Figure 8 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown.

[0020] Fig. 9 A schematic diagram of the structure of a spectrum resource allocation device provided in an embodiment of the present application is shown.

[0021] Fig.10 A structural block diagram of an electronic device provided in an embodiment of the present application is shown.

[0022] Fig.11 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0024] With the continuous development of the new generation of mobile communication technology, the application scope of the Internet of Vehicles is constantly expanding, and the requirements for communication performance in the network are also constantly increasing. Figure 1 As shown in the figure, in the future Internet of Vehicles, entertainment services such as video data and mobile games require high-bandwidth V2I links from the vehicle to the base station. Advanced driving services such as unmanned driving and automatic parking also require periodic transmission of vehicle position, speed, heading and other safety information between the vehicle and other adjacent vehicles through V2V links to improve the vehicle's "cooperative awareness" of the local driving environment. In order to meet the communication needs of the above services in the Internet of Vehicles, the effective use of spectrum resources becomes particularly important.

[0025] Compared with traditional cellular networks, the uncertain network environment of the Internet of Vehicles, such as high-speed moving vehicle nodes and constantly changing channel conditions, has brought unprecedented challenges to the allocation of spectrum resources. In one implementation, the dynamic allocation of spectrum resources can be achieved through traditional data modeling methods, but this method does not show good performance in the dynamic allocation of spectrum resources between high-speed moving vehicle nodes in the Internet of Vehicles. In another implementation, the dynamic allocation of spectrum resources can also be achieved through machine learning methods. Among them, machine learning can achieve better performance in unknown Internet of Vehicles communication environments due to its self-learning ability.

[0026] Specifically, the communication problem can be solved through deep learning methods, that is, the convolutional neural network method can be used to achieve real-time decision-making on spectrum usage and power allocation for connected vehicle users. However, the defect of convolutional neural networks (CNN) is that back propagation requires a large amount of existing data to be iterated, and the network parameters have translational variability and the pooling layer also has information loss problems. In addition, the classic Q_Learning algorithm framework in reinforcement learning can also be applied to the connected vehicle communication scenario to meet the diverse needs of different dynamic entities, but this algorithm framework can only be applied to simple discrete environments. In more complex environments, the algorithm performs poorly or even fails to converge.

[0027] Therefore, the present application provides a model training method, a spectrum resource allocation method, a device and an electronic device, which iteratively trains the initial evaluation model using an empirical data set generated by the initial evaluation model during the test vehicle driving through the target road section, so that the obtained strategy evaluation model can allocate the optimal spectrum resources for the target interactive link, improve the utilization rate of spectrum resources, and thus better meet the communication needs of different applications in the Internet of Vehicles. Among them, the specific model training method and spectrum resource allocation method are described in detail in the subsequent embodiments.

[0028] See also Figure 2, Figure 2 A flow chart of a model training method provided by an embodiment of the present application is shown below. Figure 2 The process shown is described in detail, and the model training method can specifically include the following steps:

[0029] Step S110: Obtain an empirical data set, which is generated by using an initial evaluation model when the test vehicle is driving through a target road section. The empirical data set includes multiple empirical data, each of which includes current state data and expected state data corresponding to the test interaction link of the test vehicle, spectrum resource data allocated to the test interaction link, and an evaluation return value.

[0030] In an embodiment of the present application, the electronic device can obtain the current state data corresponding to the test interaction link of the test vehicle in real time during the process of controlling the test vehicle to travel through the target road section, and determine the spectrum resource data allocated to the test interaction link through the initial evaluation model, and evaluate the expected state data corresponding to the test interaction link after the spectrum resource data is allocated. Among them, the test vehicle can actually select the spectrum resource data allocated to the test interaction link from at least one spectrum resource data when it is in the operating state corresponding to the current state data. Therefore, the initial evaluation model can also determine the probability of allocating each spectrum resource data to the test interaction link, that is, the evaluation return value corresponding to each spectrum resource data. Obviously, after selecting different spectrum resource data, the expected state data corresponding to the test interaction link must be different. Therefore, the test vehicle can combine the current state data, spectrum resource data, evaluation return value, and expected state data to obtain an empirical data.

[0031] Among them, since the test interactive link corresponds to the same current state data, the test vehicle can actually allocate different spectrum resource data to it, thereby obtaining different evaluation return values ​​and different expected state data. In other words, the test vehicle can travel through the target section multiple times, and each time it passes through the target section, it can try to allocate different spectrum resource data to the test interactive link, thereby obtaining multiple different experience data, that is, try to make the experience data in the experience data set fully cover all possible situations of the test interactive link on the target section. Therefore, by subsequently iteratively training the initial evaluation model based on a more complete experience data set, a more accurate strategy evaluation model can be obtained.

[0032] In some embodiments, the test interactive link of the test vehicle can be a V2I link between the test vehicle and the infrastructure, or a V2V link between the test vehicle and other vehicles. Specifically, the test vehicle can first allocate fixed spectrum resource data to the V2I link of the test vehicle based on actual conditions, and then allocate the remaining available spectrum resource data to all V2V links included in the test vehicle. Of course, the test vehicle can also first allocate fixed spectrum resource data to one of the important V2V links, and then allocate the remaining available spectrum resource data to the V2I link and other V2V links. The specific type of the test interactive link is not specifically limited here.

[0033] Step S120: Based on the empirical data set, the initial evaluation model is iteratively trained until the initial evaluation model meets the first preset condition, and the trained initial evaluation model is obtained as the strategy evaluation model, and the strategy evaluation model is used to output the reward value corresponding to each spectrum resource included in the resource data set corresponding to the target interaction link based on the state data set corresponding to the target interaction link of the vehicle, the state data set includes the channel gain corresponding to the target interaction link, and the resource data set includes spectrum resources corresponding to at least one frequency band that can be allocated to the target interaction link, and the spectrum resources include subcarrier spectrum and transmission power.

[0034] In an embodiment of the present application, after the test vehicle travels through the target road section and obtains the corresponding experience data set based on the initial evaluation model, the electronic device can iteratively train the initial evaluation model based on the multiple experience data included in the experience data set to optimize the policy parameters in the initial evaluation model, and obtain a policy evaluation model that can accurately output the reward value corresponding to each spectrum resource. The policy evaluation model can be applied to the driving process of the vehicle, by obtaining the state data set corresponding to the target interactive link of the vehicle, and determining the reward value corresponding to each spectrum resource in the resource data set corresponding to the target interactive link. In this way, the electronic device can allocate the spectrum resource with the largest reward value as the optimal spectrum resource to the target interactive link, so as to realize the dynamic allocation of spectrum resources and improve the utilization rate of spectrum resources in the environment of high-speed moving vehicle nodes and constantly changing channel conditions in the Internet of Vehicles.

[0035] Among them, whether it is the initial evaluation model before training is completed, or the strategy evaluation model obtained after training is completed, it is possible to determine the reward value corresponding to each spectrum resource in the resource data set corresponding to the interactive link based on the current state data corresponding to the input interactive link. However, the reward value of each spectrum resource determined based on the initial evaluation model that has not completed training may not be accurate and cannot take global factors into account. Therefore, the electronic device can generate an empirical data set based on this initial evaluation model that has not completed training, while the test vehicle is driving through the target section, and then adjust the strategy parameters in the initial evaluation model based on the empirical data set, so that the adjusted initial evaluation model can execute a better strategy during the test vehicle driving through the target section, and determine a more accurate and more global reward value for each spectrum resource.

[0036] It can be understood that if the initial evaluation model after iterative training meets the first preset condition, the initial evaluation model can be used as a strategy evaluation model in the actual driving process of the vehicle. At this time, it is considered that the strategy evaluation model can allocate a globally optimal spectrum resource for the target interactive link based on the input current state data. Specifically, the strategy evaluation model can determine the reward value corresponding to each spectrum resource based on the input current state data, and the electronic device can use the spectrum resource with the largest reward value as the spectrum resource finally allocated to the target interactive link.

[0037] However, it is worth noting that the initial evaluation model, which has not yet completed training, will also determine the reward value corresponding to each spectrum resource based on the input current state data. However, since the initial evaluation model is not applied to the actual driving of the vehicle, the test vehicle collects more experience data through the initial evaluation model while driving through the target section, so as to facilitate the subsequent training of the initial evaluation model based on the experience data. Therefore, when the test vehicle is driving through the target section, even if the initial evaluation model can determine the reward value corresponding to each spectrum resource based on the current state data, the test vehicle will not necessarily select the spectrum resource with the largest reward value as the spectrum resource allocated for the test interactive link. Instead, it will try to allocate different spectrum resources each time it drives through the target section, so as to try to traverse all spectrum resource allocation situations that may occur during the driving of the test vehicle.

[0038] That is to say, even if the initial evaluation model determines that the return value corresponding to a certain spectrum resource is very low, the test vehicle may still allocate the spectrum resource for the test interactive link. This is because the test vehicle drives through the target section in order to obtain a more complete empirical data set, not to allocate the optimal spectrum resource for the test interactive link during the driving process. Therefore, when the test vehicle drives through the target section, the spectrum resources allocated to the test interactive link should cover as many possibilities as possible, rather than allocating the optimal spectrum resource for the test interactive link.

[0039] Moreover, since the initial evaluation model has not completed iterative training, the strategy parameters therein are not perfect. Even if the initial evaluation model currently determines that the return value corresponding to a certain spectrum resource is small, if the test vehicle still allocates the spectrum resource for the test interactive link, as the test vehicle continues to drive, the vehicle as a whole may still obtain a better spectrum resource utilization efficiency, that is, it may obtain a globally optimal allocation strategy. Therefore, the test vehicle needs to try to traverse different spectrum resource allocation methods while driving through the target section. In order to ensure that the empirical data set includes more possible allocation situations, the test vehicle can repeatedly drive through the target section many times, and each time it drives through the target section, it tries to choose different spectrum resource allocation methods. In this way, the electronic device can obtain a more comprehensive empirical data set, and then determine a globally optimal spectrum resource allocation strategy based on the empirical data set, and iteratively train the initial evaluation model.

[0040] In some implementations, the spectrum resources actually allocated by the electronic device for the interactive link may include subcarrier spectrum and transmit power. The test interactive link may also correspond to a resource data set, which includes spectrum resources corresponding to at least one frequency band that can be allocated for the test interactive link.

[0041] The model training method provided in the embodiment of the present application obtains an empirical data set, which is generated by using an initial evaluation model during the process of a test vehicle driving through a target section; based on the empirical data set, the initial evaluation model is iteratively trained until the initial evaluation model meets the first preset condition, and the trained initial evaluation model is obtained as a strategy evaluation model, which is used to output the reward value corresponding to each spectrum resource included in the resource data set corresponding to the target interaction link based on the state data set corresponding to the target interaction link of the vehicle. The initial evaluation model is iteratively trained using the empirical data set generated by the initial evaluation model during the process of the test vehicle driving through the target section, so that the obtained strategy evaluation model can allocate the optimal spectrum resources for the target interaction link, improve the utilization rate of spectrum resources, and thus better meet the communication needs of different applications in the Internet of Vehicles.

[0042] See also Figure 3 , Figure 3 A flow chart of a model training method provided by another embodiment of the present application is shown below. Figure 3 The process shown is described in detail, and the model training method may specifically include the following steps:

[0043] Step S210: Acquire an experience data set.

[0044] In the embodiment of the present application, step S210 can refer to the explanation of other embodiments and will not be elaborated in detail here.

[0045] Step S220: Obtain a cumulative reward value corresponding to the initial evaluation model, wherein the cumulative reward value is determined based on the total channel capacity corresponding to the test interaction link, the remaining load, and the policy entropy corresponding to the initial evaluation model, and the size of the policy entropy is negatively correlated with the number of iterative training times of the initial evaluation model.

[0046] In the embodiment of the present application, the test vehicle can not only obtain multiple pieces of experience data each time it passes through the target road section, but also can pass through the target road section multiple times, and by allocating different spectrum resources to the test interactive link during each driving process, the electronic device can obtain an experience data set that covers most of the spectrum resource allocation possibilities. It can be understood that the experience data set obtained by the electronic device at this time is generated based on the latest trained initial evaluation model, that is, each piece of experience data in the experience data set includes the evaluation reward value determined by the latest trained initial evaluation model for the spectrum resource. It can also be said that the experience data set obtained by the electronic device at this time is generated according to the optimal spectrum resource allocation strategy obtained by the current training. Therefore, after obtaining a new and complete experience data set each time, the electronic device can first determine whether the current initial evaluation model has met the first preset condition based on the experience data set, that is, first obtain the cumulative reward value corresponding to the initial evaluation model based on the experience data set to determine whether the training operation of the initial evaluation model can be ended. Obviously, if the cumulative reward value meets the first preset condition, the electronic device can no longer iterate the initial evaluation model, but directly use the initial evaluation model as a strategy evaluation model. At this time, the spectrum resource allocation strategy corresponding to the strategy evaluation model is the optimal spectrum resource allocation strategy finally determined.

[0047] Specifically, the cumulative reward value corresponding to the initial evaluation model can reflect the pros and cons of the spectrum resource allocation strategy corresponding to the initial evaluation model to a certain extent. Generally, the more iterative training times of the initial evaluation model, the more perfect the spectrum resource allocation strategy corresponding to the initial evaluation model. Therefore, the electronic device can determine the size of the policy entropy corresponding to the model based on the number of iterations of the initial evaluation model, and then determine the cumulative reward value corresponding to the initial evaluation model based on the policy entropy. Obviously, the more iterations of the initial evaluation model, the more perfect the corresponding spectrum resource allocation strategy, and the less necessary it is to traverse all possibilities in the subsequent iterative training process. Accordingly, the policy entropy corresponding to the initial evaluation model can be smaller. Therefore, the size of the policy entropy can gradually return to zero as the number of model iterations increases. In addition, since the initial evaluation model is used to determine the corresponding spectrum resources for the interactive link of the vehicle, and the parameters such as the total channel capacity and the remaining load corresponding to the interactive link can also reflect the pros and cons of the spectrum resources allocated by the interactive link to a certain extent, the electronic device can also determine the cumulative reward value corresponding to the initial evaluation model based on the parameters such as the total channel capacity and the remaining load corresponding to the test interactive link.

[0048] More specifically, the remaining load corresponding to the interactive link can actually be divided into two cases, namely, the remaining load is equal to zero and greater than zero. Figure 4 As shown, the electronic device may determine the cumulative reward value corresponding to the initial evaluation model in different ways according to the different remaining loads corresponding to the test interaction link. Specifically, the following steps may be included:

[0049] Step S221: Obtain the remaining load corresponding to the test interaction link.

[0050] In an embodiment of the present application, the test interaction link can be a link for information interaction between the vehicle and the basic equipment, or a link for information interaction between the vehicle and other vehicles. In either case, the test interaction link needs to transmit information through data packets. The remaining load corresponding to the test interaction link can reflect the transmission of data packets on the test interaction link to a certain extent. Obviously, if the remaining load is zero, that is, the test interaction link can successfully send out all data packets on the link when the corresponding spectrum resources are determined according to the spectrum resource allocation strategy corresponding to the initial evaluation model. In this case, the electronic device can set the cumulative reward value corresponding to the initial evaluation model to a higher fixed value. On the contrary, if the remaining load is not zero, that is, when the test interaction link is allocated with the corresponding spectrum resources, there are still some data packets that have not been sent out on the link. At this time, the electronic device can determine the cumulative reward value corresponding to the initial evaluation model according to the load success delivery rate corresponding to the test link.

[0051] Step S222: If the remaining load is greater than zero, the cumulative reward value is determined based on the load success delivery rate corresponding to the test interaction link, the total channel capacity, and the policy entropy corresponding to the initial evaluation model.

[0052] In an embodiment of the present application, if the remaining load corresponding to the test interactive link is greater than zero, it indicates that the test interactive link cannot completely send out all the data packets that need to be transmitted on the link under the running state of the spectrum resources currently allocated to it. At this time, the electronic device can determine the cumulative reward value corresponding to the initial evaluation model based on the load that has been transmitted on the test interactive link. That is, the cumulative reward value is determined based on the load success delivery rate corresponding to the test interactive link. Obviously, when the total channel capacity and policy entropy remain unchanged, the higher the load success delivery rate, the higher the cumulative reward value. In addition, the purpose of the policy evaluation model is to allocate an optimal spectrum resource to the target interactive link to improve the total channel capacity and load delivery rate corresponding to the target interactive link. Therefore, the electronic device can determine the cumulative reward value corresponding to the initial evaluation model based on the total channel capacity and the load success delivery rate. In addition, the greater the policy entropy, the more helpful it is for the initial evaluation model to allocate more different spectrum resources to the test interactive link, find more possibilities, and avoid the spectrum resource allocation strategy from falling into a local optimal solution. Therefore, in the early stage of iterative training of the initial evaluation model, the policy entropy should be large, and as the number of iterative trainings of the initial evaluation model gradually increases, the policy entropy can be gradually reduced.

[0053] Step S223: If the remaining load is zero, the accumulated reward value is determined based on the maximum effective transmission rate corresponding to the test interactive link, the total channel capacity, and the policy entropy corresponding to the initial evaluation model.

[0054] In an embodiment of the present application, if the remaining load corresponding to the test interactive link is zero, that is, all data packets that need to be transmitted on the test interactive link can be successfully sent. In this case, the electronic device can use the maximum effective transmission efficiency corresponding to the test interactive link as a component of the cumulative reward value corresponding to the initial evaluation model. In other words, the cumulative reward value corresponding to the initial evaluation model can be determined based on the maximum effective transmission rate corresponding to the test interactive link, the total channel capacity, and the policy entropy corresponding to the initial evaluation model. It can be understood that when the remaining load is zero, the electronic device can also train the initial evaluation model to further improve the transmission rate of the test interactive link and improve the load delivery rate corresponding to the test interactive link. Therefore, the electronic device can determine the cumulative reward value of the initial evaluation model based on the maximum effective transmission speed corresponding to the test interactive link. In other words, the electronic device can obtain the feedback reward value based on the transmission action of the test interactive link, and then update the spectrum resource allocation strategy corresponding to the initial evaluation model based on the cumulative reward value, thereby effectively solving the problem of sparse rewards during training and no strategy learning without rewards in the early stage of training.

[0055] In some implementations, the electronic device may set a corresponding weight value for the load success delivery rate / maximum effective transmission rate, the total channel capacity, and the policy entropy, respectively, and perform weighted summation of the load success delivery rate, the total channel capacity, and the policy entropy with the corresponding weight values, and the obtained value is the cumulative reward value corresponding to the initial evaluation model. Specifically, it can be calculated according to the following formula:

[0056]

[0057]

[0058] Among them, R t+1 This is the cumulative reward value corresponding to the initial evaluation model, which consists of three parts. The first part is the total channel capacity, the second part ∑r k (t) is the successful delivery rate of the load or the maximum effective transmission rate, which is determined by whether the remaining load corresponding to the test interactive link is zero. The third part log2(π(S t )) is the policy entropy, which is positively correlated with the number of iterative training of the initial evaluation model. c , d , and α are weights pre-set for each parameter. The electronic device can perform weighted summation on the above parameters according to each weight to obtain a cumulative reward value.

[0059] In some embodiments, in order to more effectively stimulate the test interaction link to transmit information, the electronic device can set a larger reward value for it when the remaining load is zero. For example, twice the maximum effective transmission rate corresponding to the test interaction link can be directly used as the reward value, that is, φ=2V, where V is the maximum effective transmission rate corresponding to the test interaction link.

[0060] Step S230: input each piece of the experience data into the initial value model to obtain the target reward value corresponding to each piece of the experience data output by the initial value model.

[0061] In an embodiment of the present application, if the electronic device determines that the current initial evaluation model does not meet the first preset condition based on the cumulative reward value corresponding to the initial evaluation model, the initial evaluation model may be iteratively trained again based on multiple pieces of experience data in the experience data set. Specifically, the electronic device may assist in iterative training of the initial evaluation model through the initial value model. Each piece of experience data in the experience data set may be input into the initial value model in turn to obtain a target reward value corresponding to each piece of experience data output by the initial value model. The initial value model is used to evaluate each piece of experience data input to determine the value of the spectrum resources corresponding to the test interactive link allocated according to the record of the experience data.

[0062] In some embodiments, in order to improve the stability of the algorithm and the target reward value, the initial value model may include a first value model and a second value model. Figure 5 The electronic device can also determine the target reward value corresponding to each piece of experience data through the following steps:

[0063] Step S231: input each piece of the experience data into the first value model and the second value model to obtain a first reward value output by the first value model and a second reward value output by the second value model.

[0064] In an embodiment of the present application, each piece of experience data in the experience data set can be input into the first value model and the second value model at the same time. Among them, the initial first value model and the second value model can be two Soft Q neural network models with the same initial parameters. In the application process, one of the models can be used as the first value model to evaluate the current value corresponding to the input experience data and output the first reward value. The parameters of the model can be continuously updated during each training process; another model can also be used as the second value model to evaluate the target value corresponding to the input experience data and output the second reward value. The parameters of the model will only be updated after a few training intervals.

[0065] Step S232: determining a target reward value corresponding to each piece of the experience data based on the smallest reward value between the first reward value and the second reward value.

[0066] In the embodiment of the present application, in order to determine the accuracy of the target reward value corresponding to each piece of experience data finally outputted, a smaller reward value can be selected between the first reward value and the second reward value as the target reward value. Among them, the target reward value selects a smaller value, which can make the evaluation loss value corresponding to the initial evaluation model relatively more accurate, and the strategy parameters of the initial evaluation model can be updated based on the more accurate evaluation loss value in the subsequent iterative training process.

[0067] Specifically, the first reward value, the second reward value, and the target reward value corresponding to each piece of experience data can be calculated by referring to the following formula:

[0068] Q(S t )=r(S t ,a t )+γE[V(S t+1 )]

[0069] V(S t )=π(S t ) T [min(Q1(S t ),Q2(S t ))-αlog2(π(S t ))]

[0070] Among them, π(S t ) is the evaluation reward value output by the initial evaluation model. The first reward value and the second reward value can be calculated according to the previous formula. min(Q1(S t ),Q2(S t )) is to select the smallest reward value between the first reward value and the second reward value,

[0071] In other embodiments, Figure 6 As shown, step S230 may further include the following steps:

[0072] Step S233: Determine the value loss value corresponding to the initial value model based on the evaluation return value included in each piece of the experience data.

[0073] In an embodiment of the present application, when the initial evaluation model generates each piece of experience data, it determines a corresponding evaluation reward value based on the spectrum resources allocated for the test interactive link and records it in each piece of experience data. Before inputting each piece of experience data into the initial value model, the electronic device may also first iteratively train the initial value model based on multiple pieces of experience data, so that the trained initial value model can output a more accurate target reward value. Thereafter, the target reward value corresponding to each piece of experience data is determined based on the trained initial value model. Specifically, the electronic device can determine the value loss value corresponding to the initial value model based on the evaluation reward value output by the initial evaluation model included in each piece of experience data. Specifically, the value loss value can be calculated by referring to the following formula:

[0074]

[0075] Among them, π(S t ) is the evaluation return value output by the initial evaluation model, and H is the dimension of the action space.

[0076] It can be understood that the electronic device trains the initial evaluation model multiple times. Each time after the test vehicle drives through the target section and obtains a relatively complete experience data set, the initial evaluation model can be trained based on the experience data set to obtain an initial evaluation model with a more optimized spectrum resource allocation strategy. If the initial evaluation model has not yet met the first preset condition at this time, the electronic device will control the test vehicle to drive through the target section again based on the newly trained initial evaluation model to obtain a new experience data set, and iteratively train the initial evaluation model again based on the new experience data set. Thereafter, if the trained initial evaluation model still does not meet the first preset condition, the process of obtaining a new experience data set and iterative training will be repeated until the initial evaluation model meets the first preset condition, and then the initial evaluation model can be used as a strategy evaluation model. Since the process of iterative training of the initial evaluation model is based on the evaluation loss value corresponding to the initial evaluation model, in order to make the evaluation loss value more accurate, the electronic device can, after obtaining the latest experience data set corresponding to the initial evaluation model, first iteratively train the initial value model based on the new experience data set, so that the target reward value output by the initial value model is more accurate, and then based on the more accurate target reward value, the evaluation loss value corresponding to the obtained initial evaluation model is also more accurate. Specifically, the electronic device can first determine the value loss value corresponding to the initial value model based on the evaluation reward value determined by the initial evaluation model and included in each piece of experience data in the latest experience data set.

[0077] Step S234: iteratively train the initial value model according to the value loss value until the initial value model meets the second preset condition, thereby obtaining the trained initial value model.

[0078] In the embodiment of the present application, after determining the value loss value corresponding to the initial value model according to the empirical data, the electronic device can adjust the parameters in the initial value model based on the value loss value to make the value loss value smaller. The trained initial value model is obtained until the initial value model meets the second preset condition, that is, the value loss value corresponding to the initial value model is less than the preset data, or the value loss value infinitely approaches a certain value.

[0079] Step S235: input each piece of the experience data into the trained initial value model in sequence, and obtain the target reward value corresponding to each piece of the experience data output by the initial value model.

[0080] In the embodiment of the present application, after the electronic device completes the iterative training of the initial value model based on the experience data set, the target reward value output by the trained initial value model will be more accurate. Therefore, after the training of the initial value model is completed, each piece of experience data in the experience data set can be input into the initial value model so that it can output a more accurate target reward value.

[0081] It is understandable that in each iterative training of the initial evaluation model, a new experience data set will appear, and each time before the initial evaluation model is trained based on the new experience data set, the initial value model will be trained based on this new experience data set. That is to say, in each training process before the completion of the iterative training, the initial value model obtained by training is actually not able to output the target reward value very accurately. For example, in the first training process, since the parameters in the initial value model are all initial parameters, the target reward value outputted by it can be said to be chaotic and random, and in the second training process, that is, after the initial value model has been trained once according to the experience data set, the target reward value outputted by the model will be more accurate, but obviously compared to the target reward value outputted by the initial value model after many trainings, it must still be inaccurate. In other words, the accuracy of the target reward value outputted by the initial value model is also gradually accurate as the number of iterative trainings gradually increases.

[0082] Step S240: Determine the evaluation loss value corresponding to the initial evaluation model based on the target reward value corresponding to each piece of the experience data and the evaluation return value included in each piece of the experience data.

[0083] In an embodiment of the present application, after the initial value model outputs the corresponding target reward value according to each piece of experience data, the electronic device can determine the evaluation loss value corresponding to the initial evaluation model in combination with the target reward value and the evaluation return value. Among them, the target reward value is determined by the initial value model for each piece of experience data, and is a reward value fed back by the environment for the spectrum resource allocation strategy corresponding to the initial evaluation model; and the evaluation return value is the probability of allocating different spectrum resources to the test interaction link determined by the initial evaluation model based on the current state data of the test interaction link during the driving of the test vehicle. Combining the target reward value and the evaluation return value, the electronic device can determine the evaluation loss value corresponding to the initial evaluation model according to the following formula:

[0084]

[0085] In some embodiments, the initial evaluation model may be a Police neural network, wherein the policy parameters of the Police neural network may be random parameters before the first iterative training, and during the continuous iterative training, the parameters in the Police neural network continue to converge until they finally meet the first preset condition, and the obtained model may be used as a policy evaluation model applied in the actual vehicle operation process.

[0086] Step S250: iteratively updating the initial evaluation model according to the evaluation loss value until the initial evaluation model meets the first preset condition, and obtaining the trained initial evaluation model as the strategy evaluation model.

[0087] In an embodiment of the present application, after obtaining the evaluation loss value corresponding to the initial evaluation model, the electronic device can adjust and update the policy parameters in the initial evaluation model based on the evaluation loss value, so that the spectrum resource allocation strategy corresponding to the initial evaluation model after the updated parameters is more optimized. The iterative training operation of the initial evaluation model can only be ended when the initial evaluation model meets the first preset condition.

[0088] Among them, the first preset condition may include: the difference between the cumulative reward value corresponding to the initial evaluation model after this iterative training and the cumulative reward value corresponding to the initial evaluation model obtained by the previous iterative training is less than a preset threshold. The initial evaluation model meets the first preset condition, which means that the adjustment of the policy parameters in the initial evaluation model in this iterative training has been less, that is, the difference between the policy parameters of the initial evaluation model before and after training is small, that is, the various parameters in the initial evaluation model have been close to stability. In this case, it can be considered that the training of the initial evaluation model has been completed, and the initial evaluation model obtained by training can be used as a policy evaluation model in the actual vehicle operation process.

[0089] The model training method provided in the embodiment of the present application obtains the cumulative reward value corresponding to the experience data set and the initial evaluation model; inputs each piece of experience data into the initial value model to obtain the target reward value corresponding to each piece of experience data output by the initial value model; determines the evaluation loss value corresponding to the initial evaluation model based on the target reward value corresponding to each piece of experience data and the evaluation return value included in each piece of experience data; iteratively updates the initial evaluation model according to the evaluation loss value until the initial evaluation model meets the first preset condition, and obtains the trained initial evaluation model as the strategy evaluation model. By determining the target reward value corresponding to each piece of experience data through the initial value model, and then determining the evaluation loss value of the initial evaluation model based on the target reward value and the evaluation return value included in each piece of experience data, and then iteratively training the initial evaluation model, the accuracy of the evaluation loss value can be improved, and then the strategy evaluation model corresponding to the global optimal spectrum resource allocation strategy can be obtained, thereby improving the utilization rate of spectrum resources and better meeting the communication needs of different applications in the Internet of Vehicles.

[0090] See also Figure 7 , Figure 7 A schematic diagram of a spectrum resource allocation method according to another embodiment of the present application is shown below. Figure 7 The process shown is described in detail, and the spectrum resource allocation method may specifically include the following steps:

[0091] Step S310: Obtain a status data set and a resource data set corresponding to the target interactive link of the vehicle, wherein the status data set includes a channel gain corresponding to the target interactive link, and the resource data set includes spectrum resources corresponding to at least one frequency band that can be allocated to the target interactive link, and the spectrum resources include subcarrier frequency and transmission power.

[0092] In the embodiment of the present application, during the actual operation of the vehicle in the current environment, the state data set and resource data set corresponding to the target interactive link can be obtained in real time. Assuming that the target interactive link is a V2V link k between the vehicle and other vehicles corresponding to the vehicle, and the vehicle also includes a V2I link between the vehicle and the infrastructure, the elements included in the state data set corresponding to the target interactive link can be shown as the following state function:

[0093] Z k [i]={G k [i],G k '[i],G kb [i],G ki [i]}

[0094] Among them, the first element G in the state function corresponding to the target interactive link k[i] can be used to characterize the channel gain corresponding to the target interactive link; the second element G k '[i] can represent the interference channel gain of the vehicle transmitter of other interactive links corresponding to the vehicle to the vehicle receiver of the target interactive link; the third element G kb [i] can characterize the interference channel gain of the vehicle transmitter of the target interactive link to the infrastructure (base station); the fourth element G ki [i] It can represent the interference channel gain of the vehicle transmitter of the V2I link corresponding to the vehicle to the vehicle receiver of the target interactive link.

[0095] Specifically, the state data set corresponding to the target interactive link obtained by this vehicle can characterize the impact of the interference channel gain of other interactive links on the target interactive link in the current environment at the current moment. This vehicle can also synchronously obtain the resource data set corresponding to the target interactive link, that is, the spectrum resource data that can be allocated to the target interactive link when it is affected by other interactive links and various factors such as the operating status of its own vehicle, that is, the data such as the subcarrier spectrum and transmission power that can be allocated. It can be understood that this vehicle first needs to obtain the state data set and resource data set corresponding to the target interactive link in order to subsequently determine the current operating status of the target interactive link and the spectrum resources that should be allocated to the target interactive link.

[0096] In some embodiments, the value of the transmission power in the spectrum resources that can be allocated to the target interactive link corresponding to the vehicle can be pre-set. For example, it can be pre-set that the transmission power of the target interactive link can only be selected from (23, 10, 5, -100), and then the subsequent strategy evaluation model will only select the optimal transmission power among these four values.

[0097] Step S320: Input the status data set into a pre-trained strategy evaluation model, and determine the reward value corresponding to each spectrum resource in the resource data set output by the strategy evaluation model, wherein the reward value is used to characterize the probability of the vehicle allocating each spectrum resource to the target interactive link under the operating state corresponding to the status data set. The strategy evaluation model is trained according to the model training method provided in the aforementioned embodiment.

[0098] In an embodiment of the present application, after obtaining the state data set and resource data set corresponding to the target interactive link, the vehicle can input the data in the state data set into a pre-trained strategy evaluation model. The strategy evaluation model can output the reward value corresponding to each spectrum resource in the resource data set corresponding to the target interactive link, and then determine the spectrum resources finally allocated to the target interactive link according to different reward values, that is, finally determine the subcarrier spectrum and transmission power corresponding to the target interactive link. Among them, the strategy evaluation model is used to output the probability of allocating different spectrum resources to the interactive link according to the state data corresponding to the input interactive link. The strategy evaluation model can be obtained through multiple iterations of training by the model training method provided in the aforementioned embodiment.

[0099] Step S330: Determine, from the resource data set, a target spectrum resource whose reward value satisfies a target numerical condition as the spectrum resource allocated to the target interactive link.

[0100] In an embodiment of the present application, after the vehicle obtains the reward value corresponding to each spectrum resource through a pre-trained strategy evaluation model, it can determine the spectrum resource finally allocated to the target interactive link based on the different reward values ​​corresponding to each spectrum resource. Among them, the reward value corresponding to each spectrum resource is the probability of allocating each spectrum resource to the target interactive link output by the strategy evaluation model when the current operating state of the vehicle is determined according to the state data set. Obviously, the larger the reward value corresponding to the spectrum resource, the higher the utilization rate of the spectrum resource of the Internet of Vehicles after the target interactive link is allocated to this spectrum resource. Therefore, the vehicle can directly select the spectrum resource with the largest reward value in the resource data set as the spectrum resource allocated to the target interactive link, so as to maximize the utilization of the spectrum resources on the Internet of Vehicles.

[0101] The spectrum resource allocation method provided in the embodiment of the present application obtains the state data set and resource data set corresponding to the target interactive link of the vehicle; inputs the state data set into a pre-trained strategy evaluation model to determine the reward value corresponding to each spectrum resource in the resource data set output by the strategy evaluation model, and the strategy evaluation model is trained according to the model training method provided in the above embodiment; from the resource data set, the target spectrum resource whose reward value meets the target numerical condition is determined as the spectrum resource allocated to the target interactive link. Thus, through the pre-trained strategy evaluation model, according to the state data set and resource data set corresponding to the target interactive link, the reward value corresponding to each spectrum resource in the resource data set is accurately output, and then the spectrum resource with the largest reward value can be selected as the spectrum resource allocated to the target interactive link, so that the spectrum resources of the Internet of Vehicles are effectively utilized, and the optimal spectrum resources are allocated to the target interactive link, which improves the utilization rate of the spectrum resources and can better meet the communication needs of different applications in the Internet of Vehicles.

[0102] See also Figure 8 , which shows a structural block diagram of a model training device 200 provided in an embodiment of the present application, and the model training device 200 includes: a data acquisition module 210, and a model training module 220. Among them, the data acquisition module 210 is used to obtain an experience data set, which is generated by using an initial evaluation model during the process of the test vehicle driving through the target road section, and the experience data set includes multiple experience data, each of which includes the current state data and expected state data corresponding to the test interactive link of the vehicle, the spectrum resource data allocated for the test interactive link, and the evaluation reward value; the model training module 220 is used to iteratively train the initial evaluation model based on the experience data set until the initial evaluation model meets the first preset condition, and obtain the trained initial evaluation model as the strategy evaluation model, the strategy evaluation model is used to output the reward value corresponding to each spectrum resource included in the resource data set corresponding to the target interactive link based on the state data set corresponding to the target interactive link of the vehicle, the state data set includes the channel gain corresponding to the target interactive link, and the resource data set includes the spectrum resources corresponding to at least one frequency band that can be allocated to the target interactive link, and the spectrum resources include subcarrier spectrum and transmission power.

[0103] As a possible implementation, the model training module 220 includes a data input unit, a reward acquisition unit, and an iterative training unit. The data input unit is used to input each piece of experience data into the initial value model to obtain the target reward value corresponding to each piece of experience data output by the initial value model; the reward acquisition unit is used to determine the evaluation loss value corresponding to the initial evaluation model based on the target reward value corresponding to each piece of experience data and the evaluation return value included in each piece of experience data; the iterative training unit is used to iteratively update the initial evaluation model according to the evaluation loss value until the initial evaluation model meets the first preset condition, and obtain the trained initial evaluation model as the strategy evaluation model.

[0104] As a possible implementation method, the reward acquisition unit is also used to determine the value loss value corresponding to the initial value model based on the evaluation return value included in each piece of experience data; iteratively train the initial value model according to the value loss value until the initial value model meets the second preset condition, and obtain the trained initial value model; input each piece of experience data into the trained initial value model in turn, and obtain the target reward value corresponding to each piece of experience data output by the initial value model.

[0105] As a possible implementation, the initial value model includes a first value model and a second value model. The reward acquisition unit is further used to input each piece of experience data into the first value model and the second value model to obtain a first reward value output by the first value model and a second reward value output by the second value model; and determine the smallest reward value between the first reward value and the second reward value as the target reward value corresponding to each piece of experience data.

[0106] As a possible implementation, the first preset condition includes: the difference between the cumulative reward value corresponding to the initial evaluation model after this iterative training and the cumulative reward value corresponding to the initial evaluation model obtained by the previous iterative training is less than a preset threshold; the model training device 200 also includes a reward determination module, which is used to obtain the cumulative reward value corresponding to the initial evaluation model, and the cumulative reward value is determined based on the total channel capacity corresponding to the target interactive link, the remaining load, and the policy entropy corresponding to the initial evaluation model, and the size of the policy entropy is negatively correlated with the number of iterative training times of the initial evaluation model.

[0107] As a possible implementation method, the reward determination module is also used to obtain the remaining load corresponding to the target interactive link; if the remaining load is greater than zero, the cumulative reward value is determined based on the load success delivery rate corresponding to the target interactive link, the total channel capacity, and the policy entropy corresponding to the initial evaluation model; if the remaining load is zero, the cumulative reward value is determined based on the maximum effective transmission rate corresponding to the target interactive link, the total channel capacity, and the policy entropy corresponding to the original evaluation model.

[0108] See also Fig. 9 , which shows a structural block diagram of a spectrum resource allocation device 300 provided in an embodiment of the present application, and the spectrum resource allocation device 300 includes: a data acquisition module 310, a reward determination module 320 and a resource allocation module 330. Among them, the data acquisition module 310 is used to obtain a state data set and a resource data set corresponding to the target interactive link of the vehicle, the state data set includes a channel gain corresponding to the target interactive link, and the resource data set includes spectrum resources corresponding to at least one frequency band that can be allocated to the target interactive link, and the spectrum resources include subcarrier frequency and transmission power; the reward determination module 320 is used to input the state data set into a pre-trained strategy evaluation model, and determine the reward value corresponding to each spectrum resource in the resource data set output by the strategy evaluation model, and the reward value is used to characterize the probability of the vehicle allocating each spectrum resource to the target interactive link under the operating state corresponding to the state data set, and the strategy evaluation model is obtained by training according to the model training method provided in the above embodiment; the resource allocation module 330 is used to determine the target spectrum resource whose reward value meets the target numerical condition from the resource data set as the spectrum resource allocated to the target interactive link.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0110] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0111] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0112] In summary, the solution provided by the present application is to obtain an empirical data set, which is generated by using an initial evaluation model during the process of the test vehicle driving through the target road section; based on the empirical data set, the initial evaluation model is iteratively trained until the initial evaluation model meets the first preset condition, and the trained initial evaluation model is obtained as a strategy evaluation model, which is used to output the reward value corresponding to each spectrum resource included in the resource data set corresponding to the target interaction link based on the state data set corresponding to the target interaction link of the vehicle. The initial evaluation model is trained using the empirical data set generated by the initial evaluation model during the process of the test vehicle driving through the target road section, so that the obtained strategy evaluation model can allocate the optimal spectrum resources to the target interaction link according to the state data set of the target interaction link, thereby better meeting the communication needs in the Internet of Vehicles.

[0113] See also Fig.10 , which shows a structural block diagram of an electronic device 400 provided in an embodiment of the present application. The electronic device 400 in the present application may include one or more of the following components: a processor 410, a memory 420, and one or more applications, wherein the one or more applications may be stored in the memory 420 and configured to be executed by one or more processors 410, and the one or more programs are configured to execute the method described in the foregoing method embodiment.

[0114] The processor 410 may include one or more processing cores. The processor 410 uses various interfaces and lines to connect various parts of the entire computer device, and executes various functions of the computer device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 420, and calling data stored in the memory 420. Optionally, the processor 410 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 410 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 410, but may be implemented separately through a communication chip.

[0115] The memory 420 may include a random access memory (RAM) or a read-only memory (ROM). The memory 420 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 420 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the computer device during use.

[0116] Please refer to Fig.11 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0117] The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire an experience data set, wherein the experience data set is generated by using an initial evaluation model when a test vehicle is driving through a target road section, and the experience data set includes a plurality of experience data, each of which includes current state data and expected state data corresponding to a test interaction link of the test vehicle, spectrum resource data allocated to the test interaction link, and an evaluation return value; Input each piece of the experience data into the initial value model to obtain the target reward value corresponding to each piece of the experience data output by the initial value model; Determine the evaluation loss value corresponding to the initial evaluation model based on the target reward value corresponding to each piece of the experience data and the evaluation return value included in each piece of the experience data; The initial evaluation model is iteratively trained according to the evaluation loss value until the initial evaluation model meets a first preset condition, and the trained initial evaluation model is obtained as a strategy evaluation model; the first preset condition includes: the difference between the cumulative reward value corresponding to the initial evaluation model after the current iterative training and the cumulative reward value corresponding to the initial evaluation model obtained by the previous iterative training is less than a preset threshold; the cumulative reward value is determined based on the total channel capacity corresponding to the test interaction link, the remaining load, and the strategy entropy corresponding to the initial evaluation model, and the size of the strategy entropy is negatively correlated with the number of iterative training times of the initial evaluation model; The strategy evaluation model is used to output a reward value corresponding to each spectrum resource included in a resource data set corresponding to a target interactive link of the vehicle based on a state data set corresponding to the target interactive link, the state data set including a channel gain corresponding to the target interactive link, the resource data set including spectrum resources corresponding to at least one frequency band that can be allocated to the target interactive link, and the spectrum resources including subcarrier frequency and transmission power.

2. The method according to claim 1, characterized in that The step of inputting each piece of the experience data into the initial value model to obtain a target reward value corresponding to each piece of the experience data output by the initial value model includes: Determine the value loss value corresponding to the initial value model based on the evaluation return value included in each piece of the experience data; Iteratively training the initial value model according to the value loss value until the initial value model meets a second preset condition, thereby obtaining the trained initial value model; Each piece of the experience data is input into the trained initial value model to obtain a target reward value corresponding to each piece of the experience data output by the trained initial value model.

3. The method according to claim 1, characterized in that The initial value model includes a first value model and a second value model, and inputting each piece of the experience data into the initial value model to obtain a target reward value corresponding to each piece of the experience data output by the initial value model includes: Input each piece of the experience data into the first value model and the second value model to obtain a first reward value output by the first value model and a second reward value output by the second value model; Based on the smallest reward value between the first reward value and the second reward value, a target reward value corresponding to each piece of the experience data is determined.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Obtain the cumulative reward value corresponding to the initial evaluation model.

5. The method according to claim 4, characterized in that The obtaining of the accumulated reward value corresponding to the initial evaluation model includes: Obtaining the remaining load corresponding to the test interaction link; If the remaining load is greater than zero, determining the cumulative reward value based on the load success delivery rate corresponding to the test interaction link, the total channel capacity, and the policy entropy corresponding to the initial evaluation model; If the remaining load is zero, the accumulated reward value is determined based on the maximum effective transmission rate corresponding to the test interactive link, the total channel capacity, and the policy entropy corresponding to the initial evaluation model.

6. A spectrum resource allocation method, characterized in that: The method comprises: Acquire a state data set and a resource data set corresponding to a target interactive link of the vehicle, wherein the state data set includes a channel gain corresponding to the target interactive link, and the resource data set includes spectrum resources corresponding to at least one frequency band that can be allocated to the target interactive link, and the spectrum resources include a subcarrier frequency and a transmission power; Input the state data set into a pre-trained strategy evaluation model, determine the reward value corresponding to each of the spectrum resources in the resource data set output by the strategy evaluation model, the reward value is used to characterize the probability of the vehicle allocating each of the spectrum resources to the target interactive link under the operating state corresponding to the state data set, and the strategy evaluation model is trained according to the method according to any one of claims 1-5; A target spectrum resource whose reward value satisfies a target numerical condition is determined from the resource data set as a spectrum resource allocated to the target interactive link.

7. A model training device, characterized in that: The device comprises: A data acquisition module, used to acquire an experience data set, wherein the experience data set is generated by using an initial evaluation model when a test vehicle travels through a target road section, and the experience data set includes a plurality of experience data, each of which includes current state data and expected state data corresponding to a test interaction link of the test vehicle, spectrum resource data allocated to the test interaction link, and an evaluation return value; Model training module, used to: Input each piece of the experience data into the initial value model to obtain the target reward value corresponding to each piece of the experience data output by the initial value model; Determine the evaluation loss value corresponding to the initial evaluation model based on the target reward value corresponding to each piece of the experience data and the evaluation return value included in each piece of the experience data; The initial evaluation model is iteratively trained according to the evaluation loss value until the initial evaluation model meets a first preset condition, and the trained initial evaluation model is obtained as a strategy evaluation model; the first preset condition includes: the difference between the cumulative reward value corresponding to the initial evaluation model after the current iterative training and the cumulative reward value corresponding to the initial evaluation model obtained by the previous iterative training is less than a preset threshold; the cumulative reward value is determined based on the total channel capacity corresponding to the test interaction link, the remaining load, and the strategy entropy corresponding to the initial evaluation model, and the size of the strategy entropy is negatively correlated with the number of iterative training times of the initial evaluation model; The strategy evaluation model is used to output a reward value corresponding to each spectrum resource included in a resource data set corresponding to a target interactive link of the vehicle based on a state data set corresponding to the target interactive link, the state data set including a channel gain corresponding to the target interactive link, the resource data set including spectrum resources corresponding to at least one frequency band that can be allocated to the target interactive link, and the spectrum resources including subcarrier frequency and transmission power.

8. A spectrum resource allocation device, characterized in that: The device comprises: A data acquisition module, used to acquire a state data set and a resource data set corresponding to a target interactive link of the vehicle, wherein the state data set includes a channel gain corresponding to the target interactive link, and the resource data set includes spectrum resources corresponding to at least one frequency band that can be allocated to the target interactive link, and the spectrum resources include subcarrier frequency and transmission power; a reward determination module, used for inputting the state data set into a pre-trained strategy evaluation model, determining a reward value corresponding to each of the spectrum resources in the resource data set output by the strategy evaluation model, wherein the reward value is used to characterize the probability of the vehicle allocating each of the spectrum resources to the target interactive link under the operating state corresponding to the state data set, and the strategy evaluation model is trained according to the method described in any one of claims 1 to 5; The resource allocation module is used to determine the target spectrum resource whose reward value meets the target numerical condition from the resource data set, as the spectrum resource allocated to the target interactive link.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 5, or to execute the method according to claim 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method according to any one of claims 1 to 5, or to execute the method according to claim 6.

Citation Information

Patent Citations

  • Internet-of-vehicles communication resource allocation method based on graph theory and reinforcement learning

    CN113709882A

  • User behavior prediction method, device and equipment

    CN115456273A