DQN-based time slot allocation and power control method

Abstract

The application provides a time slot allocation and power control method based on DQN, comprising: acquiring multiple sets of limb activity training data; training at least two DQNs according to the limb activity training data and a preset first training number to obtain at least two first time slot allocation networks; determining a target reward value of each first time slot allocation network according to a time slot allocation and power control result of the limb activity training data from each first time slot allocation network, and determining the first time slot allocation network with the maximum target reward value as a second time slot allocation network; training the second time slot allocation network according to the limb activity training data and a preset second training number to obtain a target time slot allocation network; and performing time slot allocation on real-time limb activity data according to the target time slot allocation network to control the transmission power of the allocated time slot. The application can effectively control the transmission time slot and transmission power in the limb activity data transmission process.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of time slot allocation in a limb activity data transmission process, and particularly relates to a time slot allocation and power control method based on DQN. BACKGROUND

[0002] Limb activity data is a data basis for studying or monitoring human motion, which is generally achieved by constructing a body area network communication model of an activity object. Specifically, a transmission device of the body area network communication model is arranged as a central node at the waist of the activity object, and each sensor of the body area network communication model is arranged on the limbs, such as hands and feet, of the activity object. When the activity object moves, each sensor transmits limb activity data corresponding to the limb activity to the transmission device, and then the transmission device uploads the received limb activity data to a server or a cloud platform.

[0003] In the prior art, since the power of each sensor is limited, the power required for the transmission of the limb activity data by the sensor to the transmission device affects the use time of the sensor. When the required power is too large, the efficiency and accuracy of the limb activity data transmission may also be affected. SUMMARY

[0004] The purpose of the application is to overcome the shortcomings and deficiencies in the prior art, and to provide a time slot allocation and power control method based on DQN, which can effectively control the time slot and power allocation in the limb activity data transmission process and improve the efficiency and accuracy of data transmission.

[0005] The first aspect of the embodiment of the application provides a time slot allocation and power control method based on DQN, comprising:

[0006] obtaining a plurality of sets of limb activity training data; the limb activity training data is at least one limb activity data of an activity object for information transmission;

[0007] training at least two DQNs according to the limb activity training data and a preset first training number, to obtain at least two first time slot allocation networks;

[0008] From each of the first time slot allocation networks, a target reward value of each of the first time slot allocation networks is determined according to the time slot allocation and power control result of the limb activity training data, and the first time slot allocation network with the maximum target reward value is determined as a second time slot allocation network;

[0009] training the second time slot allocation network according to the limb activity training data and a preset second training number, to obtain a target time slot allocation network;

[0010] The limb activity real-time data is time-slot allocated according to the target time-slot allocation network, so as to control the transmission power of the allocated time slot.

[0011] Compared with the related art, the at least two DQNs are initially trained according to the preset first training number, and the corresponding target reward value is determined according to the time-slot allocation and power control result of the trained DQN, and the DQN with the maximum target reward value is selected for secondary training, so that the target time-slot allocation network experienced the primary training and the secondary training can be obtained, which can be used for controlling the time slot and power used for transmitting the limb activity real-time data, effectively controlling the transmission time slot and transmission power, and achieving the technical effects of improving the efficiency and accuracy of data transmission.

[0012] In order to make the application clearer, the specific embodiments of the application will be described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 The flow chart of the DQN-based time-slot allocation and power control method of one embodiment of the application.

[0014] Figure 2 The schematic diagram of the topology structure construction of the DQN-based time-slot allocation and power control method of one embodiment of the application.

[0015] Figure 3 The step S of the DQN-based time-slot allocation and power control method of one embodiment of the application. DETAILED DESCRIPTION

[0016] In order to make the application clearer, the specific embodiments of the application will be described below in conjunction with the accompanying drawings.

[0017] It should be clear that the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor belong to the scope of protection of the application.

[0018] The following description refers to the accompanying drawings. Unless otherwise indicated, same or similar elements in different drawings are denoted by the same reference numerals. In the description of the present application, it is to be understood that the terms "first", "second", "third", etc. are used only to distinguish similar objects, and do not necessarily indicate a particular order or sequence, nor are they intended to indicate or imply relative importance of the referenced objects. The specific meaning of the above terms in the present application can be understood according to the specific circumstances by those of ordinary skill in the art. The singular forms "a", "an" and "the" used in the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. The word "if" used herein can be interpreted as "when" or "when" or "in response to determining".

[0019] In addition, in the description of the present application, "multiple" means two or more, unless otherwise specified. The association between the objects described by "and / or" indicates that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0020] Referring to Figure 1 , which is a flowchart of a DQN-based time slot allocation and power control method according to an embodiment of the present application, wherein DQN is Deep Q-network, which is an algorithm that combines neural networks and Q-Learning. DQN uses a neural network to learn the Q-value function, which is a function that maps states and actions to Q-values, indicating the expected return obtained by performing the action in a particular state.

[0021] The DQN-based time slot allocation and power control method according to the present embodiment includes:

[0022] S1: Obtain a plurality of sets of limb activity training data; the limb activity training data is at least one limb activity data of an active object for information transmission.

[0023] The active object is a living animal, for example, a human being.

[0024] The activity training data refers to the limb activity data transmitted by the sensor disposed on the human body to the transmission device through time slot allocation. The time slot allocation algorithm is an algorithm used to coordinate data transmission, with the purpose of reasonably allocating transmission time slots according to the existing channel state.

[0025] The limb movement data refers to activity data generated by the sensors when identifying the limb movement of the living object, such as data of limb movement behaviors of the living object, such as walking, lifting legs, lifting hands, waving hands, etc. The limb movement data includes data related to limb movement, such as the corresponding limb part of the sensor, the moving direction, the moving distance, or the moving amplitude.

[0026] In addition to the limb movement data, each set of activity training data also includes the battery level of the sensor, the channel state of the sensor and the transmission device, and the average information age of the limb movement data after time slot allocation. The data format of the limb movement training data can be (CS(k), P(k), A(k), CS(k+1)), where CS(k) is the channel state before the kth time slot allocation, P(k) is the battery level of the sensor, A(k) is the average information age of the limb movement data before the kth time slot allocation, and CS(k+1) is the channel state before the k+1th time slot allocation, which can also be understood as the channel state after the kth time slot allocation.

[0027] Therefore, the S1: obtaining a plurality of sets of limb movement training data, comprises:

[0028] S11: obtaining the battery level of each sensor arranged on the limbs of the living object.

[0029] The battery level can be the real-time power of the sensor, which can be obtained by using a unit or integrated circuit for detecting current in the prior art.

[0030] S12: obtaining the channel state between each sensor and the transmission device arranged on the living object according to the topological relationship of each sensor and the transmission device.

[0031] The topological relationship refers to the topological structure obtained according to the spatial relationship of each sensor and the transmission device. Specifically, the topological relationship affects the wireless body area network communication environment of the sensor and the transmission device, and the basis of the wireless body area network communication environment is the channel. Therefore, the topological relationship and the channel state of the link between the sensor and the transmission device are related. The channel is the medium for signal transmission, and the channel state refers to the state of each time slot in the channel.

[0032] Specifically, step S12: the step of obtaining the channel state between each sensor and the transmission device arranged on the living object according to the topological relationship of each sensor and the transmission device, comprises:

[0033] S121: obtaining the topological relationship according to the spatial relationship of each sensor and the transmission device arranged on the living object.

[0034] The spatial relationship of each sensor and the transmission device is as shown inFigure 2 As shown, each sensor is arranged on a limb of the moving object. One limb can be provided with multiple sensors, and the transmission device serves as a central node for receiving data sent by the sensors and uploading the received data to a server or a cloud platform, etc. Therefore, the transmission device can be arranged between multiple sensors, for example, at the waist of the moving object. Then, a topology relationship is constructed based on the spatial relationship between the sensors arranged on each limb and the transmission device.

[0035] S122: Construct a wireless body area network communication environment of the living object according to the topology relationship.

[0036] In this embodiment, the sampling frequency of the channel information of the wireless body area network communication environment constructed according to the topology relationship is 20 Hz. However, the wireless body area network communication environment constructed by multiple sensors and a transmission device is a model of a wireless signal propagation environment. This model assumes that the signal amplitude is random after the signal passes through the wireless channel, that is, channel fading occurs. Therefore, when sampling, the different positions of multiple sensors and a transmission device will not only result in different topology relationships, but also result in different channel fading.

[0037] S123: According to the wireless body area network communication environment, obtain the channel state between each sensor and the transmission device.

[0038] The channel state refers to the channel fading state of the communication link between each sensor and the transmission device at each time slot. For example, when the communication environment of the time slot is good, the channel fading of the time slot is small; when the communication environment of the time slot is poor, the channel fading of the time slot is large. The channel state will affect the result of time slot allocation, because in the time slot allocation process, time slots with small channel fading will be selected for data transmission as much as possible. The smaller the channel fading, the smaller the transmission power required for the sensor to transmit data, so as to improve the network energy efficiency.

[0039] S13: Obtain the limb movement training data according to the channel state, the battery capacity of each sensor, and the preset initial average information age.

[0040] The initial average information age is a preset value, and the real-time average information age corresponding to the time slot allocation in the training process refers to the average information age of the time slot allocation result of the last limb movement data. The average information age can be the sum of the information ages of the data sent by the sensor in a period of time divided by the length of the period of time. The information age is the time elapsed from the generation of the data to the reception of the data at the destination.

[0041] S2: Train at least two DQNs according to the limb movement training data and a preset first training number to obtain at least two first time slot allocation networks.

[0042] The at least two DQNs for training according to the first training number are DQNs with the same initial network structure and the same initial network parameters. The first training number is a training number of the first training preset by the user, for example, 40.

[0043] At the beginning of the training, an experience replay pool of an arbitrary size can be created, for example, 10,000. In the initial state, the experience replay pool is empty, and after step S1, a plurality of sets of limb activity training data can be stored in the experience pool. The data in the experience replay pool is used for subsequent neural network training, and when 500 pieces of data are stored, the training can be started.

[0044] The training process of step S2 can be to take out 64 pieces of data from the experience replay pool, send them to each DQN for training, update the network parameters, and store the training data in the experience replay pool.

[0045] As a preferred embodiment, the data entering the experience replay pool can be limited by the information age of the data. For example, if the average information age of a sensor in the data set exceeds the preset experience pool age threshold, the corresponding sensor needs to perform an additional data transmission immediately to obtain new data and update the information age to meet the information age threshold limit. The additional data transmission required by the sensor is an immediate data transmission, which can not consider the channel state. Through the above limitation, the information age of the data in the experience replay pool can be reduced, and the training time can be saved.

[0046] S3: From each of the first time slot allocation networks, according to the time slot allocation and power control results of the limb activity training data, determine the target reward value of each of the first time slot allocation networks, and determine the first time slot allocation network with the maximum target reward value as the second time slot allocation network.

[0047] The target reward value refers to the last temporary reward value of each DQN after reaching the preset first training number, and the temporary reward value obtained each time during the training is used to indicate the training direction of the network parameters of the DQN. Each time the temporary reward value is related to the time slot allocation and power control results. For example, the smaller the power, the higher the temporary reward value, or the energy consumption corresponding to the power is less than the sum of the received energy and the battery capacity of the sensor, the temporary reward value is positive, otherwise, the temporary reward value is negative, and the like.

[0048] It should be noted that, during the training process, although the target reward value plays an indicative role in the training direction of the network parameters of the DQN, the result of the time slot allocation performed by the DQN each time has randomness, so even if the time slot allocation training is performed for power, as the number of training times increases, the network parameters of each DQN will present different degrees of difference, and thus after reaching the first number of training times, the target reward values of each DQN will also present differences, and the second time slot allocation network with the maximum target reward value is the DQN with the best training effect and the most training potential for the time slot allocation training for power. The DQN suitable for time slot allocation and power control can be obtained by continuing to train the second time slot allocation network.

[0049] As a preferred embodiment, the DQN in the training process can be limited by an information age, for example, if the average information age corresponding to the data transmission of a group of data performed by the sensor according to the time slot allocation result of the DQN exceeds a preset training age threshold, the DQN needs to perform new time slot allocation for the group of data again, so that the sensor performs data transmission of the group of data immediately according to the new time slot allocation, and only when the average information age corresponding to the new data transmission is less than or equal to the preset training age threshold, it is confirmed that the current training of the DQN is valid. The additional data transmission performed by the sensor is immediate data transmission, which can not consider the channel state. Through the above limitation, only the time slot allocation with an average information age less than or equal to the training age threshold can be retained in the training process, so that the time slot allocation of the second time slot allocation network obtained by the training has the effect of limiting the information age.

[0050] S4: training the second time slot allocation network according to the limb activity training data and a preset second number of training times to obtain a target time slot allocation network.

[0051] The second number of training times and the first number of training times are independent training times, and the second number of training times is greater than the first number of training times. According to the second number of training times, the second time slot allocation network is trained twice to improve the accuracy of the time slot allocation for power of the second time slot allocation network, so as to obtain the target time slot allocation network.

[0052] S5: performing time slot allocation on limb activity real-time data according to the target time slot allocation network to control the transmission power of the allocated time slot.

[0053] By performing time slot allocation on the limb activity real-time data transmission process of the sensor through the target time slot allocation network, the transmission power of the allocated time slot can be effectively controlled, so as to control the reduction of the energy of the sensor, improve the life cycle of the sensor, and make the data transmission more stable and accurate.

[0054] Compared with the related art, the application performs initial training on the at least two DQNs according to the preset first training number, determines the corresponding target reward value according to the time slot allocation and power control result of the trained DQN, selects the DQN with the maximum target reward value to continue secondary training, can obtain the target time slot allocation network that has experienced the initial training and the secondary training, and can be used for controlling the time slot and power used for transmitting the real-time data of the limb activity, thereby achieving effective control of the transmission time slot and the transmission power and achieving the technical effects of improving the efficiency and accuracy of data transmission.

[0055] In a feasible embodiment, the step S2 of training the at least two DQNs according to the limb activity training data and the preset first training number to obtain the at least two first time slot allocation networks comprises:

[0056] S21: determining a temporary reward value of each first time slot allocation network during training according to the time slot and power of the data transmitted by the sensor allocated by each DQN.

[0057] The calculation of the temporary reward value is associated with the power consumed by the data transmitted by the sensor after the time slot allocation, that is, associated with the consumed energy.

[0058] S22: obtaining an allocation reward value according to the network parameters of each DQN and the temporary reward value.

[0059] The allocation reward value refers to the optimal Q table value of each DQN in the training, which can be calculated according to the network parameters and the corresponding temporary reward value.

[0060] S23: training the network weight parameters of each DQN according to the first training number and the allocation reward value of each training to obtain the at least two first time slot allocation networks.

[0061] Since the allocation reward value is related to the temporary reward value, and the temporary reward value is related to the power of data transmission after time slot allocation, the allocation reward value is also related to the power of data transmission after time slot allocation, and therefore the allocation reward value has the function of guiding the training of the DQN.

[0062] The step S3 of determining the target reward value of each first time slot allocation network according to the time slot allocation and power control result of the limb activity training data comprises:

[0063] S31: determining the temporary reward value of the last training of each DQN as the target reward value.

[0064] The last training refers to the last training reaching the preset first training number of times, and the target reward value of each DQN can indicate the training effect of each DQN. Specifically, the higher the target reward value, the better the training effect of the corresponding DQN.

[0065] In the embodiment, the allocation reward value can be obtained according to the time slot allocation and the power control result to train the DQN, so that the transmission power of the allocated time slot has a guiding effect on the training, and the training effect of each DQN in the initial training is determined according to the size of the target reward value of each DQN in the last training.

[0066] Please refer to Figure 3 In an available embodiment, the step S21 of determining the temporary reward value of each first time slot allocation network during training includes:

[0067] S211: Obtain the transmission power of each DQN when the sensor transmits data in the allocated time slot.

[0068] The transmission power of the time slot can be detected by a detection device or calculated by the energy received by the sensor and the change of the battery capacity.

[0069] S212: Obtain the transmission time required for the transmission data of each DQN allocated sensor.

[0070] The transmission time refers to the time consumed by the sensor for transmitting data to the transmission device after the time slot allocation of each DQN.

[0071] S213: Obtain the first energy consumed by the transmission data according to the transmission power and the transmission time.

[0072] S214: Obtain the second energy received by the sensor.

[0073] The energy refers to the energy transmitted from the energy station to the sensor, and the energy transmitted from the energy station to the sensor can increase the use time of the sensor and reduce the charging frequency of the sensor.

[0074] S215: If the first energy is greater than the sum of the second energy and the battery capacity of the sensor, determine that the temporary reward value is negative, and if the first energy is less than or equal to the sum of the second energy and the battery capacity of the sensor, determine that the temporary reward value is positive.

[0075] The temporary reward value can be obtained by the following formula:

[0076]

[0077] wherein, is the first energy; is the transmission power of the data transmitted by the sensor n in the time slot k; τ is the transmission time; is the second energy; ρ is the energy conversion coefficient; H n is the Rayleigh fading factor of the transmission path of the data transmitted by the sensor n, P is the power of the received energy; q n is the battery power of the sensor n.

[0078] In the embodiment, the first energy corresponding to the consumed power can be obtained according to the power after time slot allocation, and then the corresponding reward value is determined according to the comparison between the first energy and the sum of the second energy received by the sensor and the battery power of the sensor, so as to accurately obtain the reward value corresponding to the power meeting the above comparison result.

[0079] In a feasible embodiment, the step S22 of obtaining the allocation reward value according to the network parameters of each DQN and the temporary reward value comprises:

[0080] The allocation reward value is obtained by the following formula:

[0081] y t =r t +γmaxQ(S t+1 ,a′;θ t )

[0082] wherein, y t is the allocation reward value; r t is the reward value; γ is the discount factor, γ∈[0,1); Q(S t+1 ,a′;θ t ) is the time slot allocation result of the DQN.

[0083] In the embodiment, the allocation reward value corresponding to the temporary reward value can be accurately obtained by the above formula.

[0084] In a feasible embodiment, the step S23 of training the network weight parameters of each DQN according to the first training number and the allocation reward value of each training to obtain at least two first time slot allocation networks comprises:

[0085] S231: training the network weight parameters of each DQN according to the first training number and the preset gradient direction.

[0086] Wherein, the network weight parameters of each DQN after training can be obtained by the following formula:

[0087]

[0088] Where, θ t+1 Let θ be the network weight parameters after the t-th training iteration; t Let be the network weight parameters before the t-th training iteration; η be the learning rate; Q(S) t+1 ,a′;θ t ) represents the time slot allocation result of DQN after the t-th training iteration; The gradient direction.

[0089] S232: Obtain the loss function after training each of the DQNs.

[0090] The loss function after training each of the DQN methods can be obtained using the following formula:

[0091]

[0092] Where Loss(θ) is the loss function; targetQ(S) t ,a;θ t Q(S) represents the target time slot allocation result. t ,a;θ t ) represents the DQN time slot allocation result after training, where N is the first training iteration.

[0093] S233: The DQN corresponding to the loss function that is less than the preset function threshold is determined as the first time slot allocation network.

[0094] Since the loss function represents the difference between DQN training before and after, when the loss function is greater than or equal to the preset function threshold, it means that the training is not satisfactory and should be re-executed. When the loss function is less than the preset function threshold, it means that the training is satisfactory and is a valid training, and does not need to be re-executed.

[0095] In this embodiment, the training of DQN network weight parameters can be guided by gradient training, and then the effectiveness of training can be judged by loss function, thereby improving the effectiveness of training.

[0096] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart and / or block diagram in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart and / or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions Figure 1 The flowchart and / or block diagram in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart and / or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions Figure 1 The flowchart and / or block diagram in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart and / or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions Figure 1 The flowchart and / or block diagram in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart and / or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions

[0098] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart and / or block diagram in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart and / or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions Figure 1 The flowchart and / or block diagram in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart and / or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions

[0099] In a typical configuration, a computing device includes one or more processing units (CPU), input / output interfaces, network interfaces, and memory.

[0100] The memory can include non-persistent memory and / or persistent memory, for example, read only memory (ROM) and / or flash memory, etc. in a computer readable storage medium. The memory is an example of computer readable media.

[0101] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0102] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0103] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A DQN-based time slot allocation and power control method, characterized in that, The method comprises the following steps: obtaining a plurality of sets of limb activity training data; the limb activity training data is at least one limb activity data of an activity object for information transmission; training at least two DQNs according to the limb activity training data and a preset first training number, to obtain at least two first time slot allocation networks, comprising: determining a temporary reward value of each first time slot allocation network during training according to the time slot and power allocated by each DQN to the data transmitted by the sensor; obtaining an allocation return value according to the network parameters of each DQN and the temporary reward value; training the network weight parameters of each DQN according to the first training number and the allocation return value of each training, to obtain at least two first time slot allocation networks; from each of the first time slot allocation networks, determining a target reward value of each first time slot allocation network according to the time slot allocation and power control result of the limb activity training data, comprising: determining the temporary reward value of the last training of each DQN as the target reward value; determining the first time slot allocation network with the maximum target reward value as the second time slot allocation network; training the second time slot allocation network according to the limb activity training data and a preset second training number, to obtain a target time slot allocation network; allocating time slots according to the target time slot allocation network for real-time limb activity data, to control the transmission power of the allocated time slots.

2. The DQN-based time slot allocation and power control method of claim 1, wherein, The step of obtaining a plurality of sets of limb activity training data comprises: obtaining the battery capacity of each sensor arranged on the limbs of a living object; obtaining the channel state between each sensor and a transmission device arranged on the living object according to the topological relationship of each sensor and the transmission device; obtaining the limb activity training data according to the channel state, the battery capacity of each sensor, and a preset initial average information age.

3. The DQN-based time slot allocation and power control method of claim 2, wherein, The step of obtaining the channel state between each sensor and a transmission device arranged on the living object according to the topological relationship of each sensor and the transmission device comprises: obtaining the topological relationship according to the spatial relationship of each sensor and the transmission device arranged on the living object; constructing a wireless body area network communication environment of the living object according to the topological relationship; obtaining the channel state between each sensor and the transmission device according to the wireless body area network communication environment.

4. The DQN-based time slot allocation and power control method of claim 1, wherein, The step of determining a temporary reward value of each first time slot allocation network during training according to the time slot and power allocated by each DQN to the data transmitted by the sensor comprises: obtaining the transmission power of each DQN when allocating the time slot for information transmission by the sensor; obtaining the transmission time required by the transmission data of each DQN; obtaining the first energy consumed by the transmission data according to the transmission power and the transmission time; obtaining the second energy received by the sensor; If the first energy is greater than the sum of the second energy and the battery power of the sensor, the temporary reward value is determined as a negative value; if the first energy is less than or equal to the sum of the second energy and the battery power of the sensor, the temporary reward value is determined as a positive value.

5. The DQN-based time slot allocation and power control method of claim 4, wherein, If the first energy is greater than the second energy, the temporary reward value is determined as a negative value. If the first energy is less than or equal to the second energy, the step of determining the temporary reward value as a positive value comprises: The temporary reward value is obtained by the following formula: wherein, is a first energy; is the transmission power of the data transmitted by sensor n in time slot k; τ is the transmission time; is a second energy; ρ is an energy conversion coefficient; H n is the Rayleigh fading factor of the transmission path of the data transmitted by sensor n, P is the power of the received energy; q n is the battery level of sensor n; r t is the temporary reward value.

6. The DQN-based time slot allocation and power control method of claim 1, wherein, The step of obtaining the allocation reward value according to the network parameters of each DQN and the temporary reward value comprises: The allocation reward value is obtained by the following formula: y t = r t + γmaxQ(S t+1 , a'; θ t ) where y t is the allocation reward value; r t is the temporary reward value; γ is the discount factor, γ ∈ [0, 1); Q(S t+1 , a′; θ t ) is the time slot allocation result of the DQN.

7. The DQN-based time slot allocation and power control method of claim 1, wherein, The step of obtaining at least two first time slot allocation networks according to the first training number and the allocation reward value of each training comprises: The network weight parameters of each DQN are trained according to the first training number and a preset gradient direction; Loss functions of each DQN after training are obtained; The DQN corresponding to the loss function less than a preset function threshold is determined as a first time slot allocation network.

8. The DQN-based time slot allocation and power control method of claim 7, wherein, The step of training the network weight parameters of each DQN according to the first training number and a preset gradient direction comprises: The network weight parameters of the DQN after training are obtained by the following formula: wherein θ t+1 is the network weight parameter after the tth training; θ t is the network weight parameter before the tth training; η is the learning rate; Q(S t+1 ,a′; θ t ) is the time slot allocation result of the DQN after the tth training. is the gradient direction. 9.The DQN-based time slot allocation and power control method of claim 7, wherein, The step of obtaining the loss functions of each DQN after training comprises: The loss functions of each DQN after training are obtained by the following formula: Where Loss(θ) is the loss function; targetQ(S) t ,a;θ t Q(S) represents the target time slot allocation result. t ,a;θ t ) represents the DQN time slot allocation result after training, where N is the first training iteration.

Images