Communication parameter adjusting method and device, computer program product and network equipment

Abstract

The invention discloses a communication parameter adjusting method and device, a computer program product and network equipment. The method comprises the following steps: acquiring an environment parameter of an area where a target terminal is located and a communication parameter of the target terminal in the area; the environment parameters are analyzed through a pre-trained communication parameter adjustment decision model, a communication parameter adjustment strategy meeting the index requirements of the multiple communication indexes is obtained, the communication parameter adjustment decision model is obtained through training based on a preset reinforcement learning algorithm, and the communication parameter adjustment strategy meets the index requirements of the multiple communication indexes. The communication indexes at least comprise a communication quality index, a network congestion index and a communication cost index; and adjusting the communication parameters according to the communication parameter adjustment strategy. The technical problem that the communication indexes after the communication parameters are adjusted are poor due to the fact that the communication parameters of the terminal cannot be intelligently adjusted in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a communication parameter adjustment method and apparatus, a computer program product, and a network device. Background Art

[0002] With the rapid development of wireless communication technology, satellite communication and terrestrial communication have their own advantages in different scenarios. In order to ensure that the services of user equipment in the network are not interrupted, it is necessary to ensure that user equipment can achieve smooth switching between different networks to improve the continuity and stability of communication.

[0003] In order to achieve the above purpose, relevant technical personnel usually only rely on human experience to adjust the communication mode of the terminal in the current environment according to environmental parameters. For example, in areas with better network quality (such as shopping malls, communities, campuses and other areas with good ground network coverage), the communication mode is adjusted from satellite communication to ground communication, and in areas with poor network quality (such as remote areas, water areas, border areas and other areas with poor ground network coverage), the communication mode is adjusted from ground communication to satellite communication, thereby providing better network services to user devices. However, this adjustment cannot synchronously adjust the communication parameters related to the communication mode, so that the adjusted communication mode performs poorly in terms of communication cost, communication quality, network congestion, etc., resulting in a poor user experience.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a communication parameter adjustment method and apparatus, a computer program product, and a network device to at least solve the technical problem that the related technology cannot intelligently adjust the communication parameters of the terminal, resulting in poor communication indicators after adjusting the communication parameters.

[0006] According to one aspect of an embodiment of the present application, a communication parameter adjustment method is provided, comprising: obtaining environmental parameters of an area where a target terminal is located and communication parameters of the target terminal in the area; analyzing the environmental parameters using a pre-trained communication parameter adjustment decision model to obtain a communication parameter adjustment strategy that enables multiple communication indicators of the network to meet preset indicator requirements, wherein the communication parameter adjustment decision model is trained based on a preset reinforcement learning algorithm, and the communication indicators include at least: a communication quality indicator, a network congestion indicator, and a communication cost indicator; and adjusting the communication parameters according to the communication parameter adjustment strategy.

[0007] Optionally, the environmental parameters include at least one of the following: communication channel characteristic parameters, mobile state parameters, energy state parameters, and communication resource parameters, wherein the communication channel characteristic parameters include at least one of the following: signal strength, signal-to-noise ratio, network transmission delay, and data packet loss rate; the mobile state parameters include at least one of the following: position coordinates, moving speed, and moving acceleration; the energy state parameters include at least one of the following: current power level and remaining available time; the communication resource parameters include: the amount of communication resources using ground communication equipment or the amount of communication resources using satellite communication equipment.

[0008] Optionally, the type of reinforcement learning algorithm includes at least one of the following: Q learning algorithm, deep Q network algorithm, actor critic algorithm.

[0009] Optionally, the training process of the communication parameter adjustment decision model includes: constructing an initial decision model and initializing the model parameters of the initial decision model; determining an experience pool and initializing the priority of each sample in the experience pool; looping the following steps until a preset number of iterations is reached to obtain a communication parameter adjustment decision model; in each iteration, initializing environmental parameters; using a reinforcement learning algorithm to analyze current environmental parameters, determining a corresponding second communication parameter adjustment strategy from a variety of communication parameter adjustment strategies, and using a preset reward function to determine the reward corresponding to the second communication parameter adjustment strategy, and updating the current environmental parameters to obtain new environmental parameters; using the current environmental parameters, new environmental parameters, the second communication parameter adjustment strategy and the corresponding rewards as samples, and storing the samples in the experience pool; and continuously adjusting the model parameters of the initial decision model based on each sample in the experience pool.

[0010] Optionally, the reward function is expressed as follows:

[0011] R(s,a,s')=ω1*ΔQ(s,a,s')+ω2*ΔE(s,a,s')-ω3*ΔC(s,a,s')

[0012] Where s represents the environmental parameter of the area where the terminal is located, a represents the communication parameter adjustment strategy adopted when the environmental parameter of the area where the terminal is located is s, s' represents the new environmental parameter after executing the communication parameter adjustment strategy a, ΔQ(s,a,s') represents the communication quality index, ΔE(s,a,s') represents the network congestion index, ΔC(s,a,s') represents the communication cost index, ω1, ω2, ω3 represent the weights corresponding to ΔQ(s,a,s'), ΔE(s,a,s'), ΔC(s,a,s'), respectively; where the expression of the communication quality index ΔQ(s,a,s') is as follows:

[0013]

[0014] P old and Pnew Respectively represent the signal strength before and after the communication parameters are adjusted, N old and N new They represent the signal-to-noise ratio before and after the communication parameters are adjusted, respectively; w1 and w2 represent the weights corresponding to the signal strength and the signal-to-noise ratio, respectively; wherein, the expression of the network congestion index ΔE(s,a,s') is as follows:

[0015]

[0016] D old and D new They represent the network transmission delay before and after the communication parameters are adjusted, L new and L old They represent the data packet loss rate before and after the communication parameters are adjusted, w3 and w4 represent the weights corresponding to the network transmission delay and the data packet loss rate, respectively. The expression of the communication cost index ΔC(s,a,s') is as follows:

[0017]

[0018] U old and U new Respectively represent the communication resource consumption before and after the communication parameters are adjusted; S new and S old They represent the energy costs before and after the communication parameters are adjusted respectively; w5 and w6 represent the weights corresponding to the communication resource consumption and energy cost respectively.

[0019] Optionally, storing the sample in the experience pool includes: storing the sample in the experience pool according to a priority experience playback mode or an experience playback mode.

[0020] Optionally, the communication parameters include at least one of the following: communication frequency, communication power, data transmission rate, and communication mode, wherein the communication mode includes: terrestrial communication or satellite communication.

[0021] According to another aspect of an embodiment of the present application, a communication parameter adjustment device is also provided, including: an acquisition module, used to obtain environmental parameters of an area where a target terminal is located and communication parameters of the target terminal in the area; a determination module, used to analyze the environmental parameters using a pre-trained communication parameter adjustment decision model, and obtain a communication parameter adjustment strategy that makes multiple communication indicators meet preset indicator requirements, wherein the communication parameter adjustment decision model is trained based on a preset reinforcement learning algorithm, and the communication indicators include at least: communication quality indicators, network congestion indicators, and communication cost indicators; an adjustment module, used to adjust the communication parameters according to the communication parameter adjustment strategy.

[0022] According to another aspect of an embodiment of the present application, a computer program product is further provided. The computer program product includes a stored computer program, wherein the computer program implements the above-mentioned communication parameter adjustment method when executed by a processor.

[0023] According to another aspect of an embodiment of the present application, an electronic device is further provided, the electronic device comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned communication parameter adjustment method through the computer program.

[0024] In an embodiment of the present application, a communication parameter adjustment decision model based on a reinforcement learning algorithm is used to analyze the environmental parameters of the area where the continuously detected terminal is located, and a corresponding communication parameter adjustment strategy is obtained. The communication parameters of the area where the terminal is located are adjusted in real time according to the communication parameter adjustment strategy to adapt to the dynamically changing communication environment and achieve the purpose of optimizing the communication quality, thereby solving the technical problem that the related technology cannot intelligently adjust the communication parameters of the terminal, resulting in poor communication indicators after adjusting the communication parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0026] Figure 1 is a hardware structure block diagram of an optional computer terminal for implementing a communication parameter adjustment method according to an embodiment of the present application;

[0027] Figure 2 is a flow chart of an optional communication parameter adjustment method according to an embodiment of the present application;

[0028] Figure 3 is a schematic structural diagram of an optional communication parameter adjustment device according to an embodiment of the present application;

[0029] Figure 4 It is a schematic diagram of the structure of an optional network device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] In addition, the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0033] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0034] Satellite communication: It is the communication between radio communication stations on the earth (including the ground and the lower atmosphere) using satellites as relays. The satellite communication system consists of two parts: satellites and earth stations. The characteristics of satellite communication are: large communication range, that is, as long as the radio waves emitted by the satellite are covered, communication can be carried out between any two points; high reliability, it is not easily affected by land disasters; rapid circuit opening, that is, as long as the earth station circuit is set up, it can be opened; multi-address communication, that is, it can be received in multiple places and broadcast can be realized economically; multi-address connection, that is, the same channel can be used in different directions or different intervals.

[0035] The Q-learning algorithm is a reinforcement learning algorithm based on value functions, which determines the optimal strategy by learning the value function of the state-action pair.

[0036] The Deep Q-Networks (DQN) algorithm is a deep learning extension of Q-learning. It can handle high-dimensional state space and action space by using neural networks to approximate the value function.

[0037] Policy Gradient algorithms can directly learn policies instead of learning value functions. They maximize the expected return by optimizing policy parameters, and are applicable to continuous action spaces and high-dimensional state spaces.

[0038] The Actor-Critic algorithm combines value function learning and policy learning, where the Actor learns the policy and the Critic learns the value function, and uses the Critic's estimation to guide the Actor's learning.

[0039] Example 1

[0040] Currently, traditional networks mainly rely on narrowband Internet of Things (NB-IoT). Things, NB-IoT), 2G and other types of ground networks to achieve data transmission, but this communication parameter is only applicable to relevant scenarios with ground network coverage on land. For areas without network coverage, for example: (1) Remote area communication: In remote areas, mountainous areas, deserts and other areas with complex terrain, the ground network may not be able to cover or the signal quality is poor; (2) Marine communication: The marine communication environment is complex, the coverage of ground base stations is limited, and marine equipment may need to communicate with land communication equipment; (3) Mobile communication: Mobile communication equipment may communicate in different locations and environments, and it is necessary to dynamically select the best communication parameters; (4) Disaster recovery communication: When natural disasters, emergency rescue and other disasters occur, the ground network may be affected or interrupted, and communication needs to be carried out through satellite communication; (5) Border area communication: Border areas usually have complex terrain and imperfect communication infrastructure, and ground communication parameters will no longer be applicable. Therefore, satellite communication parameters can be applied at this time, but how to achieve the best switching between ground communication parameters and satellite communication parameters, that is, how to achieve the best communication quality, minimum network congestion and lowest communication cost after switching communication parameters, is still a major problem that needs to be solved urgently. In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0041] According to an embodiment of the present application, a method embodiment of a communication parameter adjustment method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a communication parameter adjustment method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0043] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0044] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the communication parameter adjustment method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the communication parameter adjustment method of the above-mentioned application is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0046] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0047] In the above operating environment, Figure 2 is a flow chart of an optional communication parameter adjustment method according to an embodiment of the present application, such as Figure 2 As shown, the method at least includes steps S202-S206, wherein:

[0048] Step S202: Acquire environmental parameters of an area where a target terminal is located and communication parameters of the target terminal in the area.

[0049] Step S204: Analyze the environmental parameters using the pre-trained communication parameter adjustment decision model to obtain a communication parameter adjustment strategy that enables multiple communication indicators of the network to meet preset indicator requirements.

[0050] The communication parameter adjustment decision model is obtained by iterative training using environmental parameters, communication indicators and new parameter adjustment strategies related to the communication system, and adopting a reinforcement learning algorithm. The communication indicators at least include: communication quality indicators, network congestion indicators, and communication cost indicators. Therefore, the communication indicators all meet the preset indicator requirements, which means that the value of the communication quality indicator is not lower than the preset first threshold (i.e., the highest), the value of the network congestion indicator is lower than the preset second threshold (i.e., the lowest), and the value of the communication cost indicator is lower than the preset third threshold (i.e., the lowest). In other words, the environmental parameters are analyzed through the communication parameter adjustment decision model to obtain the best communication parameter adjustment strategy corresponding to the current environmental parameters, so that the communication system can provide the best communication service to the target terminal.

[0051] Step S206: adjusting the communication parameters according to the communication parameter adjustment strategy.

[0052] Specifically, the communication system may adjust the communication parameters of the target terminal in the current area according to the communication parameter adjustment strategy obtained above, so that the target terminal can achieve high-quality communication through the adjusted communication parameters, thereby improving user experience.

[0053] Based on the scheme defined by the above steps S202 to S206, it can be known that in an embodiment, the communication system uses a communication parameter adjustment decision model based on a reinforcement learning algorithm to analyze the environmental parameters of the area where the terminal is continuously detected, obtains a corresponding communication parameter adjustment strategy, and adjusts the communication parameters of the area where the terminal is located in real time according to the communication parameter adjustment strategy to adapt to the dynamically changing communication environment and achieve the purpose of optimizing the communication quality, thereby solving the technical problem that the related technology cannot intelligently adjust the communication parameters of the terminal, resulting in poor communication indicators after adjusting the communication parameters.

[0054] The above method of this embodiment is further introduced below.

[0055] As an optional implementation, in the technical solution provided in the above step S202, the environmental parameters include but are not limited to: communication channel characteristic parameters, mobile state parameters, energy state parameters, communication resource parameters, weather conditions, etc.

[0056] Specifically, the above-mentioned communication channel characteristic parameters include but are not limited to: signal strength (represented by P), signal-to-noise ratio (represented by N), network transmission delay (represented by D), and data packet loss rate (represented by L); the above-mentioned mobile state parameters include but are not limited to: location coordinates, moving speed, and moving acceleration; the above-mentioned energy state parameters include but are not limited to: current power and remaining available time; the above-mentioned communication resource parameters include: the amount of communication resources using ground communication equipment or the amount of communication resources using satellite communication equipment; the above-mentioned weather conditions refer to the real-time weather conditions of the current environment of the target terminal (such as sunny, cloudy, rainy, etc.).

[0057] As an optional implementation, in the technical solution provided in the above step S204, the training process of the above communication parameter adjustment decision model includes the following steps:

[0058] Step 1: Build an initial decision model and determine the experience pool, where the experience pool includes multiple samples, and each sample has a different priority;

[0059] Step 2: looping through the following steps until a preset number of iterations is reached, and obtaining a communication parameter adjustment decision model for the communication mode;

[0060] Step 3: Initialize the environmental parameters during each iteration; use the reinforcement learning algorithm to analyze the current environmental parameters, determine the corresponding second communication parameter adjustment strategy from a variety of communication parameter adjustment strategies, and use the preset reward function to determine the reward corresponding to the second communication parameter adjustment strategy, and update the current environmental parameters to obtain new environmental parameters; use the current environmental parameters, new environmental parameters, the second communication parameter adjustment strategy and the corresponding rewards as samples, and store the samples in the experience pool; continuously adjust the model parameters of the initial decision model based on the samples in the experience pool.

[0061] During the above training process, the intelligent agent (terminal) can be evaluated regularly to test its performance in a real communication environment, so as to better discover the weaknesses of the model and further improve the model.

[0062] Specifically, the expression of the above reward function is as follows:

[0063] R(s,a,s')=ω1*ΔQ(s,a,s')+ω2*ΔE(s,a,s')-ω3*ΔC(s,a,s')

[0064] Among them, s represents the environmental parameters of the area where the terminal is located, a represents the communication parameter adjustment strategy adopted when the environmental parameters of the area where the terminal is located is s, s' represents the new environmental parameters after the terminal executes the communication parameter adjustment strategy a, ΔQ(s,a,s') represents the communication quality index, ΔE(s,a,s') represents the network congestion index, ΔC(s,a,s') represents the communication cost index, ω1, ω2, and ω3 represent the weights corresponding to ΔQ(s,a,s'), ΔE(s,a,s'), and ΔC(s,a,s'), respectively.

[0065] Specifically, the expression of the above communication quality indicator ΔQ(s,a,s') is as follows:

[0066]

[0067] Among them, the above P old and P new Respectively represent the signal strength before and after the communication parameters are adjusted. The above N old and N new They represent the signal-to-noise ratio before and after the communication parameters are adjusted, respectively, w1 and w2 represent the weights corresponding to the signal strength and the signal-to-noise ratio, respectively;

[0068] Specifically, the expression of the above network congestion index ΔE(s,a,s') is as follows:

[0069]

[0070] Among them, the above D old and Dnew They represent the network transmission delay before and after the communication parameters are adjusted. The above L new and L old They represent the data packet loss rates before and after the communication parameters are adjusted, respectively. w3 and w4 represent the weights corresponding to the network transmission delay and the data packet loss rate, respectively.

[0071] Specifically, the expression of the communication cost indicator ΔC(s,a,s') is as follows:

[0072]

[0073] Among them, the above U old and U new They represent the communication resource consumption before and after the communication parameters are adjusted, which refers to the communication resource consumption of the ground or satellite communication equipment that needs to be dispatched when transmitting data before and after the terminal adjusts the communication parameters; the above S new and S old They respectively represent the energy costs before and after the communication parameters are adjusted, which refers to the charging cost required to fully charge the current power of the terminal before and after the communication parameters are adjusted; w5 and w6 respectively represent the weights corresponding to the communication resource consumption and energy cost.

[0074] In addition, the above experience pool may be a SumTree structure, and the samples in the experience pool are leaf nodes in the SumTree.

[0075] Optionally, when storing samples in the experience pool, the samples are stored in the experience pool in a priority experience playback mode. Specifically, the priority order of the samples can be determined, and samples with higher priority can be stored in the experience pool first, thereby ensuring that samples with greater impact on model training can be replayed first later.

[0076] Optionally, when storing samples in the experience pool, the samples can also be stored in the experience pool in the way of experience playback. That is, first determine whether the capacity of the experience pool is full; if the capacity of the experience pool is not full, directly store the samples in the experience pool; if the capacity of the experience pool is full, based on the first-in-first-out principle, use new samples to replace the oldest old samples stored in the experience pool.

[0077] Among them, the types of reinforcement learning algorithms used in the present application when training the communication parameter adjustment decision model include but are not limited to: Q learning algorithm, deep Q network algorithm, policy gradient algorithm, actor critic algorithm, etc.

[0078] In the embodiments of the present application, there is no specific limitation on the selection of the reinforcement learning algorithm.

[0079] The following is an explanation of the training process of the above communication parameter adjustment decision model using the DQN algorithm:

[0080] Step 1: Considering that the state and action of the Q function in the general reinforcement learning algorithm have high-dimensional complex problems, a neural network is introduced in this application, wherein the neural network includes an estimation network and a target network, and the estimation network (usually using a convolutional layer or a fully connected layer) can continuously learn and iterate to update its own network parameters, while the target network can use the network parameters updated by the estimation network to update its own parameters. In this application, the estimation network (also called Q network) - Q(s, a; ω) is used as an approximator of the function Q(s, a).

[0081] Step 2: At each time step t, the agent can take an action a according to the state s and the strategy (usually a greedy strategy), and then enter a new state s t+1 And obtain the corresponding reward r through the above reward function t .

[0082] Step 3: During the training process, the experience tuple (s t ,a t ,r t ,s t+1 ) is used for sample training, wherein the sample can be stored in the experience replay buffer in accordance with the experience replay mode or the priority experience replay mode.

[0083] Among them, α represents the learning rate and γ represents the discount factor.

[0084] Step 4: Randomly extract a batch of experience tuples from the experience replay buffer for training the neural network.

[0085] Step 5: For each experience tuple, calculate the target Q value using the following formula:

[0086]

[0087] Where γ represents the discount factor, Indicates the selection of a new state s t+1 The action a with the maximum Q value t+1 .

[0088] Step 6: Use mean square error (MSE) as the loss function, so the error between the target Q value and the predicted Q value of the estimation network is used as the loss value:

[0089]

[0090] Among them, E s [·] represents mean square error (MSE), y t Represents the objective function.

[0091] Step 7: Update the parameters of the estimated network through an optimizer such as gradient descent or Adam to minimize the loss function:

[0092]

[0093] Step 8: Update the network weight parameters of the target network based on the network weight parameters of the estimated network.

[0094] Step 9: Repeat the above steps 2 to 8, stop training after a sufficient number of training times or when a predetermined performance indicator is reached, and finally use the trained target network as the final communication parameter adjustment decision model.

[0095] Therefore, by analyzing the environmental parameters of the terminal through the above-mentioned communication parameter adjustment decision model, the optimal communication parameter adjustment strategy with the highest communication quality index, the lowest network congestion index and the smallest communication cost index can be obtained, so that the communication system can adjust the communication parameters of the target terminal according to the determined optimal communication parameter adjustment strategy, including but not limited to: communication frequency, communication power, data transmission rate, communication mode, etc., wherein the above-mentioned communication mode includes: ground communication or satellite communication.

[0096] For example, when the initial communication mode of the target terminal is terrestrial communication, the communication frequency is 900MHz, and the data transmission rate is 80kbps, the communication parameter adjustment decision model in this application is used to analyze the above parameters, and it is determined to continue to maintain the communication mode of the target terminal as terrestrial communication, and adjust the communication frequency to 2.4GHz, and the data transmission rate to 500Mbps. Alternatively, when the initial communication mode of the target terminal is terrestrial communication, the communication frequency is 900MHz, and the data transmission rate is 10kbps, the communication parameter adjustment decision model in this application is used to analyze the above parameters, and it is determined to switch the communication mode of the target terminal from terrestrial communication to satellite communication.

[0097] Through the above technical solution, the communication system can continuously detect the communication status parameters of the terminal in the current area and analyze the communication status parameters to decide on the communication parameter adjustment strategy that meets the requirements of various communication indicators, including switching of communication modes and / or adjusting parameters related to the communication mode, and scheduling actual communication resources, including communication resources of ground communication equipment or communication resources of satellite communication equipment, to ensure that the selected communication mode can be executed sequentially. Therefore, the present application solution can comprehensively consider factors such as communication quality, network congestion, and communication cost, and adjust the communication parameters in real time to adapt to the changing communication environment, thereby better improving the adaptive ability and performance optimization ability of the communication system, and thus improving user experience and communication service quality.

[0098] Example 2

[0099] Based on Embodiment 1 of the present application, an embodiment of a communication parameter adjustment device is also provided, and the communication parameter adjustment method of the above embodiment is executed when the device is running. Figure 3 is a schematic diagram of the structure of an optional communication parameter adjustment device according to an embodiment of the present application, such as Figure 3 As shown, the communication parameter adjustment device at least includes an acquisition module 31, a determination module 32 and an adjustment module 3, wherein:

[0100] The acquisition module 31 is used to acquire the environmental parameters of the area where the target terminal is located and the communication parameters of the target terminal in the area.

[0101] Among them, the above environmental parameters include but are not limited to: communication channel characteristic parameters, mobile state parameters, energy state parameters, communication resource parameters, weather conditions, etc. Specifically, the above communication channel characteristic parameters include but are not limited to: signal strength (represented by P), signal-to-noise ratio (represented by N), network transmission delay (represented by D), data packet loss rate (represented by L); the above mobile state parameters include but are not limited to: location coordinates, mobile speed, mobile acceleration; the above energy state parameters include but are not limited to: current power, remaining available time; the above communication resource parameters include: the amount of communication resources using ground communication equipment or the amount of communication resources using satellite communication equipment; the above weather conditions refer to the real-time weather conditions of the target terminal's current environment (such as sunny, cloudy, rainy, etc.).

[0102] The above communication parameters include at least one of the following: communication frequency, communication power, data transmission rate, and communication mode, and the communication mode includes: ground communication or satellite communication.

[0103] The determination module 32 is used to analyze the environmental parameters using a pre-trained communication parameter adjustment decision model to obtain a communication parameter adjustment strategy that makes multiple communication indicators meet the preset indicator requirements, wherein the communication parameter adjustment decision model is trained based on a preset reinforcement learning algorithm, and the communication indicators include at least: communication quality indicator, network congestion indicator, and communication cost indicator.

[0104] The training process of the above communication parameter adjustment decision model includes the following steps:

[0105] Step 1: Build an initial decision model and determine the experience pool, where the experience pool includes multiple samples, and each sample has a different priority;

[0106] Step 2: looping through the following steps until a preset number of iterations is reached, and obtaining a communication parameter adjustment decision model for the communication mode;

[0107] Step 3: In each iteration process, the initial environmental parameters are obtained; the initial environmental parameters are analyzed using a reinforcement learning algorithm, and the corresponding second communication mode communication parameter adjustment strategy is determined from a plurality of communication mode communication parameter adjustment strategies, and the reward corresponding to the second communication mode communication parameter adjustment strategy is determined using a preset reward function, and the initial environmental parameters are updated to obtain new environmental parameters; the initial environmental parameters, the new environmental parameters, the second communication mode communication parameter adjustment strategy and the corresponding rewards are stored as samples in an experience pool; and the model parameters of the initial decision model are continuously adjusted based on the samples in the experience pool.

[0108] Specifically, the expression of the above reward function is as follows:

[0109] R(s,a,s')=ω1*ΔQ(s,a,s')+ω2*ΔE(s,a,s')-ω3*ΔC(s,a,s')

[0110] Among them, s represents the environmental parameters of the area where the terminal is located, a represents the communication parameter adjustment strategy adopted when the environmental parameters of the area where the terminal is located is s, s' represents the new environmental parameters after executing the communication parameter adjustment strategy a, ΔQ(s,a,s') represents the communication quality index, ΔE(s,a,s') represents the network congestion index, ΔC(s,a,s') represents the communication cost index, ω1, ω2, and ω3 represent the weights corresponding to ΔQ(s,a,s'), ΔE(s,a,s'), and ΔC(s,a,s'), respectively.

[0111] Specifically, the expression of the above communication quality indicator ΔQ(s,a,s') is as follows:

[0112]

[0113] Among them, the above P old and P new Respectively represent the signal strength before and after the communication parameters are adjusted. The above N old and N new They represent the signal-to-noise ratio before and after the communication parameters are adjusted, respectively, w1 and w2 represent the weights corresponding to the signal strength and the signal-to-noise ratio, respectively;

[0114] Specifically, the expression of the above network congestion index ΔE(s,a,s') is as follows:

[0115]

[0116] Among them, the above D old and D new They represent the network transmission delay before and after the communication parameters are adjusted. The above L new and L oldThey represent the data packet loss rates before and after the communication parameters are adjusted, respectively. w3 and w4 represent the weights corresponding to the network transmission delay and the data packet loss rate, respectively.

[0117] Specifically, the expression of the communication cost indicator ΔC(s,a,s') is as follows:

[0118]

[0119] Among them, the above U old and U new They represent the communication resource consumption before and after the communication parameters are adjusted, which refers to the communication resource consumption of the ground or satellite communication equipment that needs to be dispatched when transmitting data before and after the terminal adjusts the communication parameters; the above S new and S old They respectively represent the energy costs before and after the communication parameters are adjusted, which refers to the charging cost required to fully charge the current power of the terminal before and after the communication parameters are adjusted; w5 and w6 respectively represent the weights corresponding to the communication resource consumption and energy cost.

[0120] In addition, the above experience pool may be a SumTree structure, and the samples in the experience pool are leaf nodes in the SumTree.

[0121] Optionally, when storing samples in the experience pool, the samples are stored in the experience pool in a priority experience playback mode. Specifically, the priority order of the samples can be determined, and samples with higher priority can be stored in the experience pool first, thereby ensuring that samples with greater impact on model training can be replayed first later.

[0122] Optionally, when storing samples in the experience pool, the samples can also be stored in the experience pool in the way of experience playback. That is, first determine whether the capacity of the experience pool is full; if the capacity of the experience pool is not full, directly store the samples in the experience pool; if the capacity of the experience pool is full, based on the first-in-first-out principle, use new samples to replace the oldest old samples stored in the experience pool.

[0123] Among them, the types of reinforcement learning algorithms used in the present application when training the communication parameter adjustment decision model include but are not limited to: Q-learning algorithm, Deep Q-learning algorithm, Actor-Critic algorithm, etc. There is no specific restriction on the selection of reinforcement learning algorithm in the embodiments of the present application.

[0124] The adjustment module 33 is used to adjust the communication parameters of the target terminal according to the communication parameter adjustment strategy.

[0125] The above-mentioned device continuously detects the communication state parameters of the terminal in the current communication environment and analyzes the communication state parameters to decide on the communication parameter adjustment strategy, that is, switching the communication mode and / or adjusting the parameters related to the current communication mode, and scheduling the actual communication resources, including the communication resources of the ground communication equipment or the communication resources of the satellite communication equipment, to ensure that the selected communication mode can be executed sequentially. Therefore, the present application scheme can comprehensively consider factors such as communication quality, network congestion, and communication cost, and adjust the communication parameters of the terminal in real time to adapt to the changing communication environment, thereby better improving the adaptive ability and performance optimization ability of the communication system, and then improving the user experience and communication service quality.

[0126] It should be noted that the various modules in the above-mentioned communication parameter adjustment device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0127] Example 3

[0128] According to an embodiment of the present application, a non-volatile storage medium is further provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the communication parameter adjustment method in Example 1.

[0129] Optionally, the device where the non-volatile storage medium is located implements the following steps by running the program:

[0130] Step S1, obtaining environmental parameters of an area where a target terminal is located and communication parameters of the target terminal in the area;

[0131] Step S2, using a pre-trained communication parameter adjustment decision model to analyze environmental parameters, and obtain a communication parameter adjustment strategy that enables multiple communication indicators of the network to meet preset indicator requirements, wherein the communication parameter adjustment decision model is obtained by training based on a preset reinforcement learning algorithm, and the communication indicators include at least: communication quality indicator, network congestion indicator, and communication cost indicator;

[0132] Step S3: adjusting the communication parameters of the target terminal according to the communication parameter adjustment strategy.

[0133] According to an embodiment of the present application, a computer program product is also provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the communication parameter adjustment method in Example 1 is implemented.

[0134] Optionally, the computer program executes the following steps:

[0135] Step S1, obtaining environmental parameters of an area where a target terminal is located and communication parameters of the target terminal in the area;

[0136] Step S2, using a pre-trained communication parameter adjustment decision model to analyze environmental parameters, and obtain a communication parameter adjustment strategy that enables multiple communication indicators of the network to meet preset indicator requirements, wherein the communication parameter adjustment decision model is obtained by training based on a preset reinforcement learning algorithm, and the communication indicators include at least: communication quality indicator, network congestion indicator, and communication cost indicator;

[0137] Step S3: adjusting the communication parameters of the target terminal according to the communication parameter adjustment strategy.

[0138] According to an embodiment of the present application, a processor is also provided, which is used to run a program, wherein the communication parameter adjustment method in Example 1 is executed when the program is running.

[0139] Optionally, the following steps are performed when the program is running:

[0140] Step S1, obtaining environmental parameters of an area where a target terminal is located and communication parameters of the target terminal in the area;

[0141] Step S2, using a pre-trained communication parameter adjustment decision model to analyze environmental parameters, and obtain a communication parameter adjustment strategy that enables multiple communication indicators of the network to meet preset indicator requirements, wherein the communication parameter adjustment decision model is obtained by training based on a preset reinforcement learning algorithm, and the communication indicators include at least: communication quality indicator, network congestion indicator, and communication cost indicator;

[0142] Step S3: adjusting the communication parameters of the target terminal according to the communication parameter adjustment strategy.

[0143] According to an embodiment of the present application, a network device is also provided, wherein: Figure 4 is a schematic diagram of the structure of an optional network device according to an embodiment of the present application, such as Figure 4 As shown, the network device includes one or more processors; a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the communication parameter adjustment method in the above-mentioned embodiment 1 when running.

[0144] Step S1, obtaining environmental parameters of an area where a target terminal is located and communication parameters of the target terminal in the area;

[0145] Step S2, using a pre-trained communication parameter adjustment decision model to analyze environmental parameters, and obtain a communication parameter adjustment strategy that enables multiple communication indicators of the network to meet preset indicator requirements, wherein the communication parameter adjustment decision model is obtained by training based on a preset reinforcement learning algorithm, and the communication indicators include at least: communication quality indicator, network congestion indicator, and communication cost indicator;

[0146] Step S3: adjusting the communication parameters of the target terminal according to the communication parameter adjustment strategy.

[0147] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0148] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0150] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0151] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0153] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A communication parameter adjustment method, characterized in that: include: Acquire environmental parameters of an area where a target terminal is located and communication parameters of the target terminal in the area; The environmental parameters are analyzed using a pre-trained communication parameter adjustment decision model to obtain a communication parameter adjustment strategy that meets the index requirements of multiple communication indicators, wherein the communication parameter adjustment decision model is obtained by training based on a preset reinforcement learning algorithm, and the communication indicators at least include: communication quality indicators, network congestion indicators, and communication cost indicators; The communication parameters are adjusted according to the communication parameter adjustment strategy.

2. The method according to claim 1, characterized in that The environmental parameters include at least one of the following: communication channel characteristic parameters, mobile state parameters, energy state parameters, and communication resource parameters, wherein: The communication channel characteristic parameters include at least one of the following: signal strength, signal-to-noise ratio, network transmission delay, and data packet loss rate; The movement state parameter includes at least one of the following: position coordinates, movement speed, and movement acceleration; The energy status parameter includes at least one of the following: current power, remaining available time; The communication resource parameters include: the amount of communication resources using ground communication equipment or the amount of communication resources using satellite communication equipment.

3. The method according to claim 1, characterized in that The type of the reinforcement learning algorithm includes at least one of the following: Q learning algorithm, deep Q network algorithm, actor critic algorithm.

4. The method according to claim 1, characterized in that: The training process of the communication parameter adjustment decision model includes: Construct an initial decision model and initialize the model parameters of the initial decision model; Determine the experience pool and initialize the priority of each sample in the experience pool; The following steps are executed repeatedly until a preset number of iterations is reached to obtain the communication parameter adjustment decision model; In each iteration process, the environmental parameters are initialized; the current environmental parameters are analyzed using the reinforcement learning algorithm, a corresponding second communication parameter adjustment strategy is determined from a plurality of communication parameter adjustment strategies, and a reward corresponding to the second communication parameter adjustment strategy is determined using a preset reward function, and the current environmental parameters are updated to obtain new environmental parameters; the current environmental parameters, the new environmental parameters, the second communication parameter adjustment strategy and the corresponding reward are used as samples, and the samples are stored in the experience pool; The model parameters of the initial decision model are continuously adjusted based on each sample in the experience pool.

5. The method according to claim 4, characterized in that The expression of the reward function is as follows: R(s,a,s')=ω1*ΔQ(s,a,s')+ω2*ΔE(s,a,s')-ω3*ΔC(s,a,s') Wherein, s represents the environmental parameter of the area where the terminal is located, a represents the communication parameter adjustment strategy adopted when the environmental parameter of the area where the terminal is located is s, s' represents the new environmental parameter after executing the communication parameter adjustment strategy a, ΔQ(s,a,s') represents the communication quality index, ΔE(s,a,s') represents the network congestion index, ΔC(s,a,s') represents the communication cost index, ω1, ω2, ω3 represent the weights corresponding to ΔQ(s,a,s'), ΔE(s,a,s'), and ΔC(s,a,s'), respectively; The expression of the communication quality indicator ΔQ(s,a,s') is as follows: P old and P new Respectively represent the signal strength before and after the communication parameters are adjusted, N old and N new They represent the signal-to-noise ratio before and after the communication parameters are adjusted, respectively, w1 and w2 represent the weights corresponding to the signal strength and the signal-to-noise ratio, respectively; The expression of the network congestion index ΔE(s,a,s') is as follows: D old and D new They represent the network transmission delay before and after the communication parameters are adjusted, L new and L old They represent the data packet loss rate before and after the communication parameters are adjusted, w3 and w4 represent the weights corresponding to the network transmission delay and the data packet loss rate, respectively; wherein the expression of the communication cost indicator ΔC(s,a,s') is as follows: U old and U new They represent the communication resource consumption before and after the communication parameter adjustment; S new and S old They represent the energy costs before and after the communication parameters are adjusted respectively; w5 and w6 represent the weights corresponding to the communication resource consumption and energy cost respectively.

6. The method according to claim 4, characterized in that Storing the sample in the experience pool includes: The samples are stored in the experience pool according to a priority experience playback mode or an experience playback mode.

7. The method according to claim 1, characterized in that The communication parameters include at least one of the following: communication frequency, communication power, data transmission rate, and communication mode, wherein the communication mode includes: terrestrial communication or satellite communication.

8. A communication parameter adjustment device, characterized in that: include: An acquisition module, used to acquire environmental parameters of an area where a target terminal is located and communication parameters of the target terminal in the area; A determination module, configured to analyze the environmental parameters using a pre-trained communication parameter adjustment decision model to obtain a communication parameter adjustment strategy that meets the indicator requirements of multiple communication indicators, wherein the communication parameter adjustment decision model is obtained by training based on a preset reinforcement learning algorithm, and the communication indicators at least include: a communication quality indicator, a network congestion indicator, and a communication cost indicator; The adjustment module is used to adjust the communication parameters according to the communication parameter adjustment strategy.

9. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the communication parameter adjustment method according to any one of claims 1 to 7 is implemented.

10. A network device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the communication parameter adjustment method according to any one of claims 1 to 7 is executed when the program is run.

Images