A method for controlling radio communication equipment

Abstract

The present application discloses a method for controlling a radio communication device, comprising: obtaining device data of a radio communication device; determining a priority of device calibration; collecting environmental data; a cloud server generating device calibration data through a machine learning algorithm according to the device data, environmental data, the priority of device calibration, and historical data; comparing the collected environmental data with a received calibration trigger threshold; when the collected environmental data reaches the correction trigger threshold, generating dynamic calibration parameters of the current environment through a machine learning algorithm according to a data weight parameter; calibrating the working parameters of the device according to the generated dynamic calibration parameters using an Actor‑Critic reinforcement learning framework; outputting the calibrated working parameters to control the transmission and reception of the radio communication device.

Description

Technical Field

[0001] The present application relates to the field of radio communications, and in particular to a method for controlling radio communication equipment. Background Art

[0002] With the rapid development of wireless communication technology, radio communication equipment is increasingly used in various environments. However, existing radio communication equipment has certain limitations in adaptability and performance when facing complex and changing communication environments. This is mainly because traditional communication equipment control methods are often adjusted based on static preset parameters or simple rules, which cannot fully adapt to dynamically changing communication environments.

[0003] In practical applications, radio communication equipment often faces complex communication environment challenges such as channel fading, multipath effects, electromagnetic interference, etc. Traditional static parameter adjustment methods often cannot effectively cope with these challenges, resulting in problems such as communication quality degradation, signal loss, and data transmission errors, thus affecting the stability and reliability of the communication system.

[0004] In the related art, for example, Chinese patent document CN117134840B provides a radio communication device control method and a radio communication device, which performs basic detection on the radio communication device. If the basic detection is passed, the radio communication device enters operation, collects environmental data of the environment where the radio communication device is located through an environmental data acquisition device, and compares the collected environmental data with the standard environmental data stored in the data storage module in the radio communication device to obtain the data difference of each data. If there is a data difference that is not less than the preset calibration trigger difference, the calibration module performs calibration and debugging on the calibration item according to the obtained calibration debugging strategy. After the debugging is completed, the radio communication device obtains the radio transceiver event. If no radio event is generated within the set time, the radio communication device enters sleep and completes the radio device control. However, this method uses a preset calibration trigger difference to determine whether to calibrate and debug the device, and this preset trigger difference may not be able to fully adapt to changes in various complex environments. The complexity of the communication environment may cause slight fluctuations or changes in the performance of the device, but due to the large setting of the trigger difference, these slight fluctuations may be ignored or fail to be calibrated in time, thereby reducing the device's ability to adapt to environmental changes. Summary of the invention

[0005] In response to the problem in the prior art that radio communication equipment has low adaptability to complex and changeable communication environments, the present application provides a radio communication equipment control method, which uses a cloud server to construct nonlinear mapping relationships and search for global optimal calibration parameters, and the radio communication equipment performs real-time environmental perception and Actor-Critic reinforcement learning calibration, thereby improving adaptability.

[0006] The purpose of this application is achieved through the following technical solutions.

[0007] The present specification provides a radio communication device control method, including: S1, obtaining device data of the radio communication device, the device data including antenna standing wave ratio and frequency error; uploading the obtained device data to a cloud server; after receiving the device data, the cloud server compares the antenna standing wave ratio and frequency error with preset thresholds respectively; when the antenna standing wave ratio or frequency error exceeds the threshold range, the device is judged to be in an unhealthy state; when the device is in an unhealthy state, a fuzzy reasoning algorithm is used to calculate the device health quantitative index according to the degree of deviation between the antenna standing wave ratio and the threshold, and the degree of deviation between the frequency error and the threshold; the quantitative index is mapped to a preset calibration priority level to determine the priority of device calibration; wherein, antenna standing wave ratio (Voltage Standing Wave Ratio, VSWR): a quantitative index indicating the degree of matching between the antenna load and the feeder impedance. When the antenna and feeder impedance are completely matched, the VSWR is 1; when there is an impedance mismatch, the VSWR is greater than 1. The higher the VSWR, the more serious the impedance mismatch and the lower the antenna efficiency. Frequency error: indicates the deviation between the actual operating frequency of the radio communication device and the preset frequency. Frequency error is usually caused by the frequency drift of the device's internal oscillator or external environmental interference. Excessive frequency error can lead to signal distortion, reduced communication quality and other problems.

[0008] Fuzzy Inference Algorithm: A reasoning method based on fuzzy set theory and fuzzy logic. By mapping the input variables to fuzzy sets and reasoning according to predefined fuzzy rules, fuzzy outputs are obtained, and then the deterministic results are obtained through defuzzification. Fuzzy inference algorithms are suitable for dealing with uncertainty and nonlinear problems. Quantitative indicators of equipment health: quantitative measures of the health status of radio communication equipment. By comparing the key performance indicators of the equipment (such as antenna standing wave ratio, frequency error, etc.) with the preset thresholds and comprehensively considering the deviation degree of each indicator, a quantitative health indicator is calculated. This indicator reflects the overall health level of the equipment. Deviation degree: indicates the size of the gap between the equipment performance indicator and the preset threshold. The greater the deviation degree, the more abnormal the equipment performance and the lower the health. In the fuzzy inference algorithm, the deviation degree is used as a fuzzy variable and is mapped to the fuzzy set through the membership function.

[0009] Preferably, the radio communication equipment is powered on and starts, and performs a hardware self-test, which includes the detection of antenna standing wave ratio, transmission power, and frequency error; comparing the detection data with preset thresholds to identify potential fault points; starting a self-repair mechanism for identified common faults; uploading the self-test and self-repair result data to a cloud server; during the self-test process, the detection data is compared with a preset normal threshold to identify potential fault points; for common faults, the self-repair mechanism is started to attempt automatic repair; and the self-test and self-repair result data are uploaded to a cloud server.

[0010] S2, collects environmental data, which includes temperature, humidity and electromagnetic interference data; uploads the collected environmental data to the cloud server; the cloud server generates device calibration data through machine learning algorithms based on device data, environmental data, device calibration priority, and historical data; the calibration data includes calibration trigger thresholds and data weight parameters; among them, electromagnetic interference data: represents the electromagnetic interference information in the environment where the radio communication equipment is located. The sources of electromagnetic interference include other electronic devices, radio signals, etc., which will affect the communication performance of the equipment. By collecting electromagnetic interference data through sensors, the electromagnetic environment of the equipment can be evaluated to provide a basis for dynamic calibration.

[0011] S3, the radio communication device compares the collected environmental data with the received calibration trigger threshold; when the collected environmental data reaches the correction trigger threshold, the dynamic calibration parameters of the current environment are generated by the machine learning algorithm according to the data weight parameter; wherein the machine learning algorithm is a type of computer algorithm that can improve performance through data learning. Common machine learning algorithms include supervised learning, unsupervised learning, reinforcement learning, etc. In this application, the machine learning algorithm is used to generate equipment calibration parameters and control strategies based on historical data and current environmental data. When the collected environmental data reaches the correction trigger threshold: it means that the environmental data such as temperature, humidity, electromagnetic interference, etc. collected by the radio communication device reaches the preset calibration trigger condition. When the environmental data exceeds the normal range and may affect the performance of the device, the dynamic calibration process is triggered to generate calibration parameters adapted to the current environment. Dynamic calibration parameters: calibration parameters generated in real time according to the current environmental conditions of the device. Unlike static or preset calibration parameters, dynamic calibration parameters can adaptively adjust the working state of the device to cope with environmental changes. Common dynamic calibration parameters include transmit power, receiving sensitivity, etc.

[0012] S4, according to the generated dynamic calibration parameters, the Actor-Critic reinforcement learning framework is used to calibrate the working parameters of the device, and the working parameters include transmission power and receiving sensitivity; the calibrated working parameters are output to control the transmission and reception of the radio communication equipment. Preferably, the cloud server uses big data analysis to mine the working rules and abnormal modes of the equipment, and generates fault warning prompts in time; the equipment enters a dormant state when there is no work task for a long time, and the dormant time is dynamically adjusted according to the equipment usage rules calculated by the cloud; when the cloud recognizes that a special event has occurred, it issues a forced wake-up instruction, and the device immediately wakes up and enters the working state; when the device has no work task for a long time, it enters a dormant state, and the dormant time is dynamically adjusted according to the equipment usage rules calculated by the cloud; when the cloud recognizes that a special event has occurred and issues a forced wake-up instruction, the device is immediately awakened and enters the working state. Among them, the Actor-Critic reinforcement learning framework: a reinforcement learning method that combines policy gradient (Actor) and value function estimation (Critic). The Actor network generates an action (such as a calibration parameter) according to the current state, and the Critic network estimates the long-term value of the action. Through the interactive training of the two networks, the strategy is continuously optimized to achieve adaptive calibration of the equipment working parameters. The Actor-Critic framework takes into account both the exploration of strategies and the accuracy of value estimation, and is suitable for control problems in continuous action spaces.

[0013] Further, S1, determines the priority of equipment calibration, including: when the equipment is in an unhealthy state, using a fuzzy reasoning algorithm, taking antenna standing wave ratio data, antenna standing wave ratio health threshold, frequency error data and frequency error health threshold as input variables of fuzzy reasoning; by constructing a membership function, mapping the input variables to corresponding fuzzy sets, calculating the degree of deviation of the antenna standing wave ratio data relative to the antenna standing wave ratio health threshold, and the degree of deviation of the frequency error relative to the frequency health threshold; according to the degree of deviation of the antenna standing wave ratio and the degree of deviation of the frequency error, fuzzy reasoning is performed through fuzzy reasoning rules to obtain a fuzzy value representing the health state of the equipment; fuzzy The inference rules include fuzzy rules in the form of IF-THEN, which are used to describe the fuzzy relationship between the degree of deviation of the antenna standing wave ratio and the degree of deviation of the frequency error and the health status of the equipment; the fuzzy value is defuzzified to obtain a quantitative index of the equipment health; the quantitative index is compared with a preset calibration priority level threshold, and the quantitative index is mapped to the corresponding calibration priority level according to the comparison result to obtain the equipment calibration priority; wherein the calibration priority level is a plurality of preset levels corresponding to the health of the equipment, each level corresponds to a range interval of a health quantification indicator, and the health quantification indicator is mapped to the corresponding priority level if it is within the interval.

[0014] Among them, fuzzy value: the intermediate result obtained in the fuzzy reasoning process, which indicates the degree of fuzziness of the health status of the equipment. Fuzzy value is a real number between 0 and 1, which reflects the degree of membership of the equipment status to a certain fuzzy set. The closer the fuzzy value is to 1, the more the equipment status conforms to the description of the fuzzy set; the closer the fuzzy value is to 0, the less the equipment status conforms to the description of the fuzzy set. Fuzzy inference rules: statements in the form of IF-THEN used to describe the fuzzy relationship between input variables and output variables. Fuzzy inference rules are composed of antecedents (IF part) and consequents (THEN part). The antecedents describe the fuzzy set to which the input variables belong, and the consequents describe the fuzzy set to which the corresponding output variables belong. For example: "IF the deviation degree of antenna standing wave ratio is high AND the deviation degree of frequency error is high THEN the health status of the equipment is poor". Fuzzy inference rules express expert knowledge and experience in language form and are the core of fuzzy reasoning.

[0015] Defuzzification: The process of converting fuzzy values ​​obtained by fuzzy reasoning into deterministic numerical values. Since fuzzy values ​​represent the degree of fuzziness of the health status of the equipment, they cannot be directly used for quantitative analysis and decision-making. Defuzzification is required to obtain specific health indicators. Common defuzzification methods include the centroid method, the maximum membership method, and the weighted average method. Defuzzification is the last step of fuzzy reasoning, which maps the fuzzy results back to the precise numerical domain. Quantitative Index: An accurate numerical measure of the health of the equipment obtained through defuzzification. The quantitative index is a real number within a preset range that intuitively reflects the health level of the equipment. The quantitative index is compared with the preset calibration priority level threshold, and the priority of equipment calibration is determined based on the comparison result. The quantitative index provides a universal measure of the health status of the equipment, which is convenient for priority sorting and the formulation of calibration strategies.

[0016] Furthermore, the expression of the constructed membership function is as follows:

[0017]

[0018] Among them, the center point c, the symmetric center of the function, represents the midpoint of the platform area. Position parameter λ: controls the width of the platform area. The range of the platform area is [c-λ, c+λ], and the membership degree is always 1 within this range. The left scale parameter η L : The width of the non-platform area on the left side of the control function, η L The larger the value, the wider the non-platform area on the left side of the function, indicating that the input variable has a higher degree of membership in a larger range to the left of the center point. R : The width of the non-platform area on the right side of the control function, ηR The larger the value, the wider the non-platform area on the right side of the function, indicating that the input variable has a higher degree of membership in a larger range to the right of the center point. L : Controls the shape of the non-platform region on the left side of the function, β L The larger the value, the steeper the curve of the non-platform area on the left side of the function, indicating that the membership of the input variable on the left side of the center point changes faster. R : Controls the shape of the non-platform region on the right side of the function, β R The larger the value, the steeper the curve in the non-platform area on the right side of the function, indicating that the membership of the input variable to the right of the center point changes faster.

[0019] By introducing the location parameter λ, a platform area is formed near the center point of the function, so that the function can better describe the situation that some input variables have the same membership within a certain range. By introducing an asymmetric factor, by setting different left and right parameters, the width and shape of the left and right sides of the function can be independently controlled, making the function asymmetric. This asymmetry can better describe the membership distribution of some input variables. For example, when the membership of an input variable changes faster on one side and slower on the other side, an asymmetric membership function can more accurately describe this feature. By introducing the location parameter λ, a platform area is formed near the center point of the function, and the membership is always 1 in the platform area. This feature can better describe that some input variables have the same membership within a certain range. For example, when the input variables are considered to belong to a fuzzy set completely within a certain range, a membership function with a platform area can more accurately describe this situation.

[0020] Further, S2 generates device calibration data, including: inputting the collected environmental data into a pre-trained multi-layer autoencoder network, performing feature extraction and dimensionality reduction processing on the environmental data through the multi-layer autoencoder network, and obtaining quantitative indicators representing the environmental state; the radio communication device uploads the obtained quantitative indicators to the cloud server through the communication link; after receiving the quantitative indicators, the cloud server uses the quantitative indicators, the device data in S1, and the historical data stored in the cloud service as input, and obtains the nonlinear mapping relationship between environmental factors and device performance through a pre-built bidirectional long short-term memory network; based on the obtained nonlinear mapping relationship, the cloud service takes the device performance indicators as the optimization target, combines the Bayesian optimization algorithm to search for the optimal calibration parameter combination, and generates device calibration data corresponding to the current environmental data; wherein the device performance indicators include signal-to-noise ratio, bit error rate, data throughput or power efficiency.

[0021] Among them, the stacked auto encoder (SAE) is a neural network model for unsupervised learning, which is composed of multiple auto encoders (Auto Encoders) stacked in layers. The auto encoder is a symmetrical three-layer neural network consisting of an input layer, a hidden layer (encoding layer) and an output layer (decoding layer). The auto encoder encodes the input data into a low-dimensional representation, and then decodes and reconstructs the original data to achieve data dimension reduction and feature extraction. The multi-layer auto encoder network stacks multiple auto encoders to learn the high-order features of the data layer by layer to extract more abstract and essential feature representations. In this application, the multi-layer auto encoder network is used to extract features and reduce the dimension of environmental data to obtain quantitative indicators that characterize the environmental state. Quantitative Index: A numerical index obtained by extracting features and reducing the dimension of environmental data through a multi-layer auto encoder network. The quantitative index represents the key features of the environmental state in the form of a low-dimensional vector, which condenses the information of the original environmental data. By using quantitative indicators instead of original data, data transmission and storage overhead can be reduced and processing efficiency can be improved. At the same time, quantitative indicators, as inputs of the nonlinear mapping relationship between device performance and environmental factors, help simplify modeling complexity and improve the generalization ability of the model. Bayesian Optimization Algorithm: A global optimization algorithm based on Bayesian theory, often used to solve complex black-box optimization problems. Bayesian optimization constructs a probabilistic model of the objective function (such as a Gaussian process), and updates the posterior probability distribution according to the evaluated sample points in each iteration, and selects the next candidate point that is most likely to improve the optimization result for evaluation. Compared with traditional optimization algorithms, Bayesian optimization performs well on high-dimensional, non-convex, and computationally expensive problems, and can find the global optimal solution in a smaller number of evaluations. In this application, the Bayesian optimization algorithm is used to search for the optimal combination of calibration parameters, evaluate the pros and cons of candidate parameters through device performance indicators, and select parameters that can maximize device performance as calibration data.

[0022] Furthermore, the cloud service searches for the optimal calibration parameter combination based on the acquired nonlinear mapping relationship and takes the device performance index as the target, and combines the Bayesian optimization algorithm to generate the device calibration data corresponding to the current environmental data, including: taking the acquired nonlinear mapping relationship as the objective function of the optimization problem, taking the signal-to-noise ratio, bit error rate, data throughput or power efficiency as the optimization target, and taking the calibration parameter combination to be optimized as the optimization variable to construct a parameter optimization model; based on the constructed parameter optimization model, constructing a probability distribution proxy model of the parameter space through Gaussian process regression; balancing the exploration and utilization of parameters by setting an acquisition function based on the confidence interval upper confidence boundary algorithm; wherein the confidence interval upper confidence boundary algorithm adopts a weight adjustment mechanism for the exploration factor and the utilization factor; taking the parameter optimization model, the probability distribution proxy model and the acquisition function as input, iteratively searching the parameter space to obtain the globally optimal calibration parameter combination; taking the acquired globally optimal calibration parameter combination as the device calibration data under the current environmental state, and generating calibration data including the calibration trigger threshold and data weight parameters.

[0023] Among them, Gaussian Process Regression (GPR): a non-parametric Bayesian machine learning method used to construct a probability distribution proxy model of the objective function of the optimization problem. A Gaussian process is a set of random variables, where the joint distribution of any finite number of random variables obeys a multivariate Gaussian distribution. Gaussian process regression predicts the probability distribution of the objective function at unevaluated points by learning the mean and covariance functions of the evaluated sample points. Compared with traditional parametric regression methods, Gaussian process regression can model functions of arbitrary complexity and provide uncertainty measures for prediction results. In Bayesian optimization, Gaussian process regression is used as a surrogate model of the objective function to guide the selection of the next evaluation point. Probabilistic Surrogate Model: a probabilistic model used to approximate the objective function of a black-box optimization problem. Since the true objective function is usually computationally expensive or has no analytical expression, directly optimizing the objective function is often very costly. The probability distribution surrogate model constructs a probabilistic approximation of the objective function by learning the distribution law of the evaluated sample points, which is used to predict the distribution of function values ​​at unevaluated points. Common probability distribution proxy models include Gaussian process, random forest, neural network, etc. In Bayesian optimization, the probability distribution proxy model is combined with the acquisition function to search for the next candidate point that is most likely to improve the optimization result in the parameter space, balancing the trade-off between exploration and utilization. Upper Confidence Bound (UCB): A Bayesian optimization acquisition function for balancing exploration and utilization. The UCB algorithm constructs an upper bound of the confidence interval as the acquisition function by weighted summing the mean and variance predicted by the probability distribution proxy model. The UCB algorithm tends to select candidate points with high mean (utilization) or large variance (exploration), focuses on exploring unevaluated areas in the early stage of optimization, and gradually converges to the global optimal solution as the number of iterations increases. The UCB algorithm introduces a weight adjustment mechanism for the exploration factor and the utilization factor, and flexibly balances the proportion of exploration and utilization by controlling the relative size of the two factors. When the weight of the exploration factor is large, the algorithm is more inclined to select candidate points with high uncertainty; when the weight of the utilization factor is large, the algorithm is more inclined to select candidate points with high predicted mean.

[0024] Furthermore, obtaining the globally optimal calibration parameter combination includes: initializing the exploration factor weight and the utilization factor weight in the confidence interval upper confidence boundary algorithm, and setting the initial number of iterations; adjusting the exploration factor weight and the utilization factor weight according to the current number of iterations through a preset weight adjustment function; using the adjusted exploration factor weight, utilization factor weight, posterior mean and posterior standard deviation of the candidate parameter combination as input, and calculating the confidence interval upper bound of the candidate parameter combination through the confidence interval upper bound calculation formula; using the calculated confidence interval upper bound as input, and selecting the candidate parameter combination with the largest confidence interval upper bound as the next evaluation point; inputting the evaluation point into the objective function for evaluation to obtain the evaluation result; and using the evaluation result as input to update the Gaussian process regression model; using the updated Gaussian process regression model and the current number of iterations as input, and judging whether the iteration stop condition is met through a preset iteration stop condition; if so, outputting the globally optimal calibration parameter combination obtained by the iterative search as the optimal calibration parameter combination under the current environmental state; otherwise, adding 1 to the current number of iterations, and returning the adjusted exploration factor weight and utilization factor weight.

[0025] Among them, the posterior mean: in Bayesian optimization, the Gaussian process regression model is used to predict the expectation of the objective function value at the candidate parameter combination. The posterior mean comprehensively considers the information of the prior knowledge and the evaluated sample points, and represents the most likely objective function value of the candidate parameter combination under the current known information. The posterior mean is usually used to guide the use of the acquisition function and select candidate points with high predicted means for evaluation. In the confidence interval upper confidence boundary algorithm, the weighted sum of the posterior mean and the posterior standard deviation constitutes the upper bound of the confidence interval. The higher the posterior mean, the more likely the candidate parameter combination is to obtain a good optimization result. The posterior standard deviation: in Bayesian optimization, the uncertainty of the objective function value at the candidate parameter combination is predicted by the Gaussian process regression model. The posterior standard deviation measures the variability of the objective function value at the candidate parameter combination under the current known information. The larger the posterior standard deviation, the higher the uncertainty at the candidate parameter combination, that is, the less reliable the prediction of the objective function value. In the confidence interval upper confidence bound algorithm, the posterior standard deviation is multiplied by the exploration factor weight to encourage the algorithm to explore areas with high uncertainty. The larger the posterior standard deviation, the higher the upper bound of the confidence interval of the candidate parameter combination, and the more likely it is to be selected as the next evaluation point. By balancing the posterior mean and the posterior standard deviation, the confidence interval upper confidence bound algorithm achieves a trade-off between exploration and exploitation.

[0026] Furthermore, the upper bound of the confidence interval is calculated as follows:

[0027]

[0028] Where, μ(x): the posterior mean of the candidate parameter combination {x}, which represents the estimate of the expected optimization target value of the candidate parameter combination {x} based on the evaluated sample points at the current iteration. σ(x): the posterior standard deviation of the candidate parameter combination {x}, which represents the uncertainty of the estimate of the optimization target value of the candidate parameter combination {x} at the current iteration. α t The weight function of the exploration factor is used to control the degree of global exploration. t It decreases with the increase of iteration number t, making the algorithm more focused on local utilization in the later stage of optimization. For example, β t Represents the weight function of the utilization factor, which is used to control the degree of local utilization. t As the number of iterations t increases, the algorithm focuses more on searching the target area in the later stages of optimization. t =log(t+1). t Represents an additional exploration term used to encourage the algorithm to explore areas with high uncertainty. t can be a decreasing function related to the number of iterations t, for example, ε represents a small positive constant, which is used to prevent the denominator from being zero. t and β t , achieving a dynamic balance between exploration and utilization. In the early stage of optimization, α t Larger, β t Small, encouraging global exploration; in the later stages of optimization, α t Smaller, β t Larger, encouraging local utilization and improving convergence speed. Introducing additional exploration items Additional exploration is performed on areas with high uncertainty. When the posterior standard deviation σ(x) of the candidate parameter combination {x} is large, the value of this term is large, encouraging the algorithm to explore areas with high uncertainty; when σ(x) is small, the value of this term is small, reducing repeated searches of areas that have been fully explored. By adjusting α t , β t and γ t , the balance between exploration and exploitation, as well as the degree of additional exploration, can be flexibly controlled, so that the algorithm can be adaptively adjusted according to the characteristics and needs of specific problems, thereby improving the optimization performance.

[0029] Furthermore, S3 generates dynamic calibration parameters for the current environment, including: collecting environmental data in real time, and comparing the collected environmental data with the calibration trigger threshold received in S2; when the collected environmental data reaches the calibration trigger threshold, using the data weight parameters received in S3 as input, and fitting the nonlinear relationship between the data weight parameters and the environmental data through a pre-trained extreme learning machine network; based on the nonlinear relationship obtained by fitting, combined with the currently collected environmental data, generating dynamic calibration parameters for the current environment; wherein the dynamic calibration parameters are used to dynamically adjust the working parameters of the equipment.

[0030] Among them, the Extreme Learning Machine (ELM) network is a single hidden layer feedforward neural network with the advantages of fast training speed and strong generalization ability. Compared with traditional neural networks, ELM randomly generates the connection weights and biases of hidden layer nodes. It does not need to be iteratively optimized through the back-propagation algorithm, but only needs to solve the output weights according to the least squares method, which greatly improves the training efficiency.

[0031] Further, S4, the Actor-Critic reinforcement learning framework is used to calibrate the working parameters of the equipment, including: the radio communication equipment uses the Actor-Critic reinforcement learning framework to calibrate the working parameters such as the transmission power and the receiving sensitivity according to the dynamic calibration parameters generated in step S3; wherein the Actor-Critic reinforcement learning framework includes a policy network and a value function network, the policy network is used to generate the adjustment strategy of the working parameters, and the value function network is used to evaluate the pros and cons of the adjustment strategy; the radio communication equipment inputs the current equipment state and calibration parameters as observation values ​​into the policy network of the Actor network; the policy network adopts a deep neural network structure, learns the mapping relationship from the state to the action through network training, and generates the adjustment strategy of the working parameters according to the observation values; wherein the output of the policy network is the adjustment amount of the working parameters, which is modeled using Gaussian distribution; the radio communication equipment According to the working parameter adjustment amount generated by the policy network, the working parameters such as transmission power and receiving sensitivity are adjusted; the working parameters are controlled and adjusted through the equipment hardware circuit, and the adjusted equipment output performance indicators are collected; the radio communication equipment inputs the adjusted performance indicators and expected performance indicators as observation values ​​into the value function network of the Critic network; the value function network adopts a deep neural network structure, and learns the mapping relationship from the state to the value function through network training, and calculates the value of the adjustment strategy according to the observation value; among them, the value function is defined as the cumulative discounted sum of the differences between performance indicators, which is used to evaluate the long-term utility of the adjustment strategy; the policy value output by the value function network is used as the reward value of the Actor network and fed back to the Actor network; the Actor network adopts the policy gradient algorithm to update the parameters of the policy network according to the reward value, and optimizes the strategy towards a higher cumulative reward through gradient ascent.

[0032] Among them, the Monte Carlo method is used to perform unbiased sampling estimation of the gradient; through the interactive training of the Actor network and the Critic network, the iterative update of the policy network and the value function network parameters is realized; the adjustment strategy of the working parameters is continuously optimized so that the adjusted device output performance indicators gradually meet the expected performance requirements; the radio communication equipment uploads the adjusted working parameters and performance indicators and other information to the cloud server; the cloud server performs online learning and optimization of the Actor-Critic network model based on the received information; the federated learning framework is adopted to aggregate the data uploaded by multiple devices to optimize the global network model; at the same time, the active learning strategy is used to select sample data with high contribution to the model for training to improve the model learning efficiency; the cloud server sends the optimized Actor-Critic network model parameters to the radio communication equipment; the radio communication equipment updates the local Actor-Critic framework according to the received model parameters for dynamic calibration of the next round of working parameters; through cloud-edge collaboration, the continuous optimization and update of the Actor-Critic reinforcement learning framework is realized.

[0033] Furthermore, S4, the Actor-Critic reinforcement learning framework is used to calibrate the working parameters of the equipment, and also includes: outputting the iteratively optimized working parameters to control the transmission and reception of the radio communication equipment; uploading the optimized working parameters and corresponding performance indicators to the cloud server; the cloud server performs online learning and optimization of the Actor-Critic network based on the received data.

[0034] Among them, the Discounted Cumulative Sum is a method for evaluating the long-term utility of strategies in reinforcement learning. In reinforcement learning, the agent interacts with the environment, selects actions based on the current state, obtains immediate rewards, and moves to the next state. The Discounted Cumulative Sum is a weighted sum of the immediate rewards obtained by the agent in a certain state in the future as an estimate of the long-term value of the action.

[0035] Specifically, suppose that at time t, the agent is in state s t , select action a t , get instant rewards t , and transfer to the next state s t+1 . Then the cumulative discount and G starting from time t t Defined as:

[0036] Among them, γ is a discount factor with a value range of [0, 1], which is used to measure the importance of future rewards. The closer γ is to 1, the more importance is attached to future rewards; the closer γ is to 0, the more importance is attached to current rewards. In the Actor-Critic algorithm, the value function network is used to estimate the cumulative discounted sum of state-action pairs. By taking the current state and action as the input of the value function network, the network outputs the long-term value estimate of the state-action pair, that is, the expectation of the future cumulative discounted sum. This value estimate can be used to judge the pros and cons of the current strategy and guide the optimization direction of the strategy network.

[0037] Policy Gradient Algorithm: A reinforcement learning algorithm for optimizing policy network parameters. In the policy gradient algorithm, the policy network directly parameterizes the policy function, which is the mapping from state to action probability distribution. The goal of the policy network is to maximize the expected cumulative reward, that is, the policy gradient is defined as: Among them, θ is the parameter of the policy network, J(θ) is the expected cumulative reward, and τ is a trajectory, that is, a state-action-reward sequence, (s0,a0,r0,s1,a1,r1,......,s T ,a T ,r T ), π θ is the policy function with parameter θ, G t is the cumulative discounted sum starting from time t. The policy gradient algorithm updates the policy network parameters by gradient ascent, increasing the probability of good actions and reducing the probability of bad actions. In the actual optimization process, the policy gradient is usually improved in combination with the advantage function estimated by the value function network to reduce the variance of the gradient estimate and improve the training stability. Common improved algorithms include the Actor-Critic algorithm, the A3C algorithm, the PPO algorithm, etc.

[0038] In this application, the Actor network uses the policy gradient algorithm to optimize the policy network. The policy value output by the value function network is used as the reward signal, the Monte Carlo method is used to sample and estimate the gradient, and the policy network parameters are adjusted by gradient ascent, so that the generated working parameter adjustment strategy can obtain higher long-term returns.

[0039] Compared with the prior art, the advantages of this application are:

[0040] By constructing nonlinear mapping relationships and searching for globally optimal calibration parameters on the cloud server side, we can achieve accurate construction of complex nonlinear mapping relationships and global search of calibration parameter combination space, overcoming the problem that the limited computing resources of terminal devices alone make it difficult to achieve global optimal parameter search.

[0041] When constructing the parameter optimization model, multiple performance indicators such as signal-to-noise ratio, bit error rate, data throughput, and power efficiency are comprehensively considered. Gaussian process regression and confidence interval upper confidence boundary algorithm are used to construct the probability distribution proxy model and search for balanced parameters, ensuring the efficiency of parameter search while taking into account the balanced optimization of multiple performance indicators.

[0042] By deploying a real-time environment perception and dynamic calibration parameter generation mechanism based on the calibration trigger threshold on the radio communication equipment side, the equipment can adaptively perceive and calibrate the communication environment in real time, thus improving the equipment's ability to adapt to dynamic and complex environments.

[0043] The extreme learning machine network is used to fit the nonlinear relationship between the received data weight parameters and the real-time collected environmental data, and the dynamic calibration parameters are generated in combination with the environmental data. This realizes the dynamic and real-time optimization of the calibration parameters to environmental changes, and improves the accuracy and timeliness of the calibration.

[0044] The Actor-Critic reinforcement learning framework is used to perform intelligent dynamic calibration of transmit power and receive sensitivity, overcoming the problem that traditional calibration methods are difficult to adapt to dynamic environments and have insufficient generalization capabilities. Reinforcement learning can obtain the optimal strategy through interactive learning between devices and the environment, and has strong adaptability and generalization.

[0045] The Actor-Critic algorithm uses a policy network to generate a working parameter adjustment strategy, and a value function network to evaluate the long-term effectiveness of the adjustment strategy. Through interactive training of the two networks, the strategy is continuously optimized and improved, which improves the efficiency of strategy search while ensuring the rationality of strategy evaluation. The output of the policy network is modeled using a Gaussian distribution, and the value function is defined as the cumulative discounted sum of performance index differences, which improves the rationality of strategy representation and evaluation.

[0046] The optimized working parameters and performance indicators are fed back to the cloud server, and the cloud server conducts online learning and optimization of the Actor-Critic network based on the device feedback data, continuously improving the generalization and adaptability of the algorithm, realizing collaborative optimization of the cloud, edge and end, and the algorithm is constantly evolving. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is an exemplary flow chart of a method for controlling a radio communication device according to some embodiments of this specification;

[0048] Figure 2 is an exemplary flow chart of obtaining device calibration priority according to some embodiments of this specification;

[0049] Figure 3is an exemplary flow chart of generating a calibration trigger threshold and data weight parameters according to some embodiments of this specification;

[0050] Figure 4 This is an exemplary flowchart for calibrating the working parameters of a device using an Actor-Critic reinforcement learning framework according to some embodiments of this specification. DETAILED DESCRIPTION

[0051] The method and system provided in the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0052] Figure 1 This is an exemplary flow chart of a radio communication device control method shown in some embodiments of this specification, a radio communication device control method, including: obtaining device data of the radio communication device, the device data including antenna standing wave ratio and frequency error; uploading the obtained device data to a cloud server; after the cloud server receives the device data, the antenna standing wave ratio and frequency error are compared with preset thresholds respectively; when the antenna standing wave ratio or frequency error exceeds the threshold range, it is judged that the device is in an unhealthy state; when the device is in an unhealthy state, a fuzzy reasoning algorithm is used to calculate a quantitative index of device health according to the degree of deviation between the antenna standing wave ratio and the threshold, and the degree of deviation between the frequency error and the threshold; the quantitative index is mapped to a preset calibration priority level to determine the priority of device calibration; environmental data is collected, and environmental data packets are sent to the cloud server. Including temperature, humidity and electromagnetic interference data; upload the collected environmental data to the cloud server; the cloud server generates equipment calibration data through machine learning algorithms based on equipment data, environmental data, equipment calibration priority, and historical data; the calibration data includes calibration trigger thresholds and data weight parameters; the radio communication equipment compares the collected environmental data with the received calibration trigger thresholds; when the collected environmental data reaches the correction trigger threshold, according to the data weight parameters, through machine learning algorithms, dynamic calibration parameters for the current environment are generated; according to the generated dynamic calibration parameters, the Actor-Critic reinforcement learning framework is used to calibrate the working parameters of the equipment, and the working parameters include transmission power and receiving sensitivity; the calibrated working parameters are output to control the transmission and reception of the radio communication equipment.

[0053] The cloud server constructs a nonlinear mapping relationship. First, the radio communication device collects the working parameters of the device itself to form device data. The device data includes two indicators: antenna standing wave ratio and frequency error. The antenna standing wave ratio reflects the impedance matching characteristics of the antenna, and the frequency error reflects the frequency synchronization accuracy of the device. The two indicators are important parameters for evaluating the health status of the device. The radio communication device uploads the collected device data to the cloud server through the communication module. After receiving the device data, the cloud server extracts the antenna standing wave ratio and frequency error parameters, and compares them with the preset health thresholds respectively. The health threshold is a parameter range pre-set according to the technical indicators and working requirements of the device.

[0054] When the antenna standing wave ratio or frequency error exceeds the healthy threshold range, the cloud server determines that the device is in an unhealthy state and needs to be calibrated. Compared with a single hard threshold judgment, the cloud server uses a fuzzy reasoning algorithm to calculate the device's health quantitative index according to the degree of parameter deviation from the threshold, realizing the fuzzy evaluation of the device's health status. The input of the fuzzy reasoning algorithm is the degree of deviation of the antenna standing wave ratio from the threshold and the degree of deviation of the frequency error from the threshold, and the output is the device health quantitative index. The cloud server first fuzzifies the input parameters and maps the parameter deviation degree to fuzzy language variables. Then, based on the preset fuzzy reasoning rules, the device health membership is calculated, and the device health quantitative index is output through defuzzification. The quantitative index ranges from 0 to 100. The larger the value, the lower the device health and the higher the calibration priority.

[0055] The cloud server further maps the health quantitative indicators to the preset calibration priority level. The priority level can be preset in multiple levels, such as high, middle, and low. Different quantitative indicator intervals correspond to different priority levels. The larger the indicator, the higher the corresponding priority level. The cloud server constructs a nonlinear mapping relationship between the parameters and the device health and calibration priority based on the antenna standing wave ratio and frequency error parameters uploaded by the device. The fuzzy inference algorithm is used to overcome the limitations of hard threshold judgment, and the fuzzy evaluation of the device health status and the reasonable determination of the calibration priority are realized. At the same time, the complex mathematical model calculation is implemented in the cloud, which reduces the computing burden on the device side.

[0056] Figure 2This is an exemplary flow chart for obtaining device calibration priority according to some embodiments of this specification. When determining the device calibration priority, a fuzzy inference algorithm is used. The degree of deviation of device parameters from the health threshold is quantified into a fuzzy value by constructing a membership function, and then the device health status is calculated through fuzzy inference rules, and finally mapped to the calibration priority level. First, when the cloud server determines that the device is in an unhealthy state, the antenna standing wave ratio data, the antenna standing wave ratio health threshold, the frequency error data and the frequency error health threshold are used as input variables of the fuzzy inference algorithm.

[0057] Then, the input variables are mapped to the corresponding fuzzy sets by constructing a membership function. The membership function is used to calculate the degree of deviation of the parameter data from the health threshold, and the degree of deviation is represented by fuzzy variables, such as low, middle, high, etc. In this embodiment, the membership function adopts a bilateral exponential decay function, and the expression is as follows:

[0058] Among them, x is parameter data, c is the health threshold, λ is the threshold width, η L and η R is the attenuation coefficient on the left and right sides, β L and β R is the attenuation index of the left and right sides. When the parameter is within the threshold range, the membership is 1. When it exceeds the threshold range, the membership decays exponentially, and the decay rate is determined by the attenuation coefficient and the exponent. By inputting the antenna standing wave ratio data and the frequency error data into the membership function respectively, the membership of the deviation degree of the two parameters relative to the healthy threshold can be obtained.

[0059] Next, according to the degree of membership of the parameter deviation, fuzzy reasoning calculation is performed through the preset fuzzy reasoning rules to obtain the fuzzy value representing the health status of the equipment. The fuzzy reasoning rule is a fuzzy conditional statement in the form of IF-THEN, which is used to describe the corresponding relationship between the parameter deviation degree and the health status of the equipment. For example: IF the antenna standing wave ratio deviation degree is high AND the frequency error deviation degree is high THEN the equipment health status is poor. IF the antenna standing wave ratio deviation degree is lowAND the frequency error deviation degree is low THEN the equipment health status is good.

[0060] The cloud server presets multiple fuzzy inference rules to cover various parameter deviation combinations. According to the parameter deviation membership, the fuzzy inference rules are used to calculate the device's membership in poor, middle, good and other health states.

[0061] Then, the membership of the equipment health status obtained by fuzzy reasoning is defuzzified to calculate the quantitative index of the equipment health. Defuzzification can use common algorithms such as the centroid method to convert the membership distribution of the health status into a quantitative value in the range of 0 to 100 to represent the equipment health.

[0062] Finally, the health quantitative index is mapped to the calibration priority level. The cloud server presets multiple priority levels, each of which corresponds to an interval of the health quantitative index. The quantitative index is compared with the interval boundary value of the priority level, and the priority level is determined according to the interval to obtain the calibration priority of the device. For example, three priority levels can be preset: low: the health quantitative index interval [0, 30], indicating that the device is in good health and the calibration priority is low; middle: the health quantitative index interval (30, 70], indicating that the device is in general health and the calibration priority is medium; high: the health quantitative index interval (70, 100], indicating that the device is in poor health and the calibration priority is high. If the health quantitative index of the device is calculated to be 85, it is in the interval (70, 100], corresponding to the high level, so the calibration priority level of the device is high. The cloud server uses fuzzy reasoning algorithm to determine the implementation process of the device calibration priority. Compared with simple threshold comparison, fuzzy reasoning algorithm can more comprehensively consider the comprehensive influence of various parameters of the device, give an overall evaluation of the health status of the device, and more intelligently and reasonably determine the calibration priority, which is convenient for hierarchical control and calibration scheduling of the equipment.

[0063] Figure 3 It is an exemplary flowchart for generating calibration trigger thresholds and data weight parameters according to some embodiments of this specification, generating device calibration data, including: first, the radio communication device collects current environmental data, such as channel state information, interference noise, etc. In order to extract the key features of the environmental data, the device inputs the collected environmental data into a pre-trained multi-layer autoencoder network for feature extraction and dimensionality reduction. The multi-layer autoencoder network maps the input data to a low-dimensional feature space in the encoder through unsupervised learning, and then reconstructs the original data through the decoder. The goal of network training is to minimize the reconstruction error so that the extracted features can retain the information of the original data to the greatest extent. The trained autoencoder network can convert high-dimensional environmental data into low-dimensional quantitative indicators to characterize the current environmental state.

[0064] Then, the radio communication device uploads the quantitative indicators of the environmental status to the cloud server through the communication link. After receiving the quantitative indicators, the cloud server takes them together with the acquired equipment working parameters (such as antenna standing wave ratio, frequency error, etc.) and the historical data stored in the cloud as input data, and learns the nonlinear mapping relationship between environmental factors and equipment performance through the pre-built bidirectional long short-term memory (BLSTM) network.

[0065] The BLSTM network is a recurrent neural network that can simultaneously utilize the forward and backward dependencies of data by introducing long short-term memory units in both the forward and backward directions to better model the long-distance dependencies of time series data. The network inputs are quantitative indicators of environmental status, equipment operating parameters, and historical data, and the outputs are equipment performance indicators such as signal-to-noise ratio, bit error rate, data throughput, power efficiency, etc. By training the BLSTM network, we can learn the influence of environmental factors on equipment performance and obtain a nonlinear mapping relationship model between the two.

[0066] Next, based on the learned nonlinear mapping relationship between environment and performance, the cloud server takes the device performance index as the optimization target and combines the Bayesian optimization algorithm to search for the optimal device calibration parameter combination. Based on the nonlinear mapping relationship between environmental factors and device performance constructed in step S1, the cloud server constructs a probability distribution proxy model of the parameter space through Gaussian process regression (GPR). GPR characterizes the uncertainty of the optimization objective function by defining a Gaussian probability distribution on the parameter space. The probability distribution is parameterized by the mean function and the covariance function, and the hyperparameters are optimized by maximizing the marginal likelihood estimation to fit the limited sampling data. After obtaining the parameter space probability distribution proxy model, the cloud server uses the confidence interval upper confidence bound (UCB) algorithm to search for the globally optimal calibration parameter combination. The UCB algorithm selects the next evaluation point by balancing the posterior mean and uncertainty of the candidate parameter combination, taking into account the exploration and utilization of parameters. The detailed process of the algorithm is as follows: Initialize the exploration factor weight a of the UCB algorithm t and using factor weights β t , set the initial number of iterations t = 1. According to the current number of iterations t, adjust the exploration factor weight a through the preset weight adjustment function t and using factor weights β t The adjustment function can be in the form of exponential decay, so that as the number of iterations increases, the weight of the exploration factor gradually decreases, and the weight of the utilization factor gradually increases, guiding the algorithm to gradually transition from the initial global exploration to the later local utilization. For each candidate parameter combination x, calculate its confidence interval upper bound (UCB value). The calculation formula of the UCB value is: Among them, μ(x) is the posterior mean of the candidate parameter combination x, σ(x) is the posterior standard deviation of the candidate parameter combination x, and γ t is an additional exploration term, and ε is a small positive constant. The UCB value is composed of the weighted sum of the posterior mean (utilization term) and the posterior standard deviation (exploration term). t The term introduces random exploration to escape from the local optimum. The candidate parameter combination with the largest UCB value is selected as the next evaluation point x t, input it into the optimization objective function (environment-performance mapping relationship) for evaluation, and obtain the corresponding performance index value y t . The new evaluation data (x t ,y t ) is added to the existing sampled data set and used to update the probability distribution of the GPR model. The updated GPR model can adjust the probability distribution based on the newly added sampled data and improve the accuracy of subsequent predictions.

[0067] Determine whether the iteration stop condition is met. Common stop conditions include reaching the maximum number of iterations, optimal value convergence, etc. If the stop condition is met, the currently obtained global optimal calibration parameter combination is output as the final optimization result; otherwise, the number of iterations t is increased by 1, and the iterative search is returned to continue. After multiple rounds of iterations, the UCB algorithm can find the global optimal calibration parameter combination in the parameter space to maximize the performance of the device. Finally, the cloud server uses the optimal calibration parameter combination as calibration data, extracts the calibration trigger threshold and data weight parameters, generates complete calibration data and sends it to the radio communication device to guide the adaptive calibration of the device. The dynamic adjustment of the exploration factor and the utilization factor weight in the UCB algorithm enables the algorithm to conduct more global exploration of the parameter space in the early stage and more local utilization in the promising area in the later stage, which improves the efficiency and accuracy of the search. The introduction of additional random exploration items can help the algorithm jump out of the local optimum and enhance the global search capability. Compared with traditional grid search and random search, the UCB algorithm can more efficiently approach the global optimal solution under a limited evaluation budget.

[0068] Real-time device perception and dynamic parameter generation: First, the radio communication device collects the current environmental data in real time, such as channel state information, interference noise, etc. The device compares the real-time collected environmental data with the calibration trigger threshold received from the cloud server to determine whether dynamic calibration needs to be triggered. The calibration trigger threshold is generated by the cloud server, indicating that when the environmental state changes to a certain extent, calibration needs to be started to adapt to the new environmental conditions. When the real-time collected environmental data reaches or exceeds the calibration trigger threshold, the radio communication device uses the data weight parameters received from the cloud server as input, and fits the nonlinear relationship between the data weight parameters and the environmental data through the pre-trained extreme learning machine (ELM) network.

[0069] The ELM network is a single hidden layer feedforward neural network with the advantages of fast learning speed and strong generalization ability. Compared with the traditional BP neural network, the hidden layer node weights and biases of the ELM network are randomly generated and fixed. Only the output layer weights need to be learned, which greatly reduces the computational complexity. The training process of the ELM network can be expressed as the following optimization problem: Among them, β is the output layer weight vector, h(xi ) is the hidden layer output vector, C is the regularization parameter, {(x i ,y i )} is N training samples. The optimization problem can be solved by the least squares method.

[0070] Among them, ELM consists of an input layer, a hidden layer, and an output layer. The input layer receives input data, the hidden layer performs nonlinear transformation on the input data, and the output layer obtains the final prediction result by linearly combining the outputs of the hidden layer nodes. The training process of ELM is divided into two stages: Random initialization stage: randomly generate the connection weights and biases of the hidden layer nodes, and map the input data to the random feature space. Output weight solution stage: according to the hidden layer output and the expected output, the output weight is solved by the least squares method to minimize the prediction error. The mathematical expression of ELM is as follows: Assume that there are N samples (x_i, t_i), i = 1, 2, ..., N, where x_i is an m-dimensional input vector and t_i is an n-dimensional target vector. The goal of ELM is to find a mapping function f(x) so that f(x_i) is as close to t_i as possible. Let the number of hidden layer nodes be L, and the activation function be g(x), then the mathematical model of ELM can be expressed as: ∑_{i=1}^Lβ_ig(w_i*x_j+b_i)=o_j,j=1,2,......,N,where w_i is the connection weight vector from the input layer to the i-th hidden layer node, b_i is the bias of the i-th hidden layer node, β_i is the connection weight vector from the i-th hidden layer node to the output layer, and o_j is the predicted output vector of the j-th sample. The training process of ELM can be transformed into solving the following linear equations: Hβ=T, where H is the hidden layer output matrix, β is the output weight matrix, and T is the target output matrix. The definitions of H and T are as follows: H = [g(w_1*x_1+b_1), ..., g(w_L*x_1+b_L); g(w_1*x_2+b_1), ..., g(w_L*x_2+b_L); ..., g(w_1*x_N+b_1), ..., g(w_L*x_N+b_L)], T = [t_1, t_2, ..., t_N]^T The output weight matrix β can be solved by the least squares method: β = H^+T, where H^+ is the Moore-Penrose generalized inverse matrix of H. In this application, an extreme learning machine network is used to fit the nonlinear relationship between data weight parameters and environmental data. By taking the data weight parameters as the input of ELM and the environmental data as the expected output of ELM, the ELM network is trained to approximate the mapping relationship between the two. The trained ELM network can predict and generate dynamic calibration parameters based on the currently collected environmental data, thereby realizing real-time adjustment of the equipment operating parameters.

[0071] In this embodiment, the device collects a certain amount of environmental data and the corresponding optimal calibration parameters as training samples in advance, and trains the ELM network offline. During training, the environmental data and data weight parameters are used as network inputs, and the optimal calibration parameters are used as expected outputs to learn and fit the nonlinear mapping relationship between the two. When dynamic calibration is triggered, the device inputs the data weight parameters received from the cloud and the currently collected environmental data into the trained ELM network, and generates dynamic calibration parameters suitable for the current environment through the forward propagation calculation process. The dynamic calibration parameters are real-time adjustment values ​​for the device operating parameters (such as transmit power, modulation mode, etc.), which are used to dynamically optimize the performance of the device. Finally, the radio communication device uses the generated dynamic calibration parameters to adjust the original operating parameters, so that the working state of the device dynamically adapts to the current environmental conditions, thereby maintaining the stability and optimization of the communication performance when the environment changes. The radio communication device uses the calibration trigger threshold and data weight parameters obtained from the cloud, combined with the ELM network to realize the real-time generation of dynamic calibration parameters on the end side, reducing frequent end-cloud interactions and improving the calibration response speed. The pre-trained ELM network can quickly learn the complex nonlinear relationship between data weight parameters and environmental data, generate dynamic calibration parameters according to the real-time environment, and enable the device to make adaptive adjustments to the dynamically changing environment. This solution fully utilizes the global optimization capabilities of the cloud and the real-time processing capabilities of the terminal, realizes distributed collaborative optimization of the calibration process, and improves the flexibility and efficiency of device adaptive calibration.

[0072] Figure 4 This is an exemplary flow chart of calibrating the working parameters of a device using an Actor-Critic reinforcement learning framework as shown in some embodiments of this specification, and a radio communication device uses the Actor-Critic reinforcement learning framework to adaptively calibrate the working parameters of the device. First, the radio communication device uses the Actor-Critic reinforcement learning framework to perform real-time optimization of operating parameters such as transmit power and receive sensitivity based on the generated dynamic calibration parameters.

[0073] The Actor-Critic framework consists of two parts: a policy network (Actor) and a value function network (Critic). The policy network is used to generate adjustment strategies for working parameters, and the value function network is used to evaluate the pros and cons of the adjustment strategies.

[0074] The policy network of the actor network adopts a deep neural network structure to learn the mapping relationship from the device state space to the working parameter adjustment space, and generate the optimal working parameter adjustment strategy according to the current device state and dynamic calibration parameters. State observation: Various information that characterizes the current device state, such as channel quality indicators (channel gain, channel fading, etc.), interference level (interference power, interference type, etc.), device working state (transmit power, receiving sensitivity, etc.), etc., as well as the generated dynamic calibration parameters, constitute the state observation vector o_t. The observation vector describes the environmental conditions and current working state of the device, and is the input of the policy network. Policy network structure: The policy network adopts a multi-layer feedforward neural network structure, which consists of an input layer, multiple hidden layers and an output layer. The number of neurons in the input layer corresponds to the dimension of the state observation vector o_t, and receives the input of the device state and dynamic calibration parameters. The hidden layer adopts a fully connected structure, and introduces nonlinear transformations through nonlinear activation functions (such as ReLU, Tanh, etc.) to extract high-dimensional features of the state space. The number of neurons in the output layer corresponds to the dimension of the working parameter adjustment amount a_t, and generates adjustment values ​​for the transmit power and receiving sensitivity.

[0075] Policy distribution modeling: In reinforcement learning, the policy is defined as the conditional probability distribution of actions under a given state, expressed as π_θ(a_t|o_t). In this embodiment, considering that the working parameter adjustment amount a_t is a continuous value, the Gaussian distribution is used to model the policy. That is, assuming that in the state o_t, the working parameter adjustment amount a_t obeys a Gaussian distribution with a mean of μ_θ(o_t) and a variance of σ_θ^2(o_t), recorded as a_t~N(μ_θ(o_t), σ_θ^2(o_t)). Among them, μ_θ and σ_θ are the mean and variance of the Gaussian distribution output by the policy network, respectively, and are functions of the network parameter θ. By adjusting the parameter θ of the policy network, the mean and variance of the Gaussian distribution can be changed, thereby affecting the adjustment strategy of the working parameters. Policy network training: The training goal of the policy network is to maximize the expected cumulative reward, that is, to find the optimal network parameter θ so that E[∑_{t=0}^∞γ^tr_t] reaches the maximum. Where r_t is the immediate reward for each time step, and γ is the discount factor. The policy gradient algorithm is used to train the policy network, and the policy gradient is calculated based on the immediate reward and the long-term value estimated by the value function network. The gradient ascent method is used to update the parameter θ and continuously optimize the policy network so that the working parameter adjustment strategy generated by it can obtain higher cumulative rewards.

[0076] Working parameter adjustment: During the operation of the device, the real-time collected device status information and dynamic calibration parameters are input into the trained policy network. Through the forward propagation calculation process, the Gaussian distribution parameters μ_θ(o_t) and σ_θ(o_t) of the working parameter adjustment amount a_t are obtained. Then, the specific working parameter adjustment value is sampled from the Gaussian distribution, which is used as the adjustment amount of the transmit power and receive sensitivity, respectively, to perform real-time parameter tuning on the device's transmit and receiver to adapt them to the current environmental conditions and performance requirements. Performance evaluation: The adjusted transmit power and receive sensitivity are applied to the actual communication process of the device, and the performance indicators of the device, such as bit error rate, signal-to-noise ratio, etc., are monitored in real time. The obtained performance indicators are compared with the expected target performance indicators to obtain an immediate reward r_t, which is used to evaluate the pros and cons of the current working parameter adjustment strategy. At the same time, r_t is fed back to the Actor network and the Critic network to guide the next round of strategy optimization and value estimation. The radio communication device uses the policy network of the Actor network to autonomously learn and optimize the adjustment strategy of the working parameters according to the device status and dynamic calibration parameters, thereby realizing the adaptive calibration of the device working parameters. The policy network uses a deep neural network structure and Gaussian distribution policy modeling, which can extract effective features from the state space, generate continuous parameter adjustments, and adapt to complex and changing wireless communication environments. The end-to-end parameterized policy learning method overcomes the limitations of traditional calibration methods, has stronger self-learning and self-adaptation capabilities, and improves the intelligence level and real-time performance of dynamic calibration of equipment.

[0077] The value function network of the Critic network is used to evaluate the long-term value of the working parameter adjustment strategy generated by the Actor network and provide guidance for the strategy optimization of the Actor network. State observation: The adjusted device performance indicator r_t (such as bit error rate, signal-to-noise ratio, etc.) and the preset target performance indicator r_target are used as the input observation value of the Critic network. Among them, r_t represents the immediate performance feedback obtained after taking the adjustment strategy in the current state, and r_target represents the expected performance target. The two constitute a state observation vector and are input into the value function network. Value function network structure: The value function network adopts a deep neural network structure similar to the policy network, which consists of an input layer, multiple hidden layers and an output layer. The input layer receives the observed values ​​of the device performance indicators and the target performance indicators, the hidden layer extracts the state features through nonlinear transformation, and the output layer outputs a scalar value, which represents the estimated value of the adjustment strategy in the current state.

[0078] Value function definition: The value function V(o_t) represents the expected cumulative reward that can be obtained by continuing to execute the current adjustment strategy in state o_t. In this embodiment, the reward is defined as the cumulative discounted sum of the difference between the future device performance index and the target performance index, that is, V(o_t) = E[∑_{k=0}^∞γ^k(r_{t+k}-r_target)]. Among them, γ is a discount factor, which takes a value between 0 and 1 and is used to balance immediate rewards and long-term rewards. The closer γ is to 1, the more attention is paid to long-term benefits; the closer γ is to 0, the more attention is paid to short-term benefits. The significance of the value function is that it evaluates the cumulative performance improvement effect of the current adjustment strategy in the future, rather than just focusing on the performance feedback of the current step. Value function network training: The training goal of the value function network is to minimize the mean square error between the estimated value function and the true cumulative reward, that is, to find the optimal network parameter φ to minimize E[(V_φ(o_t)-∑_{k=0}^∞γ^k(r_{t+k}-r_target))^2]. The value function network is trained using the temporal difference (TD) algorithm, and the value function of the current state is updated using the estimated value function of the next moment through the Bootstrap method. According to the TD error δ_t=r_t+γV_φ(o_{t+1})-V_φ(o_t), the gradient of the value function network is calculated, and the network parameter φ is updated using optimization algorithms such as stochastic gradient descent, so that the value function approaches the true expected cumulative reward.

[0079] Policy evaluation and update: The output of the value function network, V_φ(o_t), represents the estimated long-term value of executing the current adjustment policy in state o_t, which can be used as the reward value of the Actor network to evaluate the pros and cons of the current policy. The Actor network uses the policy gradient algorithm to update the parameters θ of the policy network according to the reward value and optimize the policy to maximize the expected cumulative reward. The policy gradient is defined as It represents the expectation of the product of the gradient of the logarithmic probability of the policy distribution with respect to the parameter θ and the value function. Intuitively, in state o_t, if the action a_t generated by the current policy can obtain a higher long-term value estimate V_φ(o_t), then the probability of the policy distribution π_θ(a_t|o_t) is increased to encourage the generation of such actions; otherwise, its probability is reduced. In order to improve the sampling efficiency of gradient estimation, the Monte Carlo method is used to approximate the expected value of the gradient by repeatedly sampling multiple state-action trajectories, and the policy gradient is rewritten using PolicyGradientTheorem to improve training stability.

[0080] Strategy-value function interactive training: The Actor network and the Critic network implement iterative optimization of the strategy and value function through interactive training. The Actor network generates and executes the working parameter adjustment strategy, obtains the immediate reward and the next state, and passes it to the Critic network to evaluate the long-term value of the strategy, and thus guides the strategy improvement of the Actor network. The Critic network uses the trajectory data generated by the interaction between the Actor network and the environment to continuously optimize the estimation accuracy of the value function through the TD algorithm. The two promote each other and eventually converge to the optimal strategy and value function. The radio communication equipment uses the value function network of the Critic network to evaluate the long-term value of the working parameter adjustment strategy generated by the Actor network, providing effective guidance for strategy optimization. The value function network adopts a deep neural network structure and a TD learning algorithm, which can extract effective features from the state space, accurately estimate the expected cumulative reward of the strategy, and overcome the limitations and contingency of immediate rewards. The Actor-Critic architecture realizes the decoupled learning of strategy and value, and improves the training efficiency and the effectiveness of strategy exploration. Finally, through the interactive optimization of the Actor network and the Critic network, an intelligent network model was obtained that could generate the optimal working parameter adjustment strategy according to the device status and dynamic calibration parameters, thus realizing the adaptive calibration of device parameters and the dynamic optimization of performance.

[0081] During the operation of the equipment, the radio communication equipment inputs the real-time collected equipment status and dynamic calibration parameters into the trained Actor network, generates real-time working parameter adjustments, and dynamically optimizes the transmit power and receive sensitivity, so that the equipment can make adaptive parameter calibration to environmental changes.

[0082] Compared with traditional PID control and parameter lookup methods, the Actor-Critic framework has stronger self-learning and self-adaptation capabilities. Through end-to-end parameterized strategy learning, the Actor network can generate continuous parameter adjustment strategies based on device status and calibration parameters to adapt to changing working environments. The Critic network provides a long-term value evaluation mechanism to guide the Actor network to optimize strategies in pursuit of long-term cumulative benefit maximization, avoiding short-sighted greedy adjustments. By using the trial-and-error exploration and strategy optimization mechanism of reinforcement learning, radio communication equipment can autonomously learn and optimize working parameters through interaction with the environment, improving the intelligence level and robustness of equipment adaptive calibration. At the same time, combined with the calibration trigger threshold and data weight parameters generated by the cloud, distributed parameter calibration optimization with end-cloud collaboration is realized, giving full play to the real-time processing capabilities of edge devices and the global optimization capabilities of the cloud.

[0083] The cloud-edge collaborative optimization method is adopted between the radio communication equipment and the cloud server, which realizes the real-time adaptive calibration of working parameters on the device side and the global strategy optimization on the cloud side, and improves the generalization ability and environmental adaptability of the algorithm. Device-side parameter control: The radio communication equipment uses the trained and optimized Actor network to generate the optimal working parameter adjustment strategy based on the real-time collected device status information and the dynamic calibration parameters sent by the cloud. The adjusted parameters such as transmit power and receive sensitivity are configured in the RF front-end circuit, and the transmitter and receiver of the radio equipment are controlled in real time to make it work in the best state and meet the communication performance and reliability requirements. Device-side data upload: While performing working parameter adjustment, the radio communication equipment uploads the optimized working parameter values ​​(such as transmit power, receive sensitivity, etc.) and the corresponding device performance indicators (such as bit error rate, signal-to-noise ratio, etc.) to the cloud server through the network. These data include the device's status information, action information, and instant reward information, reflecting the performance feedback obtained after the device takes different actions in different states, and are an important basis for cloud-side strategy optimization.

[0084] Cloud data aggregation: The cloud server receives working parameters and performance indicator data uploaded by multiple radio communication devices through the network, aggregates and stores them. The cloud maintains a global data set, which contains state-action-reward data samples of different devices under various environmental conditions. These data cover a wide range of wireless communication scenarios and device working conditions, providing rich training samples for cloud policy optimization. Cloud policy optimization: The cloud server performs online learning and optimization on the Actor-Critic network deployed on the cloud based on the aggregated data samples. Using the state-action-reward data uploaded by the device, the policy parameters of the Actor network and the value function parameters of the Critic network are updated respectively through the policy gradient algorithm and the temporal difference algorithm. The policy network learns how to generate the optimal working parameter adjustment strategy according to the device status, and the value function network learns how to evaluate the long-term value of the strategy. Through continuous parameter updates and policy iterations, the Actor-Critic network on the cloud gradually improves its generalization ability and can adapt to more variable wireless communication environments.

[0085] Global model synchronization: After completing the optimization training of the Actor-Critic network in the cloud, the updated network parameters are sent to each radio communication device. The Actor network and Critic network on the device side keep the model synchronized with the cloud, and obtain the latest strategy and value function learned by the cloud in a timely manner. This cloud-edge collaborative model update method enables the device side to share the global optimization results of the cloud and improve the decision-making quality and adaptability of local strategies. Strategy execution and feedback: Radio communication equipment uses the synchronized Actor network to generate the optimal working parameter adjustment strategy based on the real-time status, and uploads the strategy execution results and performance feedback to the cloud to form a new data sample. The cloud receives feedback data from each device and continuously optimizes the strategy and updates the model. Through this cyclic iterative method, the collaborative optimization of the device side and the cloud side is achieved, and the environmental adaptability of the algorithm is continuously improved.

[0086] Radio communication equipment and cloud servers form a closed-loop optimization system. The device side is responsible for real-time state perception, parameter adjustment, and data collection, while the cloud side is responsible for global data aggregation, policy optimization, and model updates. The two achieve collaboration through data upload and model synchronization. The device side performs local optimal control, and the cloud side improves global policy performance. This cloud-edge collaborative architecture fully utilizes the real-time processing capabilities of edge devices and the computing and storage advantages of the cloud, improving the intelligence level and adaptability of wireless communication systems. The adaptive calibration of device operating parameters no longer relies on pre-set rules and thresholds, but through end-to-end data-driven learning, it continuously optimizes and adapts to complex and changing wireless communication environments, thereby achieving dynamic optimization and intelligent management and control of communication performance.

Claims

1. A method for controlling a radio communication device, comprising: S1, obtaining device data of a radio communication device, where the device data includes antenna standing wave ratio and frequency error; Upload the acquired device data to the cloud server; After receiving the device data, the cloud server compares the antenna standing wave ratio and frequency error with the preset thresholds respectively; when the antenna standing wave ratio or frequency error exceeds the threshold range, the device is judged to be in an unhealthy state; when the device is in an unhealthy state, a fuzzy reasoning algorithm is used to calculate the quantitative indicators of device health according to the degree of deviation between the antenna standing wave ratio and the threshold, and the degree of deviation between the frequency error and the threshold; the quantitative indicators are mapped to the preset calibration priority level to determine the priority of device calibration; S2, collects environmental data, which includes temperature, humidity and electromagnetic interference data; uploads the collected environmental data to the cloud server; the cloud server generates device calibration data through machine learning algorithms based on device data, environmental data, device calibration priority, and historical data; the calibration data includes calibration trigger thresholds and data weight parameters; S3, the radio communication device compares the collected environmental data with the received calibration trigger threshold; when the collected environmental data reaches the calibration trigger threshold, a dynamic calibration parameter of the current environment is generated through a machine learning algorithm according to the data weight parameter; S4, based on the generated dynamic calibration parameters, the Actor-Critic reinforcement learning framework is used to calibrate the device's operating parameters, including transmit power and receive sensitivity; Output the calibrated working parameters to control the transmission and reception of radio communication equipment; S3, generating dynamic calibration parameters of the current environment, including: Collect environmental data in real time, and compare the collected environmental data with the calibration trigger threshold received in S2; When the collected environmental data reaches the calibration trigger threshold, the data weight parameter received in S3 is used as input, and the nonlinear relationship between the data weight parameter and the environmental data is fitted through the pre-trained extreme learning machine network; According to the nonlinear relationship obtained by fitting, combined with the currently collected environmental data, dynamic calibration parameters under the current environment are generated; wherein the dynamic calibration parameters are used to dynamically adjust the working parameters of the equipment; S4, uses the Actor-Critic reinforcement learning framework to calibrate the working parameters of the equipment, including: According to the dynamic calibration parameters generated by S3, the Actor-Critic reinforcement learning framework is used to calibrate the transmit power and receive sensitivity. The Actor-Critic reinforcement learning framework includes a policy network and a value function network. The policy network is used to generate the adjustment strategy of the working parameters, and the value function network is used to evaluate the advantages and disadvantages of the adjustment strategy. The current device state and dynamic calibration parameters are used as observation values ​​and input into the policy network of the Actor network. The policy network adopts a deep neural network structure, obtains the mapping relationship from state to action through network training, and generates an adjustment strategy for the working parameters based on the observation values. The output of the policy network is the adjustment amount of the working parameters, which is modeled using Gaussian distribution. According to the working parameter adjustment amount generated by the policy network, the transmit power and receive sensitivity are adjusted to obtain the adjusted device performance index; The adjusted equipment performance index and target performance index are used as observation values ​​and input into the value function network of the Critic network. The value function network adopts a deep neural network structure, obtains the mapping relationship from the learning state to the value function through network training, and calculates the value of the adjustment strategy according to the observation value. The value function is defined as the cumulative discounted sum of the differences between performance indicators, which is used to evaluate the long-term utility of the adjustment strategy. The policy value output by the value function network is used as the reward value of the Actor network. The Actor network uses the policy gradient algorithm to update the parameters of the policy network according to the reward value. The Monte Carlo method is used to perform unbiased sampling estimation of the gradient. Through interactive training of the Actor network and the Critic network, the working parameters are iteratively optimized.

2. The radio communication device control method according to claim 1, characterized in that: S1, determine the priority of equipment calibration, including: When the device is in an unhealthy state, a fuzzy inference algorithm is used, and the antenna standing wave ratio data, the antenna standing wave ratio health threshold, the frequency error data and the frequency error health threshold are used as input variables of the fuzzy inference; By constructing a membership function, the input variables are mapped to the corresponding fuzzy sets, and the deviation degree of the antenna standing wave ratio data from the antenna standing wave ratio health threshold and the deviation degree of the frequency error from the frequency health threshold are calculated; According to the deviation degree of antenna standing wave ratio and frequency error, fuzzy reasoning is performed through fuzzy reasoning rules to obtain a fuzzy value representing the health status of the equipment; the fuzzy reasoning rules include fuzzy rules in the form of IF-THEN, which are used to describe the fuzzy relationship between the deviation degree of antenna standing wave ratio and frequency error and the health status of the equipment; Defuzzify the fuzzy value to obtain the quantitative index of equipment health; The quantitative indicator is compared with the preset calibration priority level threshold, and the quantitative indicator is mapped to the corresponding calibration priority level according to the comparison result to obtain the equipment calibration priority; wherein, the calibration priority level is a plurality of preset levels corresponding to the health of the equipment, and each level corresponds to a range interval of a health quantitative indicator. If the health quantitative indicator is within the range, it is mapped to the corresponding priority level.

3. The radio communication device control method according to claim 2, characterized in that: The expression of the constructed membership function f(x) is as follows: Where c represents the symmetric center of the function and the midpoint of the platform area; λ represents the position parameter, which controls the width of the platform area. The range of the platform area is [c-λ, c+λ], and the degree of membership is always 1 within this range; η L Represents the left scale parameter, which controls the width of the non-platform area on the left side of the function; η R represents the right scale parameter, which controls the width of the non-platform area on the right side of the function; β L Represents the left shape parameter, which controls the shape of the non-platform area on the left side of the function; β R Represents the right-side shape parameter, which controls the shape of the non-platform area on the right side of the function.

4. The radio communication device control method according to claim 1, characterized in that: S2, generate equipment calibration data, including: The collected environmental data is input into a pre-trained multi-layer autoencoder network, and the environmental data is subjected to feature extraction and dimensionality reduction processing through the multi-layer autoencoder network to obtain quantitative indicators representing the environmental status; The radio communication device uploads the obtained quantitative indicators to the cloud server through the communication link; After receiving the quantitative indicators, the cloud server uses the quantitative indicators, the device data in S1, and the historical data stored in the cloud service as inputs, and obtains the nonlinear mapping relationship between environmental factors and device performance through the pre-built bidirectional long short-term memory network; Based on the acquired nonlinear mapping relationship, the cloud service takes the device performance indicators as the optimization target and combines the Bayesian optimization algorithm to search for the optimal calibration parameter combination to generate device calibration data corresponding to the current environmental data; among which, the device performance indicators include signal-to-noise ratio, bit error rate, data throughput or power efficiency.

5. The radio communication device control method according to claim 4, characterized in that: Generate device calibration data corresponding to the current environment data, including: The obtained nonlinear mapping relationship is used as the objective function of the optimization problem, the signal-to-noise ratio, bit error rate, data throughput or power efficiency is used as the optimization target, the calibration parameter combination to be optimized is used as the optimization variable, and a parameter optimization model is constructed; Based on the constructed parameter optimization model, a probability distribution proxy model of the parameter space is constructed through Gaussian process regression; the exploration and utilization of parameters are balanced by setting an acquisition function based on the confidence interval upper confidence boundary algorithm; among them, the confidence interval upper confidence boundary algorithm adopts a weight adjustment mechanism for the exploration factor and the utilization factor; Taking the parameter optimization model, the probability distribution proxy model and the acquisition function as input, the parameter space is iteratively searched to obtain the globally optimal calibration parameter combination; The globally optimal calibration parameter combination obtained is used as the device calibration data under the current environmental state to generate calibration data including a calibration trigger threshold and a data weight parameter.

6. The radio communication device control method according to claim 5, characterized in that: Obtain the globally optimal calibration parameter combination, including: Initialize the exploration factor weight and utilization factor weight in the confidence interval upper confidence boundary algorithm, and set the initial number of iterations; According to the current number of iterations, the exploration factor weight and the utilization factor weight are adjusted through the preset weight adjustment function; The adjusted exploration factor weight, utilization factor weight, posterior mean and posterior standard deviation of the candidate parameter combination are used as inputs, and the confidence interval upper bound of the candidate parameter combination is calculated using the confidence interval upper bound calculation formula; The calculated confidence interval upper bound is used as input, and the candidate parameter combination with the largest confidence interval upper bound is selected as the next evaluation point; The evaluation point is input into the objective function for evaluation to obtain the evaluation result; and the evaluation result is used as input to update the Gaussian process regression model; The updated Gaussian process regression model and the current number of iterations are used as input, and the preset iteration stop conditions are used to determine whether the iteration stop conditions are met; if so, the global optimal calibration parameter combination obtained by the iterative search is output as the optimal calibration parameter combination under the current environmental state; otherwise, the current number of iterations is increased by 1, and the adjustment of the exploration factor weight and the utilization factor weight is returned.

7. The radio communication device control method according to claim 6, characterized in that: The upper bound of the confidence interval is calculated by: Among them, μ(x) is the posterior mean of the candidate parameter combination {x}, σ(x) is the posterior standard deviation of the candidate parameter combination x, and a t To explore factor weights, β t To utilize factor weights, γ t is an additional exploration term and ε is a small positive constant.

8. The radio communication device control method according to claim 1, characterized in that: S4, uses the Actor-Critic reinforcement learning framework to calibrate the device's operating parameters, and also includes: Output the iteratively optimized working parameters to control the transmission and reception of the radio communication equipment; upload the optimized working parameters and corresponding performance indicators to the cloud server; The cloud server performs online learning and optimization of the Actor-Critic network based on the received data.

Images