Automatic satellite alignment method based on reinforcement learning

Abstract

The automatic satellite alignment method based on reinforcement learning includes: collecting data, constructing a predetermined set of coordinate system-signal strength data, and constructing a training data set and a test data set based on the coordinate system-signal strength data; constructing a deep neural network module, configuring a reward function, an observable state quantity and an action space, using the training data set as input data, and training the deep neural network module for a predetermined number of times; using the test data set to test the trained deep neural network module to obtain the final optimized deep neural network module. This solution can greatly improve the efficiency of satellite alignment and improve the work quality through automatic satellite alignment.

Description

Technical Field

[0001] The invention relates to communication technology, in particular to a satellite alignment method in satellite communication. Background Art

[0002] Satellite communication technology is an important wireless communication method that can be used as an emergency communication method after natural disasters such as earthquakes and tsunamis. In these emergency rescue scenarios, portable stations play a huge role. Compared with vehicle-mounted stations, portable stations are easy to carry and can be deployed quickly. However, ordinary portable stations require manual satellite alignment after startup, which requires operators to have a certain understanding of the satellite alignment method. To this end, technicians have developed an automatic satellite alignment system. The satellite communication terminal equipped with the automatic satellite alignment system is designed to lower the threshold for operators. Even those who are not familiar with the satellite alignment method can complete the satellite alignment process simply and quickly after starting up.

[0003] However, the satellite communication terminals currently on the market equipped with automatic alignment systems generally take 2-3 minutes to complete alignment, and the algorithms used are basically based on scanning. However, a skilled operator can complete alignment within 1 minute. Finding an artificial intelligence algorithm that can quickly and efficiently complete alignment, with a time close to or exceeding the time required for manual alignment, is an urgent problem that needs to be solved.

[0004] The field of machine learning can be roughly divided into three parts: supervised learning, unsupervised learning, and reinforcement learning. Supervised learning requires the use of manually labeled data for training, which is more expensive but often more effective. Unsupervised learning does not require manually labeled data and is often used for tasks such as clustering or dimensionality reduction. Reinforcement learning is different from the previous two. It is an algorithm that continuously optimizes its own strategy by continuously interacting with the environment in an uncertain environment and obtaining a large amount of data. The specific differences are as follows:

[0005] 1) The data obtained through interaction is not independent and identically distributed. Most machine learning algorithms assume that the data are independent of each other and have the same distribution, so that the model will converge during the training process. However, for reinforcement learning, the current decision may affect the distribution of subsequent data, so it is impossible to guarantee that the data are independent of each other, which will lead to unstable algorithm training.

[0006] 2) There is no strong label, and the current decision may not get immediate feedback. In reinforcement learning, there is often only a single signal based on the reward function, and there is no dedicated sample label like supervised learning. In addition, the agent may not be able to get immediate feedback from a single sample, and needs to constantly try and fail, which leads to reinforcement learning often requiring a long time to train.

[0007] 3) It is possible to surpass the upper limit of human performance. Supervised learning is based on training with manually labeled data, and its upper limit is also the upper limit of human performance. However, reinforcement learning starts from scratch and interacts with the environment, without human participation and is not affected by human prior knowledge. It is possible to surpass human performance. A typical example is that AlphaGo once defeated all the top human players in the Go game.

[0008] How to use reinforcement learning algorithms to solve the current automatic satellite alignment problems requires in-depth exploration and research by technical personnel. Summary of the invention

[0009] Purpose of the invention: To provide an automatic satellite alignment method based on reinforcement learning to solve the above-mentioned problems existing in the prior art.

[0010] Technical solution: Automatic satellite alignment method based on reinforcement learning, including:

[0011] Step 1: Collect data, construct a predetermined set of coordinate system-signal strength data, and construct a training data set and a test data set based on the coordinate system-signal strength data;

[0012] Step 2: construct a deep neural network module, configure a reward function, an observable state quantity, and an action space, use the training data set as input data, and train the deep neural network module for a predetermined number of times;

[0013] Step 3: Use the test data set to test the trained deep neural network module to obtain the final optimized deep neural network module.

[0014] According to one aspect of the present application, the reward function is: R = ΔS + 1 / (S max -S i );

[0015] In the formula, S max is the maximum signal strength; ΔS represents the change in the current signal, S i Indicates the strength of the current signal, i is a natural number.

[0016] According to one aspect of the present application, the observable state quantity includes: the currently observed signal strength S i , the action performed in the previous step, and the change in signal strength ΔS caused by the current action.

[0017] According to one aspect of the present application, the action space includes: an increase in the pitch angle by Δx°, a decrease in the pitch angle by Δx°, an increase in the azimuth angle by Δx°, and a decrease in the azimuth angle by Δx°.

[0018] According to one aspect of the present application, the process of collecting data includes the following steps:

[0019] Step 11: Start the terminal device and use the search algorithm to perform satellite alignment to obtain the maximum signal strength that can be obtained at the current coordinates;

[0020] Step 12: After obtaining the maximum signal strength of the current coordinates, suspend the search algorithm program, open the debugging interface, rotate the antenna, and obtain the coordinate parameters including the pitch angle and azimuth angle;

[0021] Step 13: Start the data acquisition program, scan the area within the predetermined range, record and store the maximum signal strength at each coordinate, and construct a signal strength heat map based on the coordinate system-signal strength data.

[0022] According to one aspect of the present application, the data preprocessing process is also included:

[0023] Step 14, reading a predetermined number of the signal strength heat maps, arbitrarily selecting two signal strength heat maps, dividing them into a number of image modules respectively by using a sliding window method, calculating the structural similarity of the corresponding image modules in the two signal strength heat maps, taking the average similarity of each module as the SSIM similarity of the two signal strength heat maps, and classifying the signal strength heat maps based on a preset threshold to form N groups of signal strength heat maps;

[0024] Step 15: Obtain each group of signal strength heat maps, fit the signal strength gradient distribution contour lines corresponding to the group of signal strength heat maps, obtain the number of local extreme value points and regional maximum value points of signal strength, and the coordinates corresponding to the local extreme value points and regional maximum value points;

[0025] Step 16: For each set of signal strength heat maps, calculate the feasible path from any point to the maximum point in the region and generate a feasible coordinate domain.

[0026] According to one aspect of the present application, it also includes:

[0027] Step 0: Based on weather conditions and environmental occlusion and reflection conditions, the data collection process is divided into several categories, and data is collected for each category.

[0028] According to one aspect of the present application, step 13 further includes obtaining the maximum signal strength at each coordinate and using a Bayesian network to analyze whether there is abnormal data:

[0029] During training, multiple terminal devices are used to collect the maximum signal strength of the current coordinate area at different times;

[0030] Input each set of coordinate data, maximum signal strength, satellite information, motor information, terminal equipment information, antenna information, frequency information, weather information and environmental information into the Bayesian network, and train and generate a Bayesian network structure;

[0031] When in use, each set of coordinate data and maximum signal strength information is input into the Bayesian network structure, and combined with the satellite information, motor information, terminal device information, antenna information and frequency information of the current terminal device, as well as the weather information and environmental information of the current terminal device, it is compared with the trained Bayesian network structure to determine whether the node data in the Bayesian network structure deviates from the normal value. If it deviates from the abnormal value, an abnormal value mark is given.

[0032] According to one aspect of the present application, a sliding window algorithm is used to statistically calculate the variance change of the maximum signal strength data corresponding to the coordinates, and whether the average value of the variance difference of several consecutive adjacent sliding windows is greater than a preset value is used as a criterion for judging whether there are abnormal values.

[0033] According to one aspect of the present application, step 14 also includes: when in use, after collecting a predetermined group of data and forming a local signal strength heat map, determining whether the maximum intensity of the signal at each coordinate is lower than the expected value; if the signal is lower than the expected value, marking it as a weak signal area; and using a signal geometric transformation enhancement method to adjust the maximum signal intensity at each coordinate.

[0034] Beneficial effects: This application greatly improves the efficiency of satellite alignment and improves work efficiency through an automatic satellite alignment system based on reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the topological structure of the present invention.

[0036] Figure 2a and Figure 2b is a signal heat map of different embodiments of the present invention.

[0037] Figure 3 It is a simplified diagram of the training and testing model of the present invention. DETAILED DESCRIPTION

[0038] like Figure 1 As shown, an automatic satellite alignment method based on reinforcement learning is provided, which mainly includes the following steps:

[0039] Step 1: Collect data, construct a predetermined set of coordinate system-signal strength data, and construct a training data set and a test data set based on the coordinate system-signal strength data;

[0040] Step 2: construct a deep neural network module, configure a reward function, an observable state quantity, and an action space, use the training data set as input data, and train the deep neural network module for a predetermined number of times;

[0041] The reward function is: R = ΔS + 1 / (S max -S i );

[0042] In the formula, S max is the maximum signal strength; ΔS represents the change in the current signal, S i Indicates the strength of the current signal, which is a natural number and represents the number of training times.

[0043] The observable state quantities include: the currently observed signal strength S i , the action performed in the previous step, and the change in signal strength ΔS caused by the current action;

[0044] The action space includes: pitch angle increase Δx°, pitch angle decrease Δx°, azimuth angle increase Δx°, and azimuth angle decrease Δx°.

[0045] Step 3: Use the test data set to test the trained deep neural network module to obtain the final optimized deep neural network module.

[0046] The optimized deep neural network module can greatly improve the speed of automatic satellite alignment and reduce manual workload. The efficiency of satellite alignment can be continuously improved by optimizing the number of training times and input data of the deep neural network module.

[0047] In order to obtain training data, it is necessary to collect input data in different scenarios at the beginning. Specifically, the data collection process is described as follows.

[0048] According to one aspect of the present application, the process of collecting data includes the following steps:

[0049] Step 11: Start the terminal device and use the search algorithm to perform satellite alignment to obtain the maximum signal strength that can be obtained at the current coordinates;

[0050] Step 12: After obtaining the maximum signal strength of the current coordinates, suspend the search algorithm program, open the debugging interface, rotate the antenna, and obtain the coordinate parameters including the pitch angle and azimuth angle;

[0051] Step 13: Start the data acquisition program, scan the area within the predetermined range, record and store the maximum signal strength at each coordinate, and construct a signal strength heat map based on the coordinate system-signal strength data.

[0052] In order to train the neural network, more training data is needed. Therefore, data is collected and stored in different states, scenes, and regions as subsequent training data. Generally speaking, based on weather conditions and environmental occlusion and reflection conditions, the data collection process is divided into several categories, and data is collected for each category.

[0053] For multi-dimensional massive data, if the matrix method is used to directly mine the data, there are at least two problems. One is that the data coupling leads to a large workload for training, and it is difficult to process the matrix into principal components or sparse matrix. The other is that it is impossible to intuitively display the distribution of signal strength and analyze the spatial distribution of signal strength as a whole. In other words, if the analysis starts from the data level, it will focus more on the processing of details and cannot better reflect the overall spatial and temporal distribution characteristics of the data.

[0054] In order to improve the quality of training data and the output effect of deep neural networks, data preprocessing is also required.

[0055] Therefore, according to one aspect of the present application, a data preprocessing method is also provided:

[0056] Step 14, read the predetermined signal strength heat map, randomly select two signal strength heat maps, divide them into several image modules respectively by sliding window method, calculate the structural similarity of the corresponding image modules in the two signal strength heat maps, and take the average similarity of each module as the SSIM similarity (SSIM refers to Structural Similarity Index measure) of the two signal strength heat maps. And classify the signal strength heat maps based on the preset threshold to form N groups of signal strength heat maps;

[0057] Step 15: Obtain each group of signal strength heat maps, fit the signal strength gradient distribution contour lines corresponding to the group of signal strength heat maps, obtain the number of local extreme value points and regional maximum value points of signal strength, and the coordinates corresponding to the local extreme value points and regional maximum value points;

[0058] Step 16: For each set of signal strength heat maps, calculate the feasible path from any point to the maximum point in the region and generate a feasible coordinate domain.

[0059] In other words, the distribution of signal strength is different under different weather conditions and environmental obstructions. Therefore, by collecting and classifying data in different scenarios, more targeted training data can be obtained.

[0060] For example, when the weather conditions are bad, the signal is weak overall, so the maximum signal strength in a certain area decreases to a certain extent. In the case of signal reflection, multiple regional maxima or maxima may appear. A two-dimensional hilly distribution is formed, such as two signal strength maximum extremes (which can be the same or different), forming a saddle-shaped distribution, so when aligning the satellite, the path will be very different. Therefore, in different situations, if the same algorithm is used, the efficiency of the neural network will be significantly different. More importantly, the deep neural network model trained in scenario A cannot achieve optimal efficiency when used in scenario B. Therefore, it is very important to distinguish the signal strength distribution in different scenarios, so as to conduct targeted training and obtain better use results.

[0061] In actual scenarios, the signal distribution caused by different factors may be similar. Therefore, by studying the signal strength heat map, the scenes can be classified, the classification of training data can be reduced, and faster training efficiency can be achieved.

[0062] More importantly, in this scenario, if the existing analysis method is used, it is generally to construct an input data matrix, and directly perform matrix analysis through a neural network to obtain the coupling relationship and causal relationship of the data, thereby generating a usable model. Due to the large differences in parameters in different scenarios, only data analysis cannot obtain the spatiotemporal distribution characteristics of the data, or the spatiotemporal structure of the data, nor can it display the spatiotemporal distribution characteristics of the data. Therefore, sticking to details will weaken the overall distribution characteristics in the current scenario.

[0063] Therefore, by representing the data as a spatiotemporal distribution heat map and then classifying it through image processing methods, the spatiotemporal distribution characteristics of the signal intensity data are obtained, and effective guidance is provided for subsequent training according to different categories.

[0064] According to experimental results, the deep neural network model obtained by this idea has faster training speed and better effect.

[0065] Due to differences in training data and scenarios, some training data may contain outliers, which will affect the training. In order to reduce the impact of outliers, you can analyze the structure of the data and pick out outlier data.

[0066] According to one aspect of the present application, step 13 further includes obtaining the maximum signal strength at each coordinate and using a Bayesian network to analyze whether there is abnormal data:

[0067] During training, multiple terminal devices are used to collect the maximum signal strength of the current coordinate area at different times;

[0068] Input each set of coordinate data, maximum signal strength, satellite information, motor information, terminal equipment information, antenna information, frequency information, weather information and environmental information into the Bayesian network, and train and generate a Bayesian network structure;

[0069] When in use, each set of coordinate data and maximum signal strength information is input into the Bayesian network structure, and combined with the satellite information, motor information, terminal device information, antenna information and frequency information of the current terminal device, as well as the weather information and environmental information of the current terminal device, it is compared with the trained Bayesian network structure to determine whether the node data in the Bayesian network structure deviates from the normal value. If it deviates from the abnormal value, an abnormal value mark is given.

[0070] According to one aspect of the present application, a sliding window algorithm is used to statistically calculate the variance change of the maximum signal strength data corresponding to the coordinates, and whether the average value of the variance difference of several consecutive adjacent sliding windows is greater than a preset value is used as a criterion for judging whether there are abnormal values.

[0071] In actual use, it is found that there are many reasons for abnormalities, which may be equipment problems, operational problems, or signal drift problems caused by the physical environment of space and time. Outliers often affect the accuracy of training and use. Therefore, if the outliers in the training data can be screened out, better training results can be obtained. If outliers can be found in time during use, the speed of satellite tracking can also be improved. Therefore, the present application constructs an outlier screening method based on a Bayesian network, which uses the Bayesian network to screen the training data and the data collected during use to obtain valid data. In this way, if a certain operation or parameter changes, resulting in a change in signal strength exceeding a predetermined value, it can be judged that an abnormal situation occurs in the window. Therefore, through this method, the data can be quickly and effectively processed to provide assistance for subsequent training.

[0072] According to one aspect of the present application, step 14 also includes: when in use, after collecting a predetermined group of data and forming a local signal strength heat map, determining whether the maximum intensity of the signal at each coordinate is lower than the expected value; if the signal is lower than the expected value, marking it as a weak signal area; and using a signal geometric transformation enhancement method to adjust the maximum signal intensity at each coordinate.

[0073] That is to say, after analyzing and classifying the signal thermal intensity distribution map, the topological structure of the signal intensity thermal map can be obtained. When in use, the topological structure is generated by a small amount of data, and then compared with the trained topological structure, the corresponding optimal path can be quickly found, so that a faster satellite alignment path can be obtained and the satellite alignment efficiency can be improved. By comparing the local satellite alignment data obtained at different time stages with several global data generated by training, the most similar situation can be found, and the optimal satellite alignment path in this situation can be obtained, which can greatly speed up the satellite alignment efficiency.

[0074] For example, in some cases, when using a trained deep neural network module for satellite alignment, in the first stage, the signal strength data and coordinate data obtained within 0.1 seconds can generate the topological structure of the current local area, and then calculate the similarity with the trained topological structure to find possible similar paths, thereby providing assistance for subsequent satellite alignment. For example, during the training process, a certain topological structure A, B, and C is obtained. During a certain satellite alignment process, the local topological structure is similar to A. Then, the optimal satellite alignment path corresponding to the topological structure A can be used for satellite alignment, which can quickly perform satellite alignment and reduce the alignment time. However, due to factors such as the scene, the signal strength distribution will present several situations. Once a similar topological structure appears, it can be carried out according to the trained satellite alignment path.

[0075] Embodiment 1

[0076] A Deep Q Network (DQN) algorithm is provided. The core of the algorithm is to optimize the Q function and use it to make decisions. t After that, the algorithm needs to estimate each action in the action space and use the action with the largest action value function as the strategy:

[0077] a t = arg max Q π (s t ,a)

[0078] In order to prevent oscillation after entering the local optimal solution, the algorithm uses the ε greedy algorithm to explore the environment. That is, before making a decision, the model has a probability of ε∈[0,1] to use a random strategy; only a probability of 1-ε will use the optimal strategy calculated by the model action value function.

[0079] The actions in the action space are defined as follows:

[0080] 1). Pitch angle increased by 0.1°

[0081] 2). Pitch angle decreased by 0.1°

[0082] 3). Azimuth angle increases by 0.1°

[0083] 4). Azimuth angle decreased by 0.1°

[0084] The observable status is as follows:

[0085] a). Current observed signal strength (db)

[0086] b). The action performed in the previous step

[0087] c) The change in signal strength caused by the current action

[0088] The reward function used is as follows:

[0089]

[0090] The maximum signal strength in this formula is an empirical value, which is usually set to be larger than the maximum signal strength obtained in actual measurement. That is, in the actual process of satellite alignment, the signal strength can never reach this value. ΔSingal Strength represents the change in the current signal.

[0091] The structure of the neural network used in this embodiment is as follows Figure 1 The input of the model is the observable state in the current environment, and the output is the action that the model believes should be taken in this state. In addition to the input and output layers, three fully connected hidden layers are added, with 256, 64, and 16 neurons respectively. The nonlinear activation function used is Relu. Compared with the traditional sigmoid function, Relu has the advantages of fast convergence and small amount of calculation. The output layer uses softmax as the activation function. The formula corresponding to the model is as follows:

[0092] H 1 =Relu(w 1 x+b 1 )

[0093] H 2 =Relu(w 2 H 1 +b 2 )

[0094] H 3 =Relu(w 3 H 2 +b 3 )

[0095] Y = Softmax(w 4 H 3 +b 4 )

[0096] Relu(α)=max(0,α)

[0097]

[0098] Where H i represents the output of hidden layer i, x represents the input of the model, and w i represents the weight matrix of each layer, b i It represents the bias vector of each layer, and Y represents the final output of the model.

[0099] The optimization algorithm used in this embodiment is Adam. Compared with the traditional SGD algorithm, Adam can accelerate the loss in the right direction when the model falls into a ravine and suppress oscillation. In addition, Adam can make an update after bias correction of the first-order and second-order momentum to ensure a relatively smooth iteration. The pseudo code of its optimization process is as follows:

[0100] Initialization: α (learning rate), β 1 , β 2 ∈[0, 1) (exponential decay rate of momentum), f(θ) (objective function with parameter θ), θ 0 (initial parameters), m 0 ←0 (initialize the first-order momentum to 0), v 0 ←0 (initialize the second-order momentum to 0), t←0 (initialize the timestamp to 0)

[0101] When θ t When it does not converge {t←t+1

[0102] #Get the current gradient

[0103] m t ←β 1 ·m t-1 +(1-β 1 )·g t #Update the bias of the first-order momentum

[0104] v t ←β 2 ·v t - 1 +(1-β 2 )·g t 2 #Update the bias of the second-order momentum

[0105] #Calculate the first-order momentum after bias correction

[0106] #Calculate the second-order momentum after bias correction

[0107] #Update parameters

[0108] }

[0109] Return to θ t .

[0110] The data used in this invention are public data. The evaluation index of satellite quality is signal strength (dB).

[0111] The resources used to collect data are shown in the following table:

[0112]

[0113] The collected signal strength heat map is shown in the attached Figure 2a and Figure 2b As shown: The coordinate system used here is the Earth North Sky coordinate system, the x-axis represents the azimuth, and 0.0° represents the angle of -35.7°; the y-axis represents the pitch angle, and 0.0° represents the angle of +35.9°. The maximum signal strength that the terminal can receive in this scenario is 74.9dB. In the initial environment, the system will randomly generate a coordinate within this range as the starting point.

[0114] The present invention trained 100 epochs, each containing 10,000 steps. The initial learning rate was set to 0.0001. The reinforcement learning framework used was Tianshou, a deep reinforcement learning platform based on pytorch developed by Tsinghua University. The platform modularizes each reinforcement learning algorithm, can interact and sample functions for any environment, and is also equipped with complete documentation, with low development difficulty. The framework used to build the simulation environment is OpenAI Gym.

[0115] The specific training process is shown in the figure: First, create two separate and independent environments for training and testing. Then create two collectors to interact with the two environments to collect training data and test data. Finally, put the collected training data into the model for training, and the training data is used for the final evaluation of the model.

[0116] 3. Training Results

[0117] The training results are shown in the following table:

[0118]

[0119] The test data in this table are the average values ​​of 100 simulated star alignment results. Each simulation performed 600 moves, and each move of 0.1° took 0.1s, so each star alignment process took about 1 minute. From the results, we can see that the algorithm has achieved a star alignment effect close to that of humans, and the speed can be faster if the number of training times is increased.

[0120] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. The automatic satellite alignment method based on reinforcement learning is characterized by: include: Step 1: Collect data, construct a predetermined set of coordinate system-signal strength data, and construct a training data set and a test data set based on the coordinate system-signal strength data; Step 2: construct a deep neural network module, configure a reward function, an observable state quantity, and an action space, use the training data set as input data, and train the deep neural network module for a predetermined number of times; Step 3: Use the test data set to test the trained deep neural network module to obtain the final optimized deep neural network module; The process of collecting data includes the following steps: Step 11: Start the terminal device and use the search algorithm to align the satellite to obtain the maximum signal strength that can be obtained at the current coordinates; Step 12: After obtaining the maximum signal strength of the current coordinates, suspend the search algorithm program, open the debugging interface, rotate the antenna, and obtain the coordinate parameters including the pitch angle and azimuth angle; Step 13: Start the data acquisition program, scan the area within the predetermined range, record and store the maximum signal strength at each coordinate, and construct a signal strength heat map based on the coordinate system-signal strength data; Step 14, reading a predetermined number of the signal strength heat maps, arbitrarily selecting two signal strength heat maps, dividing them into a number of image modules respectively by using a sliding window method, calculating the structural similarity of the corresponding image modules in the two signal strength heat maps, taking the average similarity of each module as the SSIM similarity of the two signal strength heat maps, and classifying the signal strength heat maps based on a preset threshold to form N groups of signal strength heat maps; Step 15: Obtain each group of signal strength heat maps, fit the signal strength gradient distribution contour lines corresponding to the group of signal strength heat maps, obtain the number of local extreme value points and regional maximum value points of signal strength, and the coordinates corresponding to the local extreme value points and regional maximum value points; Step 16: For each set of signal strength heat maps, calculate the feasible path from any point to the maximum point of the region and generate a feasible coordinate domain; The step 13 also includes obtaining the maximum signal strength at each coordinate and using a Bayesian network to analyze whether there is abnormal data: During training, multiple terminal devices are used to collect the maximum signal strength of the current coordinate area at different times; Input each set of coordinate data, maximum signal strength, satellite information, motor information, terminal equipment information, antenna information, frequency information, weather information and environmental information into the Bayesian network, and train and generate a Bayesian network structure; When in use, each set of coordinate data and maximum signal strength information is input into the Bayesian network structure, and combined with the satellite information, motor information, terminal device information, antenna information and frequency information of the current terminal device, as well as the weather information and environmental information of the current terminal device, it is compared with the trained Bayesian network structure to determine whether the node data in the Bayesian network structure deviates from the normal value. If it deviates from the abnormal value, an abnormal value mark is given.

2. The automatic satellite alignment method based on reinforcement learning according to claim 1, characterized in that: The reward function is: R = ΔS + 1 / (S max -S i ); In the formula, S max is the maximum signal strength; ΔS represents the change in the current signal, S i Indicates the strength of the current signal, i is a natural number.

3. The automatic satellite alignment method based on reinforcement learning as claimed in claim 2, characterized in that: The observable state quantity includes: the currently observed signal strength S i , the action performed in the previous step, and the change in signal strength ΔS caused by the current action.

4. The automatic satellite alignment method based on reinforcement learning according to claim 3, characterized in that: The action space includes: an increase in the pitch angle by Δx°, a decrease in the pitch angle by Δx°, an increase in the azimuth angle by Δx°, and a decrease in the azimuth angle by Δx°.

5. The automatic satellite alignment method based on reinforcement learning according to claim 1, characterized in that: Also includes: Step 0: Based on weather conditions and environmental occlusion and reflection conditions, the data collection process is divided into several categories, and data is collected for each category.

6. The automatic satellite alignment method based on reinforcement learning according to claim 1, characterized in that: The sliding window algorithm is used to count the variance change of the maximum signal strength data corresponding to the coordinates, and whether the average value of the variance difference of several consecutive adjacent sliding windows is greater than a preset value is used as the standard for judging whether there is an outlier.

7. The automatic satellite alignment method based on reinforcement learning according to claim 1, characterized in that: Step 14 also includes: when in use, after collecting a predetermined group of data and forming a local signal strength heat map, determining whether the maximum intensity of the signal at each coordinate is lower than the expected value; if the signal is lower than the expected value, marking it as a weak signal area; and using a signal geometric transformation enhancement method to adjust the maximum signal intensity at each coordinate.

Images