Flight deployment model pre-training method based on multi-modal data

Abstract

The invention provides a flight deployment model pre-training method based on multi-modal data, and the method comprises the steps: 1, obtaining the screen recording data of a control automation system, and obtaining the position information of an aircraft; obtaining the speed, course, height and flight number of the aircraft; step 2, acquiring voice data corresponding to the screen recording data of the control automation system, converting the voice data into characters, and extracting a control intention; 3, the position, speed, course, height and control intention of the aircraft are matched through flight numbers and combined into training data, the data are scored, and an air traffic control reinforcement learning evaluation model is trained; and step 4, training the air traffic control deep reinforcement learning flight deployment model. According to the method, the aircraft position information and the flight number in the control screen recording data and the control intention in the control voice data are extracted, manual evaluation is performed, and the flight deployment model is pre-trained, so that the training speed of the flight deployment model is increased, and the training effect of the flight deployment model is optimized.

Description

Technical Field

[0001] The present invention relates to the field of civil aviation air traffic control, and in particular to a flight dispatching model pre-training method based on multimodal data. Background Art

[0002] With the rapid development of the aviation transportation industry, the demand for daily travel and cargo transportation has increased rapidly. The average number of flights per day at busy airports is more than 1,000. In order to ensure the safe flight of aircraft, a certain degree of efficiency needs to be sacrificed, which will bring some other effects, such as the average flight time of flights being longer, the flight path deviating from the standard approach and departure routes, and the increased control pressure on controllers. Reinforcement learning is an important branch of machine learning. Its essence is to describe and solve the problem of how intelligent agents learn strategies in the process of interacting with the environment to maximize rewards or achieve specific goals. Therefore, we explore the method of using training deep reinforcement learning flight scheduling models for autonomous control.

[0003] There are many factors in deep reinforcement learning methods, such as slow training speed, low data utilization, and unstable training results. Deep reinforcement learning usually requires a lot of interaction and trial and error to learn effective strategies, especially when facing complex environments and tasks, which may result in the amount of data samples required for training being much higher than other machine learning methods. Deep reinforcement learning leads to instability in the model training process due to factors such as the dynamic balance of exploration and utilization, sparse or delayed rewards, and environmental non-stationarity, such as large fluctuations in the learning curve, slow convergence, or even no convergence. This is especially true for flight scheduling model training, which has a high trial and error cost. Therefore, using multiple modal data to pre-train the air traffic control deep reinforcement learning flight scheduling model and let it learn domain expert knowledge has practical significance for improving training speed and enhancing training stability. Summary of the invention

[0004] Purpose of the invention: The purpose of this method is to improve the training speed of the air traffic control deep reinforcement learning flight allocation model, improve the data utilization rate of the air traffic control deep reinforcement learning flight allocation model, and enhance the stability of the air traffic control deep reinforcement model.

[0005] This method is used in the training process of the air traffic control deep reinforcement learning flight allocation model. Through historical data in multiple modal forms, historical control experience is extracted, and an evaluation model is trained. The air traffic control deep reinforcement learning flight allocation model is pre-trained and combined with the evaluation model to improve the training speed, increase data utilization, and enhance model stability.

[0006] In order to achieve the above object, the present invention discloses a flight dispatch model pre-training method based on multimodal data, comprising the following steps:

[0007] Step 1: Obtain screen recording data from the control automation system, perform coordinate transformation operations through a deep neural network target detection model (reference: YOLOv7, Wang CY, Bochkovskiy A, Liao HY M. YOLOv7: Trainable bag-of-freebiessets new state-of-the-art for real-time object detectors[C] / / Proceedings ofthe IEEE / CVF conference on computer vision and pattern recognition.2023:7464-7475.) to obtain the position information of the aircraft on the screen; obtain the aircraft's speed, heading, altitude, flight number, and part of the control intention through a text recognition detection model (reference: PGNet, Wang P, Zhang C, Qi F, et al. Pgnet: Real-time arbitrarily-shaped textspotting with point gathering network[C] / / Proceedings of the AAAI Conference on Artificial Intelligence.2021,35(4):2782-2790.) or plan data matching;

[0008] Step 2: Obtain the speech data corresponding to the screen recording data of the control automation system, convert the speech data into text through the deep neural network speech recognition model (reference: Whisper, Radford A, Kim JW, Xu T, et al. Robust speech recognition via large-scale weak supervision [C] / / International conference on machine learning. PMLR, 2023: 28492-28518.), and extract the control intent through the intent extraction model (reference: CII-BERT, Yu Yaoyao, Wang Xuan, Chen Si, et al. Joint extraction method of air traffic control information and intent based on natural language processing [J]. Journal of Xihua University (Natural Science Edition), 2024.), and combine it with the control intent in the screen recording data;

[0009] Step 3, the position, speed, heading, altitude and control intention of the aircraft are matched by flight number to form training data, the data is scored (it can be scored manually), and the air traffic control reinforcement learning evaluation model is trained. The air traffic control reinforcement learning evaluation model includes a five-layer fully connected neural network, the input is the position, speed, heading, altitude and control intention of the aircraft, and the output is the data score;

[0010] Step 4: Use the training data and evaluation model to train the air traffic control deep reinforcement learning flight allocation model (reference: Liu Z, Xu Q, Shi Y, et al. Generation method of control strategy for aircraftsbased on hierarchical reinforcement learning[C] / / Artificial intelligence in China: proceedings of the 3rd international conference on artificial intelligence in China. Singapore: Springer Singapore, 2022: 109-116.).

[0011] In step 1, the control automation system screen recording data includes the flight number FN, altitude ALT, speed SPD, heading HDG, position POS and altitude and heading adjustment intentions.

[0012] In step 1, the formula for the coordinate transformation operation is:

[0013] P xy = T(P uv ) (1)

[0014] Where P xy Represents the coordinate information converted to longitude and latitude, T represents the plane coordinate conversion function, P uv Represents the target pixel center point output by the deep neural network target detection model used in step 1.

[0015] In step 1, the plan data matching is performed by calculating the track position, traversing the plan data, and searching for the plan of the nearest route. The formula is:

[0016] P true =Min(Dis(P xy ,P)) (2)

[0017] Where P trueRefers to the matching plan data, Min refers to the minimum value function, Dis refers to the distance calculation function, and P refers to all plan data.

[0018] In step 2, the voice data is converted into text through a deep neural network speech recognition model, and the formula is:

[0019] W=ASR(A) (3)

[0020] Where W is the converted text information, ASR represents the deep neural network speech recognition model, and A represents the original input speech.

[0021] In step 2, the voice data corresponds to the screen recording data time. The data alignment method uses the time of the voice data as the benchmark and the flight number as the matching condition to perform threshold judgment on the altitude, speed and heading values. The formula is:

[0022]

[0023] Match represents whether the voice and data are aligned in time. 1 represents alignment and 0 represents misalignment. new Represents the current altitude, speed or heading value, V old represents the altitude, speed or heading value at the last sampling moment, and θ represents the threshold.

[0024] In step 2, the control intention includes flight number, altitude adjustment, heading adjustment and speed adjustment intention. The control intention in the screen recording data and the control intention in the voice data are combined into the formula:

[0025]

[0026] A represents the final intention, A Voice Represents the intent extracted from the voice data, A Video Represents the intent extracted from the screen recording data. Represents empty intent.

[0027] In step 3, the altitude ALT, speed SPD, heading HDG and position POS obtained in step 1 are normalized to their maximum and minimum values ​​using the following formula:

[0028]

[0029] Where X′ represents the normalized data, X represents the data before normalization, and X min Represents the minimum value of all data in this category, X max Represents the maximum value among all data in this category.

[0030] In step 3, the data except the flight number in the control intent in step 2 is parsed into a one-hot encoding, and the format of the one-hot encoding is:

[0031] [o1,o2,o3,…,o n ] (7)

[0032] Among them, if o i =1, then the rest o j =0, i and j are 1 to n and j≠i, n represents the number of categories of control intent data; n The unique hot encoding of the nth category of data in the regulatory intent;

[0033] The one-hot encoded data and the normalized data are concatenated into a matrix as the input data of the air traffic control reinforcement learning evaluation model. The concatenation order is: [LON, LAT, ALT, SPD, HDG, O], where LON and LAT are the longitude and latitude in the position POS information, respectively, and O is the control intention data after one-hot encoding. The normalized data in step 1 and the one-hot encoded data in step 2 are preliminarily evaluated and scored by the automatic evaluation model. The scoring formula of the automatic evaluation model is:

[0034] S=c+W1T reduce -W2D route -W3C conflict (8)

[0035] Where c represents a constant value in the range [0-100]. When a serious conflict occurs, the value of c is small, and in other cases the value of c is 90. W1, W2, and W3 represent weight coefficients, T reduce Represents the time the flight time is reduced compared to the planned data, D route Represents the degree of deviation of the flight path from the original path, C conflict Represents the number of ordinary conflicts. Ordinary conflicts and severe conflicts are defined by different safety intervals. S is the score obtained by the evaluation model; the controller corrects the scoring result and normalizes the score as the label data of the air traffic control reinforcement learning evaluation model to construct a training data set, and the evaluation model is trained through the training data set.

[0036] In step 4, according to the matching data obtained in step 3, the data ranked high by score is screened, and the normalized [LON, LAT, ALT, SPD, HDG] in the matching data is used as input, and the control intention data O after one-hot encoding in the matching data is used as output. Supervised pre-training of deep reinforcement learning is performed, and the loss function uses the cross entropy loss function Loss, and the formula is:

[0037]

[0038] Where M represents the number of categories of regulatory intent; y ic is a sign function, which takes 1 when the true value of sample i is c, otherwise it takes 0; p ic represents the probability that the model output sample i is category c; when the accuracy reaches the threshold, the training process is stopped, the loss function is modified to the air traffic control reinforcement learning evaluation model, the network weights of the air traffic control deep reinforcement learning flight allocation model and the air traffic control reinforcement learning evaluation model are fixed, and exploration is carried out in a simulated training environment. When a certain amount of exploration data is accumulated, the weights of the air traffic control reinforcement learning evaluation model continue to be fixed, and the weights of the air traffic control deep reinforcement learning flight allocation model are started to be trained. The goal is to maximize the output of the air traffic control reinforcement learning evaluation model.

[0039] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0040] Beneficial effects: The present invention can effectively speed up the exploration and training of the flight dispatch model, improve the data utilization rate of the model in the exploration and training stages, and make the flight dispatch model training results closer to normal controller behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0042] Figure 1 It is a schematic diagram of the workflow of the present invention.

[0043] Figure 2 It is a schematic diagram of the screen recording data of the control automation system. DETAILED DESCRIPTION

[0044] like Figure 1 As shown, the present invention provides a flight dispatch model pre-training method based on multimodal data, comprising the following steps:

[0045] Step 1: Obtain the screen recording data of the control automation system, such as Figure 2 As shown in Figure 1, this type of data contains rich flight information, such as flight number FN, altitude ALT, speed SPD, heading HDG, position POS and other key parameters. Figure 2Taking the flight in the dotted box as an example, CXA8077 represents the flight number, 0886 represents the current flight altitude, 0978 represents the target altitude issued by the controller, 080 represents the speed, and P215 represents the next waypoint to be passed. The target altitude issued by the controller is the intention information. This information is extracted through the Pytesseract image text recognition model. The horizontal position of the aircraft is converted into coordinates through the deep neural network target tracking model. The formula is:

[0046] P xy =T(P uv ) (1)

[0047] Where P xy Represents the coordinate information converted to longitude and latitude, T represents the plane coordinate conversion function, P uv Represents the target pixel center point output by the target tracking model. The position information of the aircraft on the screen is obtained through coordinate conversion. The current position information and the latitude and longitude information of the next point can be used to obtain the current heading of the aircraft. When the relevant information is missing on the sign, the flight information of the flight can be obtained using the planned data matching method. The planned data matching method is to match by calculating the track position, traversing the planned data, and finding the plan of the nearest route. The formula is as follows:

[0048] P true =Min(Dis(P xy ,P)) (2)

[0049] Where P true Refers to the matching plan data, Min refers to the minimum value function, Dis refers to the distance calculation function, P cy Refers to the longitude and latitude coordinates after coordinate conversion, and P refers to all planning data. This information will be used as the input of the air traffic control reinforcement learning evaluation model and the input of the air traffic control deep reinforcement learning flight allocation model;

[0050] Step 2: Collect voice data corresponding to the screen recording time of the control automation system at the same time, use a deep neural network speech recognition algorithm, such as the Whisper model, to process the voice data and convert it into text data. The formula is as follows:

[0051] W=ASR(A) (3)

[0052] Where W is the converted text information, ASR represents the neural network speech recognition model, and A represents the original input voice. After obtaining the text information, the template matching and deep neural network natural language processing algorithm are used to extract the key command information such as the flight number FN, altitude adjustment value ALTA, heading adjustment value HDGA, speed adjustment SPDA, and route adjustment RA in the text data. For example, the following command: "White Egret Eight Holes Turn Turn Up to Nine Turns Keep", where "White Egret" can be converted into "CXA", "Eight Holes Turn Turn Turn" can be converted into "8077", the flight number information is "CXA8077", the command type is "altitude adjustment", and "Nine Turns" can be converted into "9700" according to the air traffic control standard, which means that the altitude is adjusted to 9700 meters.

[0053] Step 3: Match the data of step 1 and step 2 according to the flight numbers extracted in step 1 and step 2. Since the flight number is unique within the same time period, it can be used as a basis for matching. At the same time, time alignment is performed according to the speed change time in the screen recording data according to the following formula:

[0054]

[0055] Match represents whether the voice and data are aligned in time, 1 represents alignment, and 0 represents misalignment. new Represents the current altitude / speed / heading value, V old represents the altitude / speed / heading value at the last sampling moment, and θ represents the threshold value, which is set to 50 in this method. At the same time, the control intention is combined according to the following formula:

[0056]

[0057] A represents the final intention, A Voice Represents the intent extracted from the voice data, A Video Represents the intent extracted from the screen recording data. Represents the empty intention. Because the control instruction already has a clear intention value of adjusting the altitude to 9700 meters, the intention extracted from the control instruction is used as the final intention.

[0058] Normalize the data in step 1, including altitude ALT, speed SPD, heading HDG and position POS, to the maximum and minimum values, and adjust the data range to the range of 0 to 1. The formula is as follows:

[0059]

[0060] Where X′ represents the normalized data, X represents the data before normalization, and X min Represents the minimum value among all data in this category.

[0061] Parse the control instruction category and instruction value data in step 2 into one-hot encoding. The format of one-hot encoding is:

[0062] [o1,o2,o3,…,o n ] (7)

[0063] If o i =1, then the rest o j =0, (j≠i). The normalized data and the one-hot encoded data are concatenated into a matrix as the input data of the air traffic control reinforcement learning evaluation model. The concatenation order is: [LON, LAT, ALT, SPD, HDG, O], where LON and LAT are the longitude and latitude in the position POS information, and O is the one-hot encoded information. The original video data and voice data obtained in step 1 and step 2 are preliminarily evaluated and scored by the automatic evaluation model. The scoring formula of the automatic evaluation model is:

[0064] S=c+W1T reduce -W2D route -W3C conflict (8)

[0065] Where c represents a constant value in the range [0-100]. When a serious conflict occurs, c is set to 20, and in other cases, the value of c is 90. W1, W2, and W3 represent weight coefficients, which are initially set to 1, 0.2, and 10. reduce Represents the time the flight time is reduced compared to the planned data, D route Represents the degree of deviation of the flight path from the original path, C conflict Represents the number of times ordinary conflicts occur. Ordinary conflicts and severe conflicts are defined by different safety intervals. When the horizontal interval between aircraft is less than 3 nautical miles, it is considered a severe conflict. When the horizontal interval between aircraft is greater than 3 nautical miles and less than 5 nautical miles, it is considered an ordinary conflict. The scores are handed over to the training instructor for correction, and the scores are normalized to the maximum and minimum values ​​as label data for the air traffic control reinforcement learning evaluation model. In this way, an evaluation model training data set is constructed. These data will be used to train the evaluation model so that it can accurately evaluate the rationality and actual effect of control decisions. A five-layer fully connected neural network is used to construct an air traffic control reinforcement learning evaluation model, and the evaluation model is used for multiple rounds of training until the loss decreases and stabilizes, and the training accuracy is greater than 98%.

[0066] Step 4: According to the matching data obtained in step 3, the top 30% of the data ranked by score are screened, and the normalized aircraft information data [LON, LAT, ALT, SPD, HDG] in the matching data, such as position POS, speed SPD, heading HDG, altitude ALT, etc., are used as input, and the control intention data O after unique hot encoding in the matching data is used as output to perform supervised pre-training on deep reinforcement learning. The loss function uses the cross entropy loss function, and the formula is:

[0067]

[0068] Where M represents the number of categories of regulatory intent, y ic is a sign function, which takes 1 when the true value of sample i is c, otherwise it takes 0, p ic The probability that the model output sample i is category c; when the accuracy rate reaches more than 70%, the training process is stopped, the loss function is modified to the air traffic control reinforcement learning evaluation model, the network weights of the air traffic control deep reinforcement learning flight allocation model and the air traffic control reinforcement learning evaluation model are fixed, and exploration is carried out in a simulated training environment. When the exploration accumulates a certain amount of data, the weights of the air traffic control reinforcement learning evaluation model continue to be fixed, and the weights of the air traffic control deep reinforcement learning flight allocation model begin to be trained. The goal is to maximize the output of the air traffic control reinforcement learning evaluation model. When the score of the air traffic control deep reinforcement learning flight allocation model rises and tends to stabilize, stop training.

[0069] After using this method, compared with the ordinary air traffic control deep reinforcement learning flight allocation model training method based on autonomous exploration, the time required for the autonomous exploration stage of the flight allocation model in the training process is greatly shortened, and artificial experience is used to improve the convergence speed of the flight allocation model, reduce the unreasonable output actions of the flight allocation model, and make the output actions of the flight allocation model more in line with the operating habits of the controller.

[0070] The present invention provides a flight dispatch model pre-training method based on multimodal data. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A flight dispatch model pre-training method based on multimodal data, characterized in that: The following steps are involved: Step 1: Obtain screen recording data from the control automation system, perform coordinate conversion operations through a deep neural network target detection model, and obtain the position information of the aircraft on the screen; Obtain the aircraft's speed, heading, altitude, flight number and part of the control intention through text recognition detection model or plan data matching; Step 2: Acquire the voice data corresponding to the screen recording data of the control automation system, convert the voice data into text through the deep neural network speech recognition model, extract the control intent through the intent extraction model, and combine it with the control intent in the screen recording data; Step 3, the position, speed, heading, altitude and control intention of the aircraft are matched by flight number to form training data, the data is scored, and an air traffic control reinforcement learning evaluation model is trained. The air traffic control reinforcement learning evaluation model includes a five-layer fully connected neural network, the input is the position, speed, heading, altitude and control intention of the aircraft, and the output is the data score; Step 4: Use the training data and evaluation model to train the air traffic control deep reinforcement learning flight dispatch model.

2. The method according to claim 1, characterized in that In step 1, the control automation system screen recording data includes the flight number FN, altitude ALT, speed SPD, heading HDG, position POS and altitude and heading adjustment intention.

3. The method according to claim 2, characterized in that In step 1, the formula for the coordinate transformation operation is: P xy = T(P uv ) (1) Where P xy Represents the coordinate information converted to longitude and latitude, T represents the plane coordinate conversion function, P uv Represents the target pixel center point output by the deep neural network target detection model used in step 1.

4. The method according to claim 3, characterized in that In step 1, the plan data matching is performed by calculating the track position, traversing the plan data, and searching for the plan of the nearest route. The formula is: P true = Min(Dis(P xy ,P)) (2) Where P true Refers to the matching plan data, Min refers to the minimum value function, Dis refers to the distance calculation function, and P refers to all plan data.

5. The method according to claim 4, characterized in that In step 2, the voice data is converted into text through a deep neural network speech recognition model, and the formula is: W= ASR(A) (3) Where W is the converted text information, ASR represents the deep neural network speech recognition model, and A represents the original input speech.

6. The method according to claim 5, characterized in that In step 2, the voice data corresponds to the screen recording data time. The data alignment method uses the time of the voice data as the benchmark and the flight number as the matching condition to perform threshold judgment on the altitude, speed and heading values. The formula is: Match represents whether the voice and data are aligned in time. 1 represents alignment and 0 represents misalignment. new Represents the current altitude, speed or heading value, V old represents the altitude, speed or heading value at the last sampling moment, and θ represents the threshold.

7. The method according to claim 6, characterized in that In step 2, the control intention includes flight number, altitude adjustment, heading adjustment and speed adjustment intention. The control intention in the screen recording data and the control intention in the voice data are combined into the formula: A represents the final intention, A Voice Represents the intent extracted from the voice data, A Video Represents the intent extracted from the screen recording data. Represents empty intent.

8. The method according to claim 7, characterized in that In step 3, the altitude ALT, speed SPD, heading HDG and position POS obtained in step 1 are normalized to their maximum and minimum values ​​using the following formula: Where X ′ represents the normalized data, X represents the data before normalization, and X min Represents the minimum value of all data in this category, X max Represents the maximum value among all data in this category.

9. The method according to claim 8, characterized in that In step 3, the data except the flight number in the control intent in step 2 is parsed into a one-hot encoding, and the format of the one-hot encoding is: [o1,o2,o3,…,o n ](7) Among them, if o i =1, then the rest o j =0, i and j are 1 to n and j≠i, n represents the number of categories of control intent data; o n The unique hot encoding of the nth category of data in the regulatory intent; The one-hot encoded data and the normalized data are concatenated into a matrix as the input data of the air traffic control reinforcement learning evaluation model. The concatenation order is: [LON, LAT, ALT, SPD, HDG, O], where LON and LAT are the longitude and latitude in the position POS information, and O is the control intention data after one-hot encoding. The normalized data in step 1 and the one-hot encoded data in step 2 are preliminarily evaluated and scored by the automatic evaluation model. The scoring formula of the automatic evaluation model is: S=c+W1T reduce -W2D route -W3C conflict (8) Where c represents a constant value in the range [0-100]. When a serious conflict occurs, the value of c is small, and in other cases the value of c is 90; W1, W2, and W3 represent weight coefficients; T reduce Represents the time the flight time is reduced compared to the planned data, D route Represents the degree of deviation of the flight path from the original path, C conflict represents the number of ordinary conflicts; ordinary conflicts and serious conflicts are defined by different safety intervals; S is the score obtained by the evaluation model; The scoring results are corrected and normalized, and used as label data for the air traffic control reinforcement learning evaluation model to construct a training data set, and the evaluation model is trained using the training data set.

10. The method according to claim 9, characterized in that In step 4, according to the matching data obtained in step 3, the data ranked high by score is screened, and the normalized [LON, LAT, ALT, SPD, HDG] in the matching data is used as input, and the control intention data O after one-hot encoding in the matching data is used as output. Supervised pre-training of deep reinforcement learning is performed, and the loss function uses the cross entropy loss function Loss, and the formula is: Where M represents the number of categories of regulatory intent; y ic is a sign function, which takes 1 when the true value of sample i is c, otherwise it takes 0; p ic represents the probability that the model output sample i is category c; when the accuracy reaches the threshold, the training process is stopped, the loss function is modified to the air traffic control reinforcement learning evaluation model, the network weights of the air traffic control deep reinforcement learning flight allocation model and the air traffic control reinforcement learning evaluation model are fixed, and exploration is carried out in a simulated training environment. When a certain amount of exploration data is accumulated, the weights of the air traffic control reinforcement learning evaluation model continue to be fixed, and the weights of the air traffic control deep reinforcement learning flight allocation model are started to be trained. The goal is to maximize the output of the air traffic control reinforcement learning evaluation model.

Images