Civil aviation freight rate anomaly detection method, system and device and storage medium

Abstract

The invention discloses a civil aviation freight rate anomaly detection method, system and device and a storage medium, and the method comprises the steps: obtaining original freight rate data, creating a historical freight rate data training set, carrying out the feature extraction of the historical freight rate data training set, obtaining a plurality of pieces of freight rate feature data, building an optimal freight rate feature data set based on the plurality of pieces of freight rate feature data, and carrying out the detection of the freight rate anomaly. And constructing an integrated anomaly detection model based on the optimal freight rate feature data set, performing anomaly detection on the real-time freight rate data, and outputting a detection result. According to the invention, comprehensive detection of absolute anomaly and relative anomaly is realized by integrating multiple baseline models, and intelligent processing of anomaly detection can be realized by introducing multiple models. In addition, according to the integrated model provided by the invention, the anomaly detection performance is improved under the condition of balancing the accuracy and the recall rate, and the false alarm rate and the missing report rate are reduced. Compared with a traditional civil aviation freight rate anomaly detection system, the method can achieve the real-time anomaly detection, and relieves the manual checking pressure.

Description

Technical Field

[0001] The present invention belongs to the technical field of real-time monitoring of air ticket prices, and in particular relates to a method, system, device and storage medium for detecting abnormal civil aviation fares. Background Art

[0002] With the continuous development of the global aviation market, the civil aviation fare system has become increasingly complex, covering a variety of factors such as different airlines, routes, cabin classes, seasonal discounts, and special passenger groups. The interweaving of these complex factors makes the management of civil aviation fares extremely difficult. However, due to system errors, operational errors, or data anomalies, abnormal fluctuations in civil aviation fares occur from time to time, seriously affecting the revenue of airlines and agents, and causing trouble to passengers. In civil aviation fare management, airlines usually dynamically adjust flight fares based on factors such as market demand, competition, and cost. However, with the increase in the number of flights and the frequency of fare adjustments, the amount of fare data has also increased exponentially. Faced with such a huge amount of data, traditional fare management systems seem to be unable to monitor and detect fare anomalies. Existing civil aviation fare anomaly detection technologies mainly include the following: 1. Manual inspection, that is, fare management personnel manually check the rationality of fare data. This method relies on the experience and expertise of the staff, is inefficient and prone to missing anomalies, and cannot monitor and process large-scale data in real time. Second, the rule engine, that is, anomaly detection based on predefined rules, such as setting upper and lower limits of prices, fluctuation ranges, etc. When the price exceeds the preset range, the system will trigger an alarm. Although this method is simple to implement and suitable for certain specific scenarios, the patterns of freight rate anomalies are diverse and complex, and fixed rules are difficult to cover all situations. Third, statistical analysis: that is, using basic statistical methods, using historical data and maximum likelihood estimation to calculate the mean and standard deviation, fitting a Gaussian distribution model, and setting a certain probability density as a threshold to detect abnormal data that deviates from the normal range (below the threshold). The above methods all have obvious limitations, which are specifically manifested in the following aspects: First, low efficiency and poor real-time performance: Faced with huge freight rate data, manual inspection is not only time-consuming and labor-intensive, but also easy to miss anomalies, and it is impossible to achieve real-time monitoring of freight rates. Second, high false alarm and missed alarm rates: Traditional freight rate anomaly detection technology is limited by manual omissions, low rule coverage, and limited detection capabilities of statistical analysis, resulting in frequent false alarms and missed alarms. Third, lack of intelligent processing capabilities: With the development of big data and artificial intelligence technologies, existing technologies have failed to fully utilize these new technologies to detect freight rate anomalies, making it difficult to cope with the dynamically changing market environment and complex freight rate adjustment strategies, and unable to dynamically update to adapt to complex and changing situations. Summary of the invention

[0003] To solve the above problems, the present invention provides a civil aviation fare anomaly detection method, system, device and storage medium to solve the problems of low efficiency, poor real-time performance, high false alarm and missed alarm rates, and difficulty in coping with dynamically changing market environments and complex fare adjustment strategies in existing civil aviation fare anomaly detection methods.

[0004] A method for detecting abnormal civil aviation freight rates, comprising:

[0005] Obtain original freight rate data and create a training set of historical freight rate data;

[0006] Perform feature extraction on the historical freight rate data training set to obtain multiple freight rate feature data;

[0007] Construct an optimal freight rate feature data set based on multiple freight rate feature data;

[0008] Build an integrated anomaly detection model based on the optimal freight rate feature dataset;

[0009] Based on the integrated anomaly detection model, anomaly detection is performed on real-time freight rate data and the detection results are output.

[0010] According to a specific embodiment of the present invention, the original fare data includes fare rule data and transaction log data, wherein the transaction log data includes field information of passengers, fares and flights.

[0011] According to a specific embodiment of the present invention, obtaining original freight rate data and creating a historical freight rate data training set includes:

[0012] Obtain transaction log data from the original freight rate data and parse it to obtain freight rate parsing data;

[0013] A historical freight rate data training set is constructed based on the freight rate analysis data, where the freight rate analysis data includes flight information data, passenger information data and freight rate information data.

[0014] According to a specific embodiment of the present invention, feature extraction is performed on a historical freight rate data training set to obtain a plurality of freight rate feature data including:

[0015] Screen and clean the original freight rate data in the historical freight rate data training set to obtain the original freight rate processing data;

[0016] A plurality of fare characteristic data are extracted from the original fare processing data, including flight characteristic data, passenger characteristic data and fare characteristic data.

[0017] According to a specific embodiment of the present invention, constructing an optimal freight rate feature data set based on multiple freight rate feature data includes:

[0018] Characteristic coding is performed on a plurality of freight rate characteristic data to obtain a plurality of freight rate characteristic coding data;

[0019] Based on multiple freight rate feature coding data, feature reorganization is performed to obtain multiple freight rate feature combination data;

[0020] Construct the optimal freight rate feature dataset based on multiple freight rate feature combination data.

[0021] According to a specific embodiment of the present invention, multiple freight rate feature data are feature-coded to obtain multiple freight rate feature-coded data including:

[0022] The frequency coding method is used to perform feature coding on multiple freight rate feature data to obtain corresponding freight rate feature values.

[0023] According to a specific embodiment of the present invention, building an integrated anomaly detection model based on the optimal freight rate feature data set includes:

[0024] Build multiple basic anomaly detection models based on the optimal freight rate feature dataset;

[0025] Evaluate the comprehensive precision and recall of basic anomaly detection models and determine the corresponding unsupervised baseline models;

[0026] Multiple unsupervised baseline models are weighted and integrated to obtain an integrated anomaly detection model;

[0027] Among them, the basic anomaly detection models include an anomaly detection model based on the K-nearest neighbor algorithm, an anomaly detection model based on the single-class support vector machine algorithm, and an anomaly detection model based on the Gaussian mixture model.

[0028] According to a specific embodiment of the present invention, multiple unsupervised baseline models are weighted and integrated to obtain an integrated anomaly detection model including:

[0029] The recall rate of the unsupervised baseline model is set as the weight according to the actual business impact factors;

[0030] Integrate multiple unsupervised baseline models with set weights to obtain an integrated anomaly detection model;

[0031] The anomaly index of the integrated anomaly detection model is determined based on the weight, and the output with an anomaly index higher than the anomaly threshold is regarded as an anomaly point.

[0032] According to a specific embodiment of the present invention, determining anomaly indicators of an integrated anomaly detection model based on weights includes:

[0033] The anomaly score outputs of the unsupervised baseline model are averaged according to the recall weights to obtain the integrated anomaly index.

[0034] According to a specific embodiment of the present invention, the method further includes:

[0035] Early warning and classification are carried out based on the test results, and the test results are visualized on the client.

[0036] According to a specific embodiment of the present invention, the method further includes:

[0037] After the anomaly detection is completed, the detection log information is recorded and stored.

[0038] A civil aviation freight rate anomaly detection system, comprising:

[0039] Data acquisition module, used to obtain original freight rate data and create a historical freight rate data training set;

[0040] A feature extraction module is used to extract features from a historical freight rate data training set to obtain multiple freight rate feature data;

[0041] A feature data set construction module, used to construct an optimal freight rate feature data set based on multiple freight rate feature data;

[0042] Integrated model building module, used to build an integrated anomaly detection model based on the optimal freight rate feature data set;

[0043] The anomaly detection module is used to perform anomaly detection on real-time freight rate data based on the integrated anomaly detection model and output the detection results.

[0044] According to a specific embodiment of the present invention, the feature extraction module further includes:

[0045] The data processing module is used to screen and clean the original freight rate data in the historical freight rate data training set to obtain the original freight rate processing data;

[0046] The feature screening module is used to extract multiple fare feature data from the original fare processing data, including flight feature data, passenger feature data and fare feature data.

[0047] According to a specific embodiment of the present invention, the feature data set construction module further includes:

[0048] A feature coding module, used for performing feature coding on a plurality of freight rate feature data to obtain a plurality of freight rate feature coding data;

[0049] A feature reorganization module is used to perform feature reorganization based on multiple freight rate feature coding data to obtain multiple freight rate feature combination data;

[0050] The optimal feature data set construction module is used to construct the optimal freight rate feature data set based on multiple freight rate feature combination data.

[0051] According to a specific embodiment of the present invention, the integrated model building module includes:

[0052] Basic model building module, used to build multiple basic anomaly detection models based on the optimal freight rate feature data set;

[0053] The baseline model building module is used to evaluate the comprehensive precision and recall of the basic anomaly detection model and determine the corresponding unsupervised baseline model;

[0054] An integration module is used to weight and integrate multiple unsupervised baseline models to obtain an integrated anomaly detection model;

[0055] Among them, the basic anomaly detection models include an anomaly detection model based on the K-nearest neighbor algorithm, an anomaly detection model based on the single-class support vector machine algorithm, and an anomaly detection model based on the Gaussian mixture model.

[0056] According to a specific embodiment of the present invention, the integrated module further includes:

[0057] The weight setting module is used to set the recall rate of the unsupervised baseline model as a weight according to the actual business influencing factors;

[0058] A model integration module is used to integrate multiple unsupervised baseline models with set weights to obtain an integrated anomaly detection model;

[0059] The indicator setting module is used to determine the abnormal indicator of the integrated anomaly detection model based on the weight, and take the output of the abnormal indicator higher than the abnormal threshold as the abnormal point.

[0060] According to a specific embodiment of the present invention, the system further includes:

[0061] The warning and display module is used to issue warnings and classify according to the detection results, and to visualize the detection results on the client.

[0062] According to a specific embodiment of the present invention, the system further includes:

[0063] The data storage module is used to record and store detection log information after completing anomaly detection.

[0064] A computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned civil aviation fare anomaly detection method.

[0065] A computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the above-mentioned civil aviation fare anomaly detection method.

[0066] Compared with the prior art, the method, system, device and storage medium for detecting abnormal civil aviation fares provided by the present invention have the following advantages:

[0067] The present invention first collects and captures the freight rate data stream to obtain a training data set, and then screens and efficiently models the important features that affect civil aviation freight rates to ensure that the model accurately reflects the law of price fluctuations. In order to cope with the problem of lack of labeled data, the present invention designs an unsupervised anomaly detection method, determines the model and parameters used, realizes preliminary freight rate anomaly detection, and then realizes the integration of multiple unsupervised algorithms by setting anomaly indicators and thresholds with different weight strategies, thereby improving the timeliness and accuracy of detection. The integrated model proposed in the present invention improves the anomaly detection performance while balancing the accuracy and recall rate, and reduces the false alarm rate and missed alarm rate. By integrating multiple baseline models, comprehensive detection of absolute anomalies and relative anomalies is achieved. At the same time, the system does not limit the selection and number of baseline models, and intelligent processing of anomaly detection can be achieved by introducing multiple models. The present invention solves the problem of low efficiency and poor real-time performance of traditional civil aviation freight rate anomaly detection technology, can realize real-time anomaly detection, and relieves the pressure of manual verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0069] Figure 1 The present invention is a flowchart of a method for detecting abnormal civil aviation fares according to an embodiment of the present invention.

[0070] Figure 2 The present invention is a flowchart of a method for obtaining original freight rate data and creating a training set of historical freight rate data according to an embodiment of the present invention.

[0071] Figure 3 The present invention is a flowchart of a method for extracting features from a historical freight rate data training set according to an embodiment of the present invention.

[0072] Figure 4 It is a flow chart of a method for constructing an optimal freight rate feature data set according to an embodiment of the present invention.

[0073] Figure 5 It is a flow chart of a method for building an integrated anomaly detection model according to an embodiment of the present invention.

[0074] Figure 6 It is a flow chart of a method for weighting and integrating multiple unsupervised baseline models provided according to an embodiment of the present invention.

[0075] Figure 7The figure is a schematic diagram of the overall process of detecting abnormal civil aviation fares according to an embodiment of the present invention.

[0076] Figure 8 It is a schematic diagram of the overall process of building an integrated anomaly detection model according to an embodiment of the present invention.

[0077] Fig. 9 It is a schematic diagram of the structure of a civil aviation fare anomaly detection system provided according to an embodiment of the present invention.

[0078] Fig.10 It is a schematic diagram of the structure of a feature extraction module provided according to an embodiment of the present invention.

[0079] Fig.11 It is a schematic diagram of the structure of a feature data set construction module provided according to an embodiment of the present invention.

[0080] Fig.12 It is a schematic diagram of the structure of an integrated model building module provided according to an embodiment of the present invention.

[0081] Fig.13 It is a schematic diagram of the integrated module structure provided according to an embodiment of the present invention.

[0082] Fig.14 The figure is a schematic diagram of the data structure of a civil aviation fare anomaly detection system provided according to an embodiment of the present invention.

[0083] Fig.15 It is a schematic diagram of the structure of a computer device provided according to an embodiment of the present invention.

[0084] Reference numerals:

[0085] 01-Data acquisition module; 02-Feature extraction module; 03-Feature data set construction module; 04-Integrated model construction module; 05-Anomaly detection module; 06-Warning and display module; 07Data storage module;

[0086] 021-data processing module; 022-feature screening module;

[0087] 031-feature encoding module; 032-feature recombination module; 033-optimal feature data set construction module;

[0088] 041-basic model building module; 042-baseline model building module; 043-integration module;

[0089] 0432-Weight setting module; 0432-Model integration module; 0433-Indicator setting module. DETAILED DESCRIPTION

[0090] In order to make those skilled in the art understand the concept and thought of the present invention more clearly, the present invention is described in detail below in conjunction with specific embodiment.It should be understood that the embodiment provided herein is only a part of all embodiments that the present invention may have.Those skilled in the art, after reading the specification of the application, have the ability to make improvements, transformations, or replacements to part or the whole of the following embodiments, and these improvements, transformations, or replacements are also included in the scope of the present invention.

[0091] In this article, the terms "first", "second" and other similar words are not intended to imply any order, quantity and importance, but are only used to distinguish different elements. In this article, the terms "one", "an" and other similar words are not intended to indicate that there is only one thing, but to indicate that the relevant description is only for one of the things, and the thing may have one or more. In this article, the terms "comprise", "include" and other similar words are intended to indicate logical mutual relations, and cannot be regarded as indicating spatial structural relations. For example, "A includes B" is intended to indicate that B belongs to A logically, but does not mean that B is located inside A in space. In addition, the meanings of the terms "comprise", "include" and other similar words should be regarded as open, rather than closed. For example, "A includes B" is intended to indicate that B belongs to A, but B does not necessarily constitute the whole of A, and A may also include other elements such as C, D, and E.

[0092] In this document, the terms "embodiment", "this embodiment", "one embodiment", and "an embodiment" do not mean that the relevant description is only applicable to a specific embodiment, but rather that the description may also be applicable to one or more other embodiments. Those skilled in the art should understand that in this document, any description of a certain embodiment can be replaced, combined, or combined in other ways with the relevant descriptions in one or more other embodiments, and the new embodiments generated by the replacement, combination, or combination in other ways are easily conceivable by those skilled in the art and fall within the scope of protection of the present invention.

[0093] Example 1

[0094] Additional aspects and advantages of embodiments of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through practice of embodiments of the present invention.

[0095] Combination Figure 1-Figure 8 The embodiment of the present invention provides a method for detecting abnormal civil aviation fares, comprising:

[0096] S1: Obtain original freight rate data and create a historical freight rate data training set.

[0097] S2: Extract features from the historical freight rate data training set to obtain multiple freight rate feature data.

[0098] S3: Construct an optimal freight rate feature data set based on multiple freight rate feature data.

[0099] S4: Build an integrated anomaly detection model based on the optimal freight rate feature dataset.

[0100] S5: Perform anomaly detection on real-time freight rate data based on the integrated anomaly detection model and output the detection results.

[0101] S6: Provide early warning and classification based on the detection results, and visualize the detection results on the client.

[0102] S7: After the anomaly detection is completed, the detection log information is recorded and stored.

[0103] Since abnormal price detection in civil aviation involves multiple feature dimensions and has high real-time requirements, and faces the problem of zero start of labeled data, the present invention first screens the important influencing factors of abnormal prices, realizes efficient modeling of features for civil aviation scenarios, and provides preliminary detection services for the freight rate system by designing and screening unsupervised anomaly detection methods, and then uses a scheme of integrated learning of multiple unsupervised algorithms to achieve a balance between the timeliness and accuracy of anomaly detection. The present invention first collects and captures the freight rate data stream to obtain a training data set. Then, the important features that affect civil aviation freight rates are screened and efficiently modeled to ensure that the model accurately reflects the law of price fluctuations. In order to cope with the problem of lack of labeled data, the present invention designs an unsupervised anomaly detection method, tries a variety of algorithms based on statistics, neighborhoods, and classification, and finally determines the model and parameters used to realize preliminary freight rate anomaly detection. After that, the integration of multiple unsupervised algorithms is realized by setting abnormal indicators and thresholds with different weight strategies, thereby improving the timeliness and accuracy of detection.

[0104] Specifically, step S1 of obtaining original freight rate data and creating a historical freight rate data training set includes:

[0105] S11: Acquire and parse the transaction log data from the original freight rate data to obtain freight rate parsing data.

[0106] In an embodiment of the present invention, the original fare data includes fare rule data and transaction log data, wherein the transaction log data includes field information of passengers, fares and flights. The present invention obtains transaction log data streams and fare rule data from the fare system respectively to form original log data. By parsing the transaction log data, fare parsing data containing fare information, flight information and passenger information can be obtained. For example, flight information, passenger information and fare information are extracted from the transaction log, and the original fare feature data obtained by the above parsing is pushed to the anomaly detection server for further processing and analysis.

[0107] S12: constructing a historical freight rate data training set based on the freight rate analysis data, wherein the freight rate analysis data includes flight information data, passenger information data and freight rate information data.

[0108] The embodiment of the present invention constructs an original fare feature data set including flight information data, passenger information data and fare information data into a historical fare data training set so as to be used as input data for model training later.

[0109] Specifically, step S2 extracts features from the historical freight rate data training set to obtain multiple freight rate feature data including:

[0110] S21: Screening and cleaning the original freight rate data in the historical freight rate data training set to obtain original freight rate processing data.

[0111] The embodiment of the present invention is used to remove noise data and irrelevant features by screening and cleaning the original freight rate data in the historical freight rate data training set.

[0112] S22: extracting a plurality of fare characteristic data from the original fare processing data, including flight characteristic data, passenger characteristic data and fare characteristic data.

[0113] The embodiment of the present invention analyzes the fare characteristics and selects data variables that have a greater impact on the price as fare characteristic data, including flight characteristic data, passenger characteristic data and fare characteristic data.

[0114] Specifically, step S3 constructs the optimal freight rate feature data set based on multiple freight rate feature data, including:

[0115] S31: Characteristically encode a plurality of freight rate characteristic data to obtain a plurality of freight rate characteristic encoding data.

[0116] The embodiment of the present invention adopts a frequency coding method to perform feature coding on multiple freight rate feature data, converts the category feature into a numerical feature through frequency coding, and finally obtains the corresponding freight rate feature value. In data processing, frequency coding replaces the original category label with the frequency of each category in the data set, and then converts the category feature into a numerical feature. The embodiment of the present invention performs feature coding on the freight rate feature data so as to convert the freight rate feature information into a format that can be directly used by the machine learning model.

[0117] S32: Perform feature reorganization based on the multiple freight rate feature coding data to obtain multiple freight rate feature combination data.

[0118] S33: Construct an optimal freight rate feature data set based on multiple freight rate feature combination data.

[0119] Feature recombination is to recombine the extracted features in different ways and derive new features. The embodiment of the present invention recombines the encoded fare features and identifies special flight features by combination determination, thereby obtaining the optimal fare feature combination data. By constructing the optimal fare feature data set, the performance and accuracy of the model can be improved. For example, by identifying the special red-eye flight by combination determination, the fare feature combination data constituting the red-eye flight can constitute an optimal fare feature data set for the category of features.

[0120] Specifically, step S4 constructs an integrated anomaly detection model based on the optimal freight rate feature data set, including:

[0121] S41: construct multiple basic anomaly detection models based on the optimal freight rate feature data set, wherein the basic anomaly detection models include an anomaly detection model based on the K-nearest neighbor algorithm, an anomaly detection model based on the single-class support vector machine algorithm, and an anomaly detection model based on the Gaussian mixture model.

[0122] The embodiment of the present invention constructs a plurality of different basic anomaly detection models to perform preliminary detection on the optimal freight rate feature data. Since each basic anomaly detection model has its unique advantages in detecting abnormal data, these basic anomaly detection models can be used to analyze the data from multiple angles, thereby improving the robustness and accuracy of the detection. The embodiment of the present invention respectively constructs an anomaly detection model based on the K-nearest neighbor algorithm, an anomaly detection model based on the single-class support vector machine algorithm, and an anomaly detection model based on the Gaussian mixture model, which are used to perform multi-angle analysis on the optimal freight rate feature data and obtain multi-angle detection results.

[0123] S42: Evaluate the comprehensive precision and recall of the basic anomaly detection model and determine the corresponding unsupervised baseline model.

[0124] Comprehensive accuracy refers to the average accuracy of the model on multiple categories or multiple tasks. For example, in a multi-classification problem, the comprehensive accuracy can be the average of the accuracy of all categories, or the result obtained by weighted average to reflect the performance of the model in different categories. When evaluating the performance of a system or model, the comprehensive accuracy can provide a comprehensive performance indicator to reflect the performance of the system or model in multiple aspects. Recall rate refers to the proportion of samples that are successfully predicted as positive in all samples that are actually positive. The higher the recall rate, the more positive samples the model can capture and reduce missed detection. Search engines with high recall rates can return more relevant documents and increase the probability that users can find the information they need. The embodiment of the present invention uses a combined evaluation method of comprehensive accuracy and recall rate to comprehensively evaluate the basic anomaly detection model to establish an unsupervised baseline model and hyperparameters. The unsupervised baseline model is a model that does not automatically learn the features and patterns of data through an algorithm without manually annotated data. The model does not require a large amount of manually annotated data, so it can save a lot of manpower and time costs. The present invention improves the anomaly detection performance and reduces the false alarm rate and missed alarm rate while balancing the comprehensive accuracy and recall rate.

[0125] S43: Weight and integrate multiple unsupervised baseline models to obtain an integrated anomaly detection model.

[0126] The embodiment of the present invention obtains an integrated anomaly detection model by weighting and integrating multiple unsupervised baseline models to obtain the final detection result. By using the integrated learning scheme of multiple unsupervised baseline models, the timeliness and accuracy of anomaly detection are improved. The present invention achieves comprehensive detection of absolute anomalies and relative anomalies by integrating multiple unsupervised baseline models. At the same time, the system does not limit the category and number of baseline models. By introducing multiple models, intelligent processing of anomaly detection can be achieved.

[0127] Specifically, step S43 weights and integrates multiple unsupervised baseline models to obtain an integrated anomaly detection model including:

[0128] S431: The recall rate of the unsupervised baseline model is set as a weight according to actual business influencing factors.

[0129] S432: Integrate multiple unsupervised baseline models with set weights to obtain an integrated anomaly detection model.

[0130] S433: Determine an abnormality index of the integrated abnormality detection model based on the weight, and take the output whose abnormality index is higher than the abnormality threshold as an abnormal point.

[0131] The embodiment of the present invention first sets the recall rate of the unsupervised baseline model as a weight according to the actual business influencing factors, and then takes the average of the anomaly score outputs of multiple unsupervised baseline models according to the recall rate weight to obtain an integrated anomaly index output, and by setting a suitable anomaly threshold, the output with an anomaly index higher than the anomaly threshold is identified as an anomaly point. By integrating the detection results of these basic models, the outputs of multiple basic models are output according to different weight strategies to obtain anomaly index outputs, and then setting a suitable threshold to obtain the final detection result, the detection advantages of different basic models can be combined through the integrated strategy to improve the robustness and accuracy of anomaly detection.

[0132] In another specific embodiment of the present invention, the unsupervised baseline model may be replaced by other unsupervised models, such as an anomaly detection model based on a local outlier factor algorithm (LOF). The integration strategy may adopt a local selective combination (LSCP) approach.

[0133] Specifically, step S6 performs early warning and classification according to the detection results, and visualizes the detection results on the client.

[0134] The embodiment of the present invention implements data interaction with the user end by early warning and classifying the detection results, and visually displaying the detection results on the client. The user can realize early warning of abnormalities by setting static rules. For example, the warning can be set to be triggered when the price is lower than the preset threshold, and the detection results can be filtered and classified by defining rules to further improve the accuracy and reliability of the early warning. For example, it can be filtered and classified according to conditions such as the origin and destination. When an abnormality is detected, the system promptly notifies the relevant personnel by email to ensure that timely measures are taken. The embodiment of the present invention displays the real-time detection data and detection results through a visual interface, which is convenient for users to monitor the freight rate situation and the statistical distribution of abnormal freight rates in real time. Users can view historical data and real-time detection results through the monitoring dashboard.

[0135] Example 2

[0136] Based on the above method, the embodiment of the present invention also provides a civil aviation fare anomaly detection system, such as Figure 9-Figure 14 As shown, including:

[0137] The data acquisition module 01 is used to obtain the original freight rate data and create a historical freight rate data training set.

[0138] The feature extraction module 02 is used to extract features from the historical freight rate data training set to obtain a plurality of freight rate feature data.

[0139] The feature data set construction module 03 is used to construct an optimal freight rate feature data set based on multiple freight rate feature data.

[0140] The integrated model building module 04 is used to build an integrated anomaly detection model based on the optimal freight rate feature data set.

[0141] The anomaly detection module 05 is used to perform anomaly detection on the real-time freight rate data based on the integrated anomaly detection model and output the detection results.

[0142] The warning and display module 06 is used to issue warnings and classify according to the detection results, and to visualize the detection results on the client.

[0143] The data storage module 07 is used to record and store detection log information after completing anomaly detection.

[0144] The embodiment of the present invention relies on existing civil aviation fare data, based on machine learning and big data analysis technology, to develop a visualization system for abnormal detection of civil aviation fares, which can realize real-time dynamic monitoring of online air ticket prices. Fig.14 As shown in the figure, the system mainly includes data layer, feature engineering layer, anomaly detection layer and service layer, among which,

[0145] The data layer is responsible for obtaining real-time fare data from the civil aviation fare system, extracting historical training data sets, and storing and recording detection log information. Specifically, the system obtains the transaction log data stream from the civil aviation fare system, parses the transaction log data, and extracts the original feature set. For example, flight information, passenger information, and fare information are extracted from the log. The parsed original feature set is pushed to the anomaly detection server for further processing and analysis. At the same time, the system stores historical fare training data sets and detection log information. These historical data will serve as an important data resource for model training and evaluation.

[0146] The feature engineering layer is used to perform feature modeling on the original freight rate data, including operations such as feature extraction, encoding, combination and construction, as well as processing various features of the freight rate data and building the optimal feature set. The anomaly detection layer is used to build an integrated anomaly detection model based on the optimal freight rate feature data set, and perform anomaly detection on the real-time freight rate data based on the integrated anomaly detection model and output the detection results, thereby improving the robustness and accuracy of anomaly detection.

[0147] The service layer is used to implement multiple functions such as real-time detection, message notification, monitoring dashboard, static warning and rule filtering based on the abnormal detection results. The service layer receives the real-time detection results of the abnormal detection layer, displays and notifies users in a timely manner, and can set static rule alarms and filter the same type of data. The distribution of flight historical statistical data can be analyzed through the monitoring dashboard. Specifically, the following functions are implemented: Real-time detection: The system can detect abnormalities as soon as the data is updated and display the detection results to users. Message notification: When an abnormality is detected, the system notifies relevant personnel in a timely manner by email to ensure that timely measures are taken. Monitoring dashboard: The system provides a visual interface for users to monitor the freight rate situation and the statistical distribution of abnormal freight rates in real time. Users can view historical data and real-time detection results through the monitoring dashboard. Static warning: The system allows users to set static rules to realize warning of abnormalities. For example, you can set a warning to be triggered when the price is lower than a certain threshold. Rule filtering: Filter and classify the detection results through defined rules to further improve the accuracy and reliability of the warning. For example, you can filter and classify according to conditions such as origin and destination.

[0148] like Figure 7 As shown in the system data flow, the data layer captures the transaction log data flow and freight rate rule data from the freight rate system to obtain the original log data, and obtains the training set and test set after feature modeling, which correspond to the input data for model training and prediction respectively. The data are input into the above-mentioned various basic anomaly detection models to obtain the preliminary anomaly detection output, and then the preliminary detection results are integrated through the integration model to obtain the final detection results. Finally, the results are applied to the visualization display function of the service layer.

[0149] Specifically, the data acquisition module 01 first obtains and analyzes the transaction log data from the original freight rate data to obtain the freight rate analysis data, and then constructs a historical freight rate data training set based on the freight rate analysis data, wherein the freight rate analysis data includes flight information data, passenger information data and freight rate information data.

[0150] In the embodiment of the present invention, the data acquisition module 01 obtains the transaction log data stream and the fare rule data from the fare system respectively to form the original log data. By parsing the transaction log data, the fare parsing data including the fare information, flight information and passenger information can be obtained. For example, the flight information, passenger information and fare information are extracted from the transaction log, and the original fare feature data obtained by the above parsing is pushed to the anomaly detection server for further processing and analysis.

[0151] The embodiment of the present invention constructs an original fare feature data set including flight information data, passenger information data and fare information data into a historical fare data training set so as to be used as input data for model training later.

[0152] Specifically, the feature extraction module 02 also includes:

[0153] The data processing module 021 is used to screen and clean the original freight rate data in the historical freight rate data training set to obtain the original freight rate processed data.

[0154] The embodiment of the present invention uses the data processing module 021 to screen and clean the original freight rate data in the historical freight rate data training set to remove noise data and irrelevant features.

[0155] The feature screening module 022 is used to extract a plurality of fare feature data from the original fare processing data, including flight feature data, passenger feature data and fare feature data.

[0156] The embodiment of the present invention analyzes the fare characteristics through the feature screening module 022 and selects data variables that have a greater impact on the price as fare characteristic data, including flight characteristic data, passenger characteristic data and fare characteristic data.

[0157] Specifically, the feature data set construction module 03 also includes:

[0158] The feature coding module 031 is used to perform feature coding on a plurality of freight rate feature data to obtain a plurality of freight rate feature coding data.

[0159] The embodiment of the present invention uses the feature encoding module 031 to perform feature encoding on multiple freight rate feature data, converts the category feature into a numerical feature, and finally obtains the corresponding freight rate feature value. The embodiment of the present invention performs feature encoding on the freight rate feature data so as to convert the freight rate feature information into a format that can be directly used by the machine learning model.

[0160] The feature reorganization module 032 is used to perform feature reorganization based on multiple freight rate feature coding data to obtain multiple freight rate feature combination data.

[0161] The embodiment of the present invention recombines the encoded fare features through the feature recombining module 032, and identifies the special flight features through combination determination, so as to obtain the optimal fare feature combination data.

[0162] The optimal feature data set construction module 033 is used to construct an optimal freight rate feature data set based on multiple freight rate feature combination data.

[0163] The embodiment of the present invention can improve the performance and accuracy of the model by constructing the optimal fare feature data set through the optimal feature data set construction module 033. For example, by identifying a special red-eye flight through a combination determination method, the fare feature combination data constituting the red-eye flight can constitute an optimal fare feature data set for the category feature.

[0164] Specifically, the integrated model building module 04 includes:

[0165] The basic model construction module 041 is used to construct multiple basic anomaly detection models based on the optimal freight rate feature data set. The basic anomaly detection models include an anomaly detection model based on the K-nearest neighbor algorithm, an anomaly detection model based on the single-class support vector machine algorithm, and an anomaly detection model based on the Gaussian mixture model.

[0166] The embodiment of the present invention constructs a plurality of different basic anomaly detection models through the basic model construction module 041 to perform preliminary detection on the optimal freight rate feature data. Since each basic anomaly detection model has its own unique advantages in detecting abnormal data, these basic anomaly detection models can be used to analyze the data from multiple angles, thereby improving the robustness and accuracy of the detection. The embodiment of the present invention respectively constructs an anomaly detection model based on the K-nearest neighbor algorithm, an anomaly detection model based on the single-class support vector machine algorithm, and an anomaly detection model based on the Gaussian mixture model, which are used to perform multi-angle analysis on the optimal freight rate feature data and obtain multi-angle detection results.

[0167] The baseline model building module 042 is used to evaluate the comprehensive accuracy and recall rate of the basic anomaly detection model and determine the corresponding unsupervised baseline model.

[0168] The embodiment of the present invention uses the baseline model construction module 042 to comprehensively evaluate the basic anomaly detection model to establish an unsupervised baseline model and hyperparameters. The unsupervised baseline model is a model that automatically learns the characteristics and patterns of data through an algorithm without manually annotated data. This model does not require a large amount of manually annotated data, so it can save a lot of manpower and time costs.

[0169] The integration module 043 is used to weight and integrate multiple unsupervised baseline models to obtain an integrated anomaly detection model.

[0170] The embodiment of the present invention weights and integrates multiple unsupervised baseline models through the integration module 043 to obtain an integrated anomaly detection model to obtain the final detection result.

[0171] Specifically, the integrated module 043 also includes:

[0172] The weight setting module 0431 is used to set the recall rate of the unsupervised baseline model as a weight according to actual business influencing factors.

[0173] The model integration module 0432 is used to integrate multiple unsupervised baseline models with set weights to obtain an integrated anomaly detection model.

[0174] The indicator setting module 0433 is used to determine the abnormal indicator of the integrated abnormality detection model based on the weight, and take the output of the abnormal indicator higher than the abnormal threshold as the abnormal point.

[0175] The embodiment of the present invention first sets the recall rate of the unsupervised baseline model as a weight through the weight setting module 0431, and then uses the index setting module 0433 to average the abnormal score outputs of multiple unsupervised baseline models according to the recall rate weight to obtain an integrated abnormal index output, and by setting a suitable abnormal threshold, the output with an abnormal index higher than the abnormal threshold is identified as an abnormal point. The detection results of these basic models are integrated through the model integration module 0432, and the outputs of multiple basic models are output according to different weight strategies to obtain abnormal index outputs, and then the appropriate threshold is set to obtain the final detection result. The detection advantages of different basic models can be combined through the integration strategy to improve the robustness and accuracy of abnormal detection.

[0176] Example 3

[0177] like Fig.15 As shown, an embodiment of the present invention further provides a computer device, the computer device includes a processor and a memory, the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the above-mentioned civil aviation fare anomaly detection method. The device in the present invention can be a server, a PC, a PAD, a mobile phone, etc.

[0178] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned civil aviation fare anomaly detection method.

[0179] Furthermore, an embodiment of the present invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing the initialization of the civil aviation fare anomaly detection method as described above.

[0180] In summary, the method, system, device and storage medium for detecting abnormal civil aviation fares of the present invention have the following advantages:

[0181] The present invention first collects and captures the freight rate data stream to obtain a training data set, and then screens and efficiently models the important features that affect civil aviation freight rates to ensure that the model accurately reflects the law of price fluctuations. In order to cope with the problem of lack of labeled data, the present invention designs an unsupervised anomaly detection method, determines the model and parameters used, realizes preliminary freight rate anomaly detection, and then realizes the integration of multiple unsupervised algorithms by setting anomaly indicators and thresholds with different weight strategies, thereby improving the timeliness and accuracy of detection. The integrated model proposed in the present invention improves the anomaly detection performance while balancing the accuracy and recall rate, and reduces the false alarm rate and missed alarm rate. By integrating multiple baseline models, comprehensive detection of absolute anomalies and relative anomalies is achieved. At the same time, the system does not limit the selection and number of baseline models, and intelligent processing of anomaly detection can be achieved by introducing multiple models. The present invention solves the problem of low efficiency and poor real-time performance of traditional civil aviation freight rate anomaly detection technology, can realize real-time anomaly detection, and relieves the pressure of manual verification.

[0182] The above describes in detail the concepts, principles and ideas of the present invention in conjunction with specific implementation methods (including embodiments and examples). Those skilled in the art should understand that the implementation methods of the present invention are not limited to the forms given above. After reading this application document, those skilled in the art can make any possible improvements, substitutions and equivalent forms to the steps, methods, systems and components in the above implementation methods. These improvements, substitutions and equivalent forms should be deemed to fall within the scope of the present invention, and the protection scope of the present invention is subject only to the claims.

Claims

1. A method for detecting abnormal civil aviation freight rates, characterized in that: include: Obtain original freight rate data and create a training set of historical freight rate data; Extracting features from the historical freight rate data training set to obtain a plurality of freight rate feature data; Constructing an optimal freight rate feature data set based on the plurality of freight rate feature data; Building an integrated anomaly detection model based on the optimal freight rate feature data set; Based on the integrated anomaly detection model, anomaly detection is performed on the real-time freight rate data and the detection result is output.

2. The method for detecting abnormal civil aviation fares according to claim 1, characterized in that: The original fare data includes fare rule data and transaction log data, wherein the transaction log data includes field information of passengers, fares and flights.

3. The method for detecting abnormal civil aviation fares according to claim 2, characterized in that: The obtaining of original freight rate data and creating a historical freight rate data training set includes: Acquire transaction log data from the original freight rate data and parse it to obtain freight rate parsing data; A historical freight rate data training set is constructed based on the freight rate analysis data, wherein the freight rate analysis data includes flight information data, passenger information data and freight rate information data.

4. The method for detecting abnormal civil aviation fares according to claim 3, characterized in that: The feature extraction of the historical freight rate data training set to obtain a plurality of freight rate feature data includes: Screening and cleaning the original freight rate data in the historical freight rate data training set to obtain original freight rate processed data; A plurality of fare characteristic data are extracted from the original fare processing data, including flight characteristic data, passenger characteristic data and fare characteristic data.

5. The method for detecting abnormal civil aviation fares according to claim 4, characterized in that: The constructing of the optimal freight rate feature data set based on the plurality of freight rate feature data comprises: Performing feature coding on the plurality of freight rate feature data to obtain a plurality of freight rate feature coding data; Based on the multiple freight rate feature coding data, feature reorganization is performed to obtain multiple freight rate feature combination data; An optimal freight rate feature data set is constructed based on a plurality of the freight rate feature combination data.

6. The method for detecting abnormal civil aviation fares according to claim 5, characterized in that: The feature encoding of the plurality of freight rate feature data to obtain a plurality of freight rate feature encoding data comprises: A frequency coding method is used to perform feature coding on the plurality of freight rate feature data to obtain corresponding freight rate feature values.

7. The method for detecting abnormal civil aviation fares according to claim 1, characterized in that: The constructing of an integrated anomaly detection model based on the optimal freight rate feature data set includes: Building multiple basic anomaly detection models based on the optimal freight rate feature data set; Evaluate the comprehensive precision and recall of the basic anomaly detection model and determine the corresponding unsupervised baseline model; Weighting and integrating the multiple unsupervised baseline models to obtain an integrated anomaly detection model; Among them, the basic anomaly detection model includes an anomaly detection model based on the K-nearest neighbor algorithm, an anomaly detection model based on the single-class support vector machine algorithm and an anomaly detection model based on the Gaussian mixture model.

8. The method for detecting abnormal civil aviation fares according to claim 7, characterized in that: The step of weighting and integrating the plurality of unsupervised baseline models to obtain an integrated anomaly detection model comprises: The recall rate of the unsupervised baseline model is set as a weight according to actual business influencing factors; Integrate the plurality of unsupervised baseline models with set weights to obtain an integrated anomaly detection model; An abnormality index of the integrated abnormality detection model is determined based on the weight, and an output where the abnormality index is higher than an abnormality threshold is taken as an abnormal point.

9. The method for detecting abnormal civil aviation fares according to claim 8, characterized in that: The determining of the anomaly index of the integrated anomaly detection model based on the weight includes: The anomaly score output of the unsupervised baseline model is averaged according to the recall weight to obtain an integrated anomaly index.

10. The method for detecting abnormal civil aviation fares according to claim 1, characterized in that: The method further comprises: Early warning and classification are performed based on the detection results, and the detection results are visualized on the client.

11. The method for detecting abnormal civil aviation fares according to claim 1, characterized in that: The method further comprises: After the anomaly detection is completed, the detection log information is recorded and stored.

12. A civil aviation fare anomaly detection system, characterized in that: include: Data acquisition module, used to obtain original freight rate data and create a historical freight rate data training set; A feature extraction module is used to extract features from the historical freight rate data training set to obtain a plurality of freight rate feature data; A feature data set construction module, used to construct an optimal freight rate feature data set based on a plurality of the freight rate feature data; An integrated model building module, used to build an integrated anomaly detection model based on the optimal freight rate feature data set; The anomaly detection module is used to perform anomaly detection on the real-time freight rate data based on the integrated anomaly detection model and output the detection result.

13. The civil aviation fare anomaly detection system according to claim 12, characterized in that: The feature extraction module also includes: A data processing module, used for screening and cleaning the original freight rate data in the historical freight rate data training set to obtain original freight rate processed data; The feature screening module is used to extract a plurality of fare feature data from the original fare processing data, including flight feature data, passenger feature data and fare feature data.

14. The civil aviation fare anomaly detection system according to claim 12, characterized in that: The feature data set construction module also includes: A feature coding module, used for performing feature coding on the plurality of freight feature data to obtain a plurality of freight feature coding data; A feature reorganization module, used for performing feature reorganization based on the plurality of freight rate feature coding data to obtain a plurality of freight rate feature combination data; The optimal feature data set construction module is used to construct the optimal freight rate feature data set based on a plurality of the freight rate feature combination data.

15. The civil aviation fare anomaly detection system according to claim 12, characterized in that: The integrated model building module includes: A basic model building module, used to build multiple basic anomaly detection models based on the optimal freight rate feature data set; A baseline model building module, used to evaluate the comprehensive accuracy and recall of the basic anomaly detection model and determine the corresponding unsupervised baseline model; An integration module, used for weighting and integrating the multiple unsupervised baseline models to obtain an integrated anomaly detection model; Among them, the basic anomaly detection model includes an anomaly detection model based on the K-nearest neighbor algorithm, an anomaly detection model based on the single-class support vector machine algorithm and an anomaly detection model based on the Gaussian mixture model.

16. The civil aviation fare anomaly detection system according to claim 15, characterized in that: The integrated module also includes: A weight setting module, used to set the recall rate of the unsupervised baseline model as a weight according to actual business influencing factors; A model integration module, used to integrate the plurality of unsupervised baseline models with set weights to obtain an integrated anomaly detection model; The indicator setting module is used to determine the abnormal indicator of the integrated abnormality detection model based on the weight, and take the output of the abnormal indicator higher than the abnormal threshold as the abnormal point.

17. The civil aviation fare anomaly detection system according to claim 12, characterized in that: The system further comprises: The warning and display module is used to issue warnings and classify according to the detection results, and to visually display the detection results on the client.

18. The civil aviation fare anomaly detection system according to claim 12, characterized in that: The system further comprises: The data storage module is used to record and store detection log information after completing anomaly detection.

19. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the civil aviation fare anomaly detection method as described in any one of claims 1 to 11 above.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the civil aviation fare anomaly detection method according to any one of claims 1 to 11.

Images