Abnormal data processing method and device, storage medium and electronic equipment

Abstract

The invention belongs to the technical field of freight rate release, and provides an abnormal data processing method and device, a storage medium and electronic equipment, and the method comprises the steps: carrying out the interception processing of real-time freight rate data of a freight rate release system through a data interceptor; matching a target detection rule strategy from a detection rule base; and detecting the real-time freight rate data based on the target detection rule strategy, determining abnormal data in the real-time freight rate data, and sending early warning information to the abnormal data. According to the processing method, abnormal data can be automatically detected; according to the processing method, only the real-time freight rate data is detected, so that the data volume of the to-be-detected data is greatly reduced, and the data processing speed is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of freight rate publishing, and in particular to a method, device, storage medium and electronic device for processing abnormal data. Background Art

[0002] The formulation of air ticket pricing policies is the core business of airlines. In order to maximize ticket sales and revenue, airlines not only consider operating costs in the formulation and release of fares, but also combine the consumption psychology and ticket purchasing behavior characteristics of passengers to formulate a rich variety of fare products and release a large amount of fare and related rules data to support ticket sales. As a user of fare data, the fare system is responsible for comprehensive analysis and processing of the received airline pricing data, passenger itinerary data and business rules, and selects the most appropriate results for fare service users. The fare data ecosystem chain is as follows: Figure 1 As shown. Figure 1 The fare data ecosystem shown includes data makers, data users, and service users. Data makers include: Airline A, Airline B, and Airline C; data users push data to the fare search system and fare calculation system by running the publishing system; service users include: airlines (direct sales), ticket platforms / agents (distribution), and travel platforms / wholesalers (distribution).

[0003] The entire data chain involves many roles from release to application, and there is human intervention in many links, which can easily lead to errors, resulting in the inability to sell air tickets. In more serious cases, abnormally low-priced tickets will be produced, causing significant economic losses. Therefore, it is essential to check various types of data during the process of air ticket price formulation and release. How to ensure the correctness of fare data to avoid sales losses is an urgent problem that the industry needs to solve. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, storage medium and electronic device for processing abnormal data to address the defect that the existing freight rate data detection method cannot achieve automatic detection.

[0005] In a first aspect, an embodiment of the present invention provides a method for processing abnormal data, the method comprising:

[0006] Through the data interceptor, the real-time freight rate data of the freight rate publishing system is intercepted and processed;

[0007] Match target detection rule strategy from the detection rule library;

[0008] Based on the target detection rule strategy, the real-time freight rate data is detected to determine abnormal data in the real-time freight rate data, so as to issue early warning information for the abnormal data.

[0009] Optionally, before matching the target detection rule strategy from the detection rule library, the method further includes:

[0010] Determine the target detection rule strategy.

[0011] Optionally, determining the target detection rule strategy includes:

[0012] Acquire multiple key parameters, the multiple key parameters include: a first ID value, multiple matching fields, and a threshold or anomaly value, the multiple matching fields include: airline, origin, destination, and class;

[0013] The target detection rule strategy is determined based on multiple key parameters.

[0014] Optionally, the determining the target detection rule strategy further includes:

[0015] Obtain a record whose freight rate basic field is empty or an empty string, and use the record as a target record; and obtain a second ID value corresponding to the target record;

[0016] The target detection rule strategy is determined based on the target record and the second ID value.

[0017] Optionally, after determining the target detection rule strategy, the method further includes:

[0018] Obtaining the target detection rule strategy;

[0019] Determine a plurality of remaining detection rule strategies other than the target detection rule strategy based on the plurality of key parameters;

[0020] Based on the target detection rule strategy and the multiple detection rule strategies, the detection rule base is constructed, and the rules corresponding to the detection rule base at least include a minimum value rule and a maximum value rule.

[0021] Optionally, constructing the detection rule library based on the target detection rule strategy and the multiple detection rule strategies includes:

[0022] Based on the target detection rule strategy, the multiple detection rule strategies and historical data, read the value domain of each matching dimension attribute;

[0023] Based on the value domains of the attributes of each matching dimension, obtaining the minimum value of the number of target fields under all matching dimension value combinations, and obtaining the maximum value of the number of target fields under all matching dimension value combinations;

[0024] Get the abnormal low value threshold, and get the abnormal high value threshold;

[0025] Determine the minimum value rule based on a set of the minimum value of the number of target fields under all matching dimension value combinations and the product of the abnormal low value threshold;

[0026] The maximum value rule is determined based on a set of products of the maximum value of the target field quantity under all matching dimension value combinations and the abnormally high value threshold.

[0027] In a second aspect, an embodiment of the present invention provides a device for processing abnormal data, the device comprising:

[0028] The interception module is used to intercept and process the real-time freight rate data of the freight rate publishing system through the data interceptor;

[0029] A matching module is used to match target detection rule strategies from the detection rule library;

[0030] The anomaly detection module is used to detect the real-time freight rate data based on the target detection rule strategy, determine the abnormal data in the real-time freight rate data, and issue early warning information for the abnormal data.

[0031] Optionally, the device further comprises:

[0032] The determination module is used to determine the target detection rule strategy before matching the target detection rule strategy from the detection rule library.

[0033] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0034] According to a fourth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to the first aspect is implemented.

[0035] In the embodiment of the present invention, the real-time freight rate data of the freight rate publishing system is intercepted and processed by a data interceptor; the target detection rule strategy is matched from the detection rule library; based on the target detection rule strategy, the real-time freight rate data is detected to determine the abnormal data in the real-time freight rate data. The abnormal data processing method provided by the embodiment of the present invention can automatically detect abnormal data; the processing method, because only the real-time freight rate data is detected, greatly reduces the amount of data to be detected and improves the speed of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] By referring to the following drawings, a more complete understanding of the exemplary embodiments of the present invention can be obtained. The drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the drawings, the same reference numerals generally represent the same components or steps.

[0037] Figure 1 This is a schematic diagram of the freight rate data ecosystem;

[0038] Figure 2 It is a schematic diagram of the improved freight rate publishing system;

[0039] Figure 3 A flowchart of a method for processing abnormal data according to an exemplary embodiment of the present invention;

[0040] Figure 4 A flowchart for detecting abnormal data;

[0041] Figure 5 It is a schematic diagram of the structure of an abnormal data processing device 500 provided according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0042] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present invention should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0044] In addition, the terms "first" and "second" etc. are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0045] The embodiments of the present invention provide a method and device for processing abnormal data, a computer-readable medium, and an electronic device, which are described below with reference to the accompanying drawings.

[0046] The standard fare publishing system mainly has three core modules, namely: collection module (used to collect fare data), storage module and push module. Collection module: collect the fare and related rule data sources of the airline side through the front-end system or docking with the external system. Storage module: after passing the business verification, the collected fare and related rule data sources of the airline side are stored in the database. Push module (used to push fare data): the fare data in the database is published to the fare search and calculation system in batches according to the preset rules to participate in the ticket sales. Although most of the above processes have been basically automated, there are still some links that need to be completed by manual assistance, such as the formulation, adjustment and release of prices on the airline side, and the reissue of special data in the data transmission link. Even automatic services can only reduce but not eliminate the possibility of errors. Once the data is wrong and pushed to the fare search and calculation system, the error will be further amplified with the current developed ticket sales network.

[0047] In order to avoid this situation, the method for processing abnormal data provided by the embodiment of the present invention improves the existing freight rate publishing system to obtain an improved freight rate publishing system. A detection module (for detecting abnormal data in real-time freight rate data) is added in the data collection and processing stage, so that verification detection and notification warning of abnormal data are realized before the source data is stored in the database. The process of detecting abnormal data in real-time freight rate data through the adopted detection mechanism is specifically described as follows: historical data is obtained, and data statistical analysis is performed on the obtained historical data to obtain analysis results; based on the analysis results, a corresponding detection rule library is set, and the detection rule library contains multiple thresholds that match different rules, so that abnormal data (especially extremely low price data) in real-time freight rate data can be verified through the set threshold.

[0048] For example Figure 2 The improved freight rate publishing system shown is described as follows:

[0049] Note 1: The real-time freight rate data is intercepted and processed through the data interceptor, and stored in Redis through the storage module. The detection module obtains all the data to be stored from Redis and verifies the data to be stored. The detection process of abnormal data corresponding to the detection module is a bypass branch of the main data collection process corresponding to the data collection module, which is decoupled from the main process. Data verification does not affect the function and performance of the main process.

[0050] It should be noted that Redis (Remote Dictionary Server) is a key-value pair storage database that can implement message push function based on the publish / subscribe model.

[0051] Note 2: Each abnormal data detection is for real-time freight rate data. In this way, the amount of data to be detected is small and the processing speed is fast. Abnormal data is detected synchronously during the data collection process, and after abnormal data is detected in the real-time freight rate data, an early warning notification is sent in real time.

[0052] Note 3: The detection rule strategy used by the detection module to detect whether there is abnormal data in the real-time freight rate data is stored in the detection rule library. The detection rule strategy used by the detection module is obtained by statistically analyzing the historical freight rate data. The detection rule strategy used by the detection module is for a single source data type and does not involve complex freight rate calculation logic. Therefore, the matching and verification dimensions can be set by general freight rate business personnel, thereby simplifying the configuration process of the detection rule strategy. System operation and maintenance personnel can also configure the detection module to start the construction service at a scheduled time to create corresponding detection rule strategies for newly entered data.

[0053] Please refer to Figure 3 , which shows a flowchart of a method for processing abnormal data provided by some embodiments of the present invention, such as Figure 3 As shown, the method for processing abnormal data may include the following steps:

[0054] Step S301: intercepting and processing the real-time freight rate data of the freight rate publishing system through a data interceptor.

[0055] It should be noted that the sources of real-time freight rate data intercepted by the data interceptor include: real-time freight rate data sent to the freight rate publishing system at preset intervals, and real-time freight rate data collected and transmitted from the front-end collection device.

[0056] It should be noted that the preset duration is not specifically limited here, and the preset duration can be adjusted according to the requirements of different application scenarios, which will not be elaborated here.

[0057] Step S302: Match target detection rule strategy from the detection rule library.

[0058] In actual application scenarios, the target detection rule strategy matched from the detection rule library can be matched based on parameters such as matching fields (the matching fields are composed of airlines, origins, destinations, and cabins), thresholds, and comparison types.

[0059] For example, in one example, based on the matching field 1E / BJS / SHA / Y, the threshold or abnormal value 600, and the comparison type is lower than, based on the above parameters, the target detection rule strategy matched from the detection rule library is: matching based on the 1E / BJS / SHA / Y matching field, if the current freight rate data is lower than the threshold 600, the current marriage leave data is determined to be abnormal data; otherwise, the current freight rate data is determined to be normal data.

[0060] For another example, based on the matching field 1E / BJS / SHA / C, the threshold or abnormal value 1200, and the comparison type is lower than, based on the above parameters, the target detection rule strategy matched from the detection rule library is: matching based on the 1E / BJS / SHA / C matching field, if the current freight rate data is lower than the threshold 1200, the current marriage leave data is determined to be abnormal data; otherwise, the current freight rate data is determined to be normal data.

[0061] For another example, in one example, based on the matching field 1E / BJS / SHA / F, the threshold or abnormal value 3000, and the comparison type is lower than, based on the above parameters, the target detection rule strategy matched from the detection rule library is: matching based on the 1E / BJS / SHA / F matching field, if the current freight rate data is lower than the threshold 3000, the current marriage leave data is determined to be abnormal data; otherwise, the current freight rate data is determined to be normal data.

[0062] For another example, in one example, if empty data appears, it is determined to be abnormal data.

[0063] The above is only an example. In actual application scenarios, there is no specific limitation on the target detection rule strategy. The target detection rule strategy can be adjusted according to the requirements of different application scenarios, which will not be described in detail here.

[0064] In one example, before matching the target detection rule strategy from the detection rule library, the method for processing abnormal data provided by the embodiment of the present invention may further include the following steps:

[0065] Determine the target detection rule strategy.

[0066] In one example, determining a target detection rule strategy includes the following steps:

[0067] Acquire multiple key parameters, the multiple key parameters include: a first ID value, multiple matching fields, and a threshold or anomaly value, the multiple matching fields include: airline, origin, destination, and class;

[0068] Determine the target detection rule strategy based on multiple key parameters.

[0069] In one example, determining the target detection rule strategy further includes the following steps:

[0070] Obtain a record whose freight rate basic field is empty or an empty string, and use the record as a target record; and obtain a second ID value corresponding to the target record;

[0071] The target detection rule strategy is determined based on the target record and the second ID value.

[0072] In one example, after determining the target detection rule strategy, the method for processing abnormal data provided by the embodiment of the present invention further includes the following steps:

[0073] Get target detection rule strategy;

[0074] Determine multiple detection rule strategies other than the target detection rule strategy based on multiple key parameters;

[0075] Based on the target detection rule strategy and multiple detection rule strategies, a detection rule base is constructed, and the rules corresponding to the detection rule base include at least a minimum value rule and a maximum value rule.

[0076] In one example, building a detection rule library based on a target detection rule strategy and multiple detection rule strategies includes the following steps:

[0077] Based on the target detection rule strategy, multiple detection rule strategies and historical data, read the value domain of each matching dimension attribute;

[0078] Based on the value domain of each matching dimension attribute, obtain the minimum value of the target field quantity under all matching dimension value combinations, and obtain the maximum value of the target field quantity under all matching dimension value combinations;

[0079] Get the abnormal low value threshold, and get the abnormal high value threshold;

[0080] Determine the minimum value rule based on the set of the minimum value of the target field quantity under all matching dimension value combinations and the product of the abnormal low value threshold;

[0081] The maximum value rule is determined based on the set of the maximum value of the target field quantity under all matching dimension value combinations multiplied by the abnormal high value threshold.

[0082] In actual application scenarios, target detection rule strategies and multiple detection rule strategies define the target fields to be detected for the freight rate data type that requires warning, as well as the matching dimension fields for data detection.

[0083] The process of building a detection rule base is as follows:

[0084] Assume that there is a data type DataType1, which contains a numeric field Amount and several other fields A 1, A 2 , ..., A m In the process of building the detection rule library, first, define the detection rule strategy of DataType1: use Amount as the target field to be detected by warning, and use A 1 , A 2 ,...A m As the matching dimension field for detection, two threshold parameters are defined: the abnormal low value threshold is 0.6 and the abnormal high value threshold is 1.5.

[0085] The anomaly detection module reads historical data, performs statistics on the historical data, and then reads the value range of each matching dimension attribute according to the detection rule strategy (including target detection rule strategy and multiple detection rule strategies) as follows:

[0086] V(A 1 )={a1,a2,...,a t1},V(A 2 )={b1,b2,...,b t2},...,V(A m )={m1,m2,...,m tm};

[0087] On this basis, the minimum and maximum values ​​of the target field Amount under all matching dimension value combinations are obtained. Finally, according to the thresholds (thresholds include abnormally low value thresholds and abnormally high value thresholds), the minimum value rule and maximum value rule as shown below are constructed respectively:

[0088] The minimum value rules are as follows:

[0089]

[0090] The maximum value rules are as follows:

[0091]

[0092] The minimum rule is the product of the minimum value of the target field Amount under all matching dimension value combinations and the abnormal low value threshold, and the maximum rule is the product of the maximum value of the target field Amount under all matching dimension value combinations and the abnormal high value threshold. 1 )×V(A 2 )×...×V(A m ) indicates matching dimension A 1 , A 2 , ..., A m The Cartesian product of the range of .

[0093] It should be noted that the data types that usually require early warning in the freight rate business field include but are not limited to freight rates, regular freight rates, etc. The target field for early warning detection in freight rates is usually but not limited to the freight rate amount, and the freight rate amount field value is detected to see if it is abnormally low; the matching dimension fields are usually but not limited to airlines, origin, destination, etc.; the target field for early warning detection in regular freight rates is usually but not limited to the calculation ratio, and the calculation ratio field value is detected to see if it is abnormally low; the matching dimension fields usually include but are not limited to airlines, discount codes, etc.

[0094] Step S303: Based on the target detection rule strategy, the real-time freight rate data is detected to determine abnormal data in the real-time freight rate data, so as to issue warning information for the abnormal data.

[0095] In actual application scenarios, warning information is issued for abnormal data to notify airlines to process the abnormal data.

[0096] like Figure 4 As shown, it is a flowchart for detecting abnormal data. The process of detecting abnormal data is specifically described as follows: start the detection and build a detection rule base; intercept and process the real-time freight data through the data interceptor, and store the real-time freight data in the database (for example, Redis); read the real-time freight data from the database (for example, Redis); detect the abnormal data; determine whether there is abnormal data in the real-time freight data; if there is abnormal data in the real-time freight data, issue an early warning message to notify the target user to conduct further detection, for example, through manual detection, determine whether there is abnormal data in the real-time freight data; if it is determined through manual detection that there is abnormal data in the real-time freight data, notify the airline to process it and end the detection process; on the contrary, if it is determined through manual detection that there is no abnormal data in the real-time freight data, end the detection process.

[0097] The above process of constructing the detection rule library is specifically described as follows: The detection rule library defines the standards for data anomalies. After determining the source data type to be verified, for example, freight rates and rule freight rates, the freight rate business personnel set the fields used for matching dimensions in the data type, such as airlines, origins, destinations, etc., as well as the fields for verification dimensions. If the verification dimension field is a numeric type, further set the threshold for determining abnormal data; if the verification dimension field is a character type, configure the abnormal value of the set field. After defining the standards for abnormal data based on the above, build the corresponding detection rule strategy based on historical data, and build a detection rule library based on each detection rule strategy constructed.

[0098] It should be noted that the process of constructing a corresponding detection rule strategy based on historical data and constructing a detection rule library based on each constructed detection rule strategy can be found in the description of the same or similar parts above, which will not be repeated here.

[0099] The above process of reading real-time freight rate data from a database (e.g., Redis) is specifically described as follows: The real-time freight rate data collected by the collection module is intercepted and processed through a data interceptor, and each incremental modification or newly added data is written to the database (e.g., Redis) in accordance with an agreed format. A detection module for detecting whether there is abnormal data in the real-time freight rate data polls the database (e.g., Redis) in real time, sequentially obtains each batch of data therein, processes each batch of data, and removes it from the database (e.g., Redis) after processing. The interception, writing, and acquisition of real-time freight rate data, as well as the main process of freight rate data collection and processing and abnormal data detection are all decoupled.

[0100] The above-mentioned abnormal data detection process is specifically described as follows: for the real-time freight rate data to be detected, the applicable rules are searched in the detection rule library according to the matching dimension field value combination, and then the target field value to be detected is compared with the threshold or abnormal value in the rule to determine whether the data record is abnormal data.

[0101] Assume that there is a data of type DataType in the data to be detected, Data1, and its field A 1 , A 2 , ..., A m The values ​​are a x , b y ,...,m z , the value of the Amount field is amount1, if a x ∈{a1,a2,...,a t1}, b y ∈{b1,b2,...,b t2},m z ∈{m1,m2,...,m tm},and

[0102] amount1 <min(Amount(a x ,b y , ..., m z ))*0.6, at this time, the amount of Data1 is lower than the abnormally low value threshold of the warning.

[0103] The above abnormal data detection process will be applied to all data entering the fare publishing system of the airline. The data released by airline users in different operations will correspond to different batches of data detection, and data detection will be carried out strictly in the chronological order of user operations to ensure the integrity and consistency of data detection.

[0104] Through manual detection, it is determined whether there is abnormal data in the real-time freight rate data; if there is abnormal data in the real-time freight rate data, an early warning message (for example, an early warning email) is sent to a designated user (for example, a duty officer) so that the duty officer can review it in time to determine whether there is abnormal data in the real-time freight rate data through review, and if it is determined that there is abnormal data in the real-time freight rate data, the airline user is notified to correct the abnormal data in time, or the data is terminated.

[0105] The following is a specific example of detecting real-time freight rate data based on the target detection rule strategy to determine abnormal data in the real-time freight rate data, as described below:

[0106] Step a1: Based on the collected real-time freight rate data, the following two target detection rule strategies are set. As shown in Table 1, Table 1 is the target detection rule strategy.

[0107]

[0108] Table 1

[0109] Step a2: Assuming that the freight rate table contains the freight rate historical data in Table 2, when detecting abnormal data, it is set to rebuild the detection rule base at a specific time every day. According to the target detection rule strategy set in Table 1, the detection rule base is rebuilt based on the freight rate historical data, and a detection rule base record as shown in Table 3 is generated.

[0110]

[0111]

[0112] Table 2

[0113] ID Match Fields Threshold or outlier 31 1E / BJS / SHA / Y 600 32 1E / BJS / SHA / C 1200 33 1E / BJS / SHA / F 3000 34 Blank

[0114] Table 3

[0115] Among them, when constructing a rule record according to the detection rule strategy (ID11), because the comparison type is "lower than", when there are records with the same matching fields in the historical records, the lowest value of the check field, that is, the freight amount field, is applied, and this is multiplied by the threshold ratio (similarly, the abnormal data processing method provided by the embodiment of the present invention can also check abnormal high-price data). The meanings of the 4 records recorded in the detection rule library are:

[0116] ID31: When the airline, origin, destination, and cabin combination is 1E / BJS / SHA / Y, the freight rate amount is less than 600, which is abnormal data;

[0117] ID32: When the airline, origin, destination, and cabin combination is 1E / BJS / SHA / C, the freight rate amount is less than 1200, which is abnormal data;

[0118] ID33: When the airline, origin, destination, and cabin combination is 1E / BJS / SHA / F, the freight rate is less than 3,000, which is abnormal data;

[0119] ID34, records whose freight rate base field is blank (Blank or empty string) are abnormal data.

[0120] Step a3: Assume that at this time, airline 1E releases a batch of new fare data, as shown in Table 4. Table 4 is the new fare data released by airline 1E. Test the data in Table 4 to determine its abnormal data.

[0121] ID Airlines Origin destination Cabin Freight amount Freight rate basis 41 1E BJS SHA Y 850 YAB1 42 1E BJS CAN Y 550 YAB2 43 1E BJS SHA C 1000 CAB1 44 1E BJS CAN C 2200 45 1E BJS SHA F 3300 FAB1 46 1E BJS CAN F 4000 FAB2

[0122] Table 4

[0123] The data in Table 4 were tested and 2 abnormal data were detected, and warning information was issued, as follows:

[0124] The recorded freight amount of ID42 is 550, which is less than the threshold of 600 in Table 3. However, its matching field is 1E / BJS / CAN / Y. There is no corresponding detection rule in the detection rule base, so the detection is skipped. However, the detection rule base will be rebuilt at a specific time the next day to create a new rule that meets the matching dimension. The new rule is used to detect the freight data to determine whether it is abnormal data.

[0125] The matching field of ID43 data is 1E / BJS / SHA / C, and its freight rate amount is 1000, which is less than the threshold of 1200 corresponding to the detection rule. This record is recorded as abnormal data;

[0126] The value of the freight rate base field of ID44 data is empty, and this record is abnormal data. Such abnormal data will not cause extreme low or high price errors when calculating freight rates, but there may be no calculation results, resulting in the inability to issue tickets, thus affecting the airline's revenue. Other data are normal.

[0127] When it is determined that the above data (ID43 data and ID44 data) may be abnormal data, it is further determined whether the above two data are abnormal data (for example, manual detection). Through manual detection, ID43 data is a new product released by 1E airline, and this data is excluded. Finally, it is determined that the above ID44 data is abnormal data, and an early warning message is issued to notify the airline user to handle the abnormal data through the early warning message.

[0128] The abnormal data processing method provided by the embodiment of the present invention can automatically detect abnormal data; through the processing method, since only real-time freight rate data is detected, the amount of data to be detected is greatly reduced, and the speed of data processing is improved.

[0129] In the above-mentioned embodiment, a method for processing abnormal data is provided, and correspondingly, the present invention also provides a device for processing abnormal data. The device for processing abnormal data provided in the embodiment of the present invention can implement the above-mentioned method for processing abnormal data, and the device for processing abnormal data can be implemented by software, hardware, or a combination of software and hardware. For example, the device for processing abnormal data can include integrated or separate functional modules or units to perform the corresponding steps in the above-mentioned methods.

[0130] Please refer to Figure 5 , which shows a schematic diagram of an abnormal data processing device provided by some embodiments of the present invention. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0131] like Figure 5 As shown, the abnormal data processing device 500 may include:

[0132] The interception module 501 is used to intercept and process the real-time freight rate data of the freight rate publishing system through a data interceptor;

[0133] A matching module 502, used to match a target detection rule strategy from a detection rule library;

[0134] The anomaly detection module 503 is used to detect the real-time freight rate data based on the target detection rule strategy, determine the abnormal data in the real-time freight rate data, and issue warning information for the abnormal data.

[0135] In some implementations of the embodiments of the present invention, the abnormal data processing device 500 provided by the embodiments of the present invention may further include:

[0136] Determine the module (in Figure 5), which is used to determine the target detection rule strategy before matching the target detection rule strategy from the detection rule base.

[0137] In some implementations of the embodiments of the present invention, the determination module is specifically used to:

[0138] Acquire multiple key parameters, the multiple key parameters include: a first ID value, multiple matching fields, and a threshold or anomaly value, the multiple matching fields include: airline, origin, destination, and class;

[0139] Determine the target detection rule strategy based on multiple key parameters.

[0140] In some implementations of the embodiments of the present invention, the determination module is further specifically used to:

[0141] Obtain a record whose freight rate basic field is empty or an empty string, and use the record as a target record; and obtain a second ID value corresponding to the target record;

[0142] The target detection rule strategy is determined based on the target record and the second ID value.

[0143] In some implementations of the embodiments of the present invention, the abnormal data processing device 500 provided by the embodiments of the present invention may further include:

[0144] Detection rule base building module (in Figure 5 ), specifically used for: after determining the target detection rule strategy, obtaining the target detection rule strategy; determining multiple detection rule strategies other than the target detection rule strategy based on multiple key parameters; and constructing a detection rule library based on the target detection rule strategy and multiple detection rule strategies, wherein the rules corresponding to the detection rule library include at least a minimum rule and a maximum rule.

[0145] In some implementations of the embodiments of the present invention, the detection rule base construction module is specifically used to:

[0146] Based on the target detection rule strategy, multiple detection rule strategies and historical data, read the value domain of each matching dimension attribute;

[0147] Based on the value domain of each matching dimension attribute, obtain the minimum value of the target field quantity under all matching dimension value combinations, and obtain the maximum value of the target field quantity under all matching dimension value combinations;

[0148] Get the abnormal low value threshold, and get the abnormal high value threshold;

[0149] Determine the minimum value rule based on the set of the minimum value of the target field quantity under all matching dimension value combinations and the product of the abnormal low value threshold;

[0150] The maximum value rule is determined based on the set of the maximum value of the target field quantity under all matching dimension value combinations multiplied by the abnormal high value threshold.

[0151] In some implementations of the embodiments of the present invention, in some implementations of the embodiments of the present invention, the abnormal data processing device 500 provided by the embodiments of the present invention is based on the same inventive concept as the abnormal data processing method provided by the aforementioned embodiments of the present invention and has the same beneficial effects.

[0152] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 1 The method described.

[0153] According to another embodiment of the present invention, there is also provided an electronic device, comprising a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, the Figure 1 The method described.

[0154] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0155] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for processing abnormal data, characterized in that: The method comprises: Through the data interceptor, the real-time freight rate data of the freight rate publishing system is intercepted and processed; Match target detection rule strategy from the detection rule library; Based on the target detection rule strategy, the real-time freight rate data is detected to determine abnormal data in the real-time freight rate data, so as to issue early warning information for the abnormal data.

2. The processing method according to claim 1, characterized in that: Before matching the target detection rule strategy from the detection rule library, the method further includes: Determine the target detection rule strategy.

3. The processing method according to claim 2, characterized in that: The determining of the target detection rule strategy includes: Acquire multiple key parameters, the multiple key parameters include: a first ID value, multiple matching fields, and a threshold or anomaly value, the multiple matching fields include: airline, origin, destination, and class; The target detection rule strategy is determined based on multiple key parameters.

4. The processing method according to claim 3, characterized in that: The determining of the target detection rule strategy also includes: Obtain a record whose freight rate basic field is empty or an empty string, and use the record as a target record; and obtain a second ID value corresponding to the target record; The target detection rule strategy is determined based on the target record and the second ID value.

5. The processing method according to claim 4, characterized in that: After determining the target detection rule strategy, the method further includes: Obtaining the target detection rule strategy; Determine a plurality of remaining detection rule strategies other than the target detection rule strategy based on the plurality of key parameters; Based on the target detection rule strategy and the multiple detection rule strategies, the detection rule base is constructed, and the rules corresponding to the detection rule base at least include a minimum value rule and a maximum value rule.

6. The processing method according to claim 5, characterized in that: The step of constructing the detection rule library based on the target detection rule strategy and the multiple detection rule strategies includes: Based on the target detection rule strategy, the multiple detection rule strategies and historical data, read the value domain of each matching dimension attribute; Based on the value domains of the attributes of each matching dimension, obtaining the minimum value of the number of target fields under all matching dimension value combinations, and obtaining the maximum value of the number of target fields under all matching dimension value combinations; Get the abnormal low value threshold, and get the abnormal high value threshold; Determine the minimum value rule based on a set of the minimum value of the number of target fields under all matching dimension value combinations and the product of the abnormal low value threshold; The maximum value rule is determined based on a set of products of the maximum value of the target field quantity under all matching dimension value combinations and the abnormally high value threshold.

7. A device for processing abnormal data, characterized in that: The device comprises: The interception module is used to intercept and process the real-time freight rate data of the freight rate publishing system through the data interceptor; A matching module is used to match target detection rule strategies from the detection rule library; The anomaly detection module is used to detect the real-time freight rate data based on the target detection rule strategy, determine the abnormal data in the real-time freight rate data, and issue early warning information for the abnormal data.

8. The processing device according to claim 7, characterized in that The device also includes: The determination module is used to determine the target detection rule strategy before matching the target detection rule strategy from the detection rule library.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 6 is implemented.

Images