Multi-factor recommendation ranking method and system for international flight itinerary search results

Abstract

The application discloses a multi-factor recommendation sorting method and system for international flight itinerary search results, relates to the technical field of computer information processing and intelligent recommendation, and comprises the following steps: acquiring multiple-source concurrent flight data, extracting original attribute features and source identifiers of the flight data through analysis and processing, obtaining a heterogeneous data set, classifying and processing the original attribute features by using a random forest algorithm according to the heterogeneous data set, identifying attribute combinations under different source identifiers, and obtaining field mapping relationships containing dynamic differences; the multi-factor recommendation sorting method and system for international flight itinerary search results improve the accuracy, rationality and recommendation efficiency of international flight itinerary search results.

Description

Technical Field

[0001] This invention relates to the field of computer information processing and intelligent recommendation technology, specifically to a multi-factor recommendation ranking method and system for international flight itinerary search results. Background Technology

[0002] With the development of the air transport industry and the increasing demand for international air travel, flight search and ticket booking systems need to process flight information from multiple data sources simultaneously to provide users with services such as flight search, price comparison, and itinerary recommendations. Existing international flight data typically comes from global distribution systems, direct airline interfaces, and third-party flight data aggregation platforms. Due to differences in data organization methods, interface specifications, and business rules among different data sources, the same flight may correspond to different data structures, field definitions, time formats, and encoding standards in different data sources. Therefore, flight search systems usually need to integrate flight data returned concurrently from multiple data sources to form a unified flight search result that can be used for subsequent comparison, filtering, and sorting.

[0003] In existing technologies, when integrating multi-source flight data, fixed field mapping rules are typically pre-configured for each data source to transform data from different sources into a unified data structure. For example, field correspondences can be pre-established for attributes such as flight number, departure airport, arrival airport, departure time, arrival time, price, and cabin class, and then the transformation of flight data from different sources can be completed based on these field correspondences. However, different data sources may have significant differences in field names, field hierarchy, time representation methods, encoding standards, and attribute combinations. Furthermore, the aforementioned data structure may change as airline interfaces, distribution systems, or business rules are adjusted. Using a fixed field mapping method is difficult to adapt to the dynamic changes of multi-source heterogeneous flight data, and it is prone to problems such as incorrect field correspondences or inaccurate alignment of some attributes, thereby affecting the reliability of subsequent flight data processing. In addition, during the concurrent return of flight data from multiple data sources, there may be cases where some attributes are missing or data records are incomplete. For example, data records from different sources may lack some time, price, cabin class, or flight segment attributes. When missing data is directly used in subsequent flight comparisons, it can easily lead to a lack of a unified and complete comparison basis between different data records, thus affecting the accuracy of flight identification, duplicate data judgment, and subsequent sorting results. Therefore, how to reduce the impact of attribute differences and data gaps in multi-source heterogeneous flight data on the unified comparison process is also an issue that needs to be considered in the processing of international flight search results. Furthermore, the same actual flight may exist simultaneously in data returned by global distribution systems, airline direct connection interfaces, and third-party aggregation platforms. Although data from different sources actually correspond to the same flight, there may be certain differences in price, cabin class description, time expression method, and flight segment organization. If duplicate flights are identified based on only a single field or a fixed combination of fields, it is easily affected by the differences in fields from different data sources and the degree of data completeness, making it difficult to reliably determine whether different flight records correspond to the same actual flight. When multiple data sources return search results concurrently, the above problems may also cause the same flight to appear in search results in the form of multiple different records, which not only increases the amount of subsequent data processing but may also affect users' judgment of different flight itineraries.

[0004] After integrating flight data and processing duplicate data, existing flight search systems typically need to sort search results based on attributes such as price, departure time, and arrival time. However, international flight itineraries involve multiple interrelated evaluation factors, including price, departure and arrival times, flight duration, and number of stopovers, and different flights often vary in these attributes. Current technologies that use single-factor sorting or sorting according to a fixed field order struggle to comprehensively reflect the overall differences between flights across multiple itinerary attributes. For example, lower-priced flights may have longer flight durations or more stopovers, while flights with better departure and arrival times may correspond to higher prices. Relying solely on a single attribute makes it difficult to generate a recommended order that considers multiple factors, potentially leading to a mismatch between the sorted results and the overall situation of the flight itinerary. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-factor recommendation and ranking method and system for international flight itinerary search results, thereby solving the problems existing in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multi-factor recommendation and ranking method for international flight itinerary search results, comprising: S1. Obtain concurrent flight data from multiple sources, extract the original attribute features and source identifiers of the flight data, and obtain a heterogeneous data set; S2. Based on the heterogeneous data set, classify the original attribute features, identify attribute combinations under different source identifiers, and obtain field mapping relationships containing dynamic differences. S3. Convert the time format and encoding standard in the heterogeneous data set through field mapping relationship, unify the attribute combination under different source identifiers, and obtain the first data alignment result; S4. If there are missing values ​​in the first data alignment result, interpolation calculation is performed on the missing values ​​to complete the attribute combinations under different source identifiers and determine the second data alignment result. S5. Based on the second data alignment results, extract the take-off and landing times and flight numbers from different sources, construct a basic feature vector, and obtain common factors; S6. If there is an extraction bias in the common factors, the common factors are standardized to eliminate the dimensional differences in the basic feature vectors and obtain standard common factors. S7. Compare similarity using standard common factors, calculate the distance between basic feature vectors under different source identifiers, and determine whether duplicate flights exist; S8. If duplicate flights exist, they are eliminated based on the distance between the basic feature vectors to obtain a unique set of flights. S9. Based on the unique flight set, sort the flight records in the unique flight set in real time to determine the final recommended results of the international flight itinerary search results.

[0007] Preferably, step S1 includes acquiring multi-source concurrent flight data streams, separating the underlying data packets in the flight data streams through packet decomposition processing; extracting packet structure features based on the underlying data packets to determine field mapping rules; extracting features from the underlying data packets through the field mapping rules to obtain an original attribute feature set; if the original attribute feature set is missing, completing the original attribute feature set to obtain a complete attribute feature set; assigning corresponding source identifiers based on the complete attribute feature set; and obtaining a heterogeneous data set by concatenating the complete attribute feature set with the source identifiers.

[0008] Preferably, step S2 includes extracting the original attribute features and source identifiers of the heterogeneous data set to construct a feature sample matrix; the feature sample matrix is ​​processed by a random forest algorithm to extract a feature splitting threshold; the original attribute features are divided using the feature splitting threshold to obtain attribute combinations; the frequency of the attribute combinations is statistically analyzed to generate a frequency distribution matrix; if the value of the frequency distribution matrix is ​​greater than the frequency threshold, association rules are extracted and dynamic differences in the association rules are identified; the initial mapping table is updated according to the dynamic differences to obtain field mapping relationships containing dynamic differences.

[0009] Preferably, step S3 includes obtaining field mapping relationships and heterogeneous data sets; parsing the heterogeneous data sets through field mapping relationships; converting them to obtain target time format and target encoding standard; reconstructing the heterogeneous data sets according to the target time format and target encoding standard; extracting reconstructed attribute combinations under different source identifiers; constructing an alignment matrix using the reconstructed attribute combinations; and uniformly reconstructing the attribute combinations through the alignment matrix to obtain the first data alignment result.

[0010] Preferably, step S4 includes obtaining the missing values ​​and corresponding attribute combinations present in the first data alignment result, and constructing an initial feature vector; constructing a hyperplane using a support vector machine based on the initial feature vector, and determining the predicted value for the missing value; if the predicted value meets a preset confidence threshold, then using the predicted value to interpolate the missing value to obtain a completed record; and completing the attribute combinations under different source identifiers based on the completed record to determine the second data alignment result.

[0011] Preferably, step S5 includes extracting flight identifiers and departure and arrival times based on the second data alignment result; matching flight routes based on flight identifiers and departure and arrival times to obtain cross-source features; orthogonally transforming the cross-source features to obtain a base vector; if the variance of the base vector is greater than a threshold, extracting the base vector to construct a deviation matrix; decomposing the singular values ​​of the deviation matrix to determine a stable comparison set; extracting cross-source takeoff and landing times and flight number features from the stable comparison set to construct a base feature vector for cross-source stable comparisons, and obtaining common factors.

[0012] Preferably, step S6 includes obtaining the mean vector and covariance matrix of the common factors to obtain the Mahalanobis distance; if the Mahalanobis distance is greater than a preset threshold, it is determined that there is an extraction bias in the common factors and the basic feature vector is extracted; extreme values ​​are calculated for the basic feature vector to obtain dimensional differences; a normalization matrix is ​​constructed based on the dimensional differences; the common factors are standardized using the normalization matrix to eliminate dimensional differences in the basic feature vector and obtain standard common factors.

[0013] Preferably, step S7 includes obtaining flight times and flight routes under different source identifiers; mapping the flight times and flight routes using standard common factors to obtain a first basic feature vector; reducing the dimensionality of the first basic feature vector using aircraft type coding to obtain a second basic feature vector; constructing a comparison matrix based on the second basic feature vector; performing a similarity comparison on the comparison matrix to obtain vector distance and determine whether there are duplicate flights.

[0014] Preferably, step S8 includes, if duplicate flights exist, obtaining the duplicate flights and their corresponding basic feature vector distances and source identifiers; matching preset priority weights and data confidence levels according to the source identifiers to obtain a source evaluation score; if the basic feature vector distance is less than a preset distance threshold, generating removal markers and retained records according to the source evaluation scores; filtering the duplicate flights through removal markers and extracting retained records to construct a target set; and using the target set to reorganize flight records under a single source identifier to obtain a unique flight set.

[0015] Preferably, step S9 includes extracting the departure time, arrival time, flight price, flight duration, and number of stopovers of each flight record in the unique flight set to generate a multidimensional recommendation factor matrix; performing a linear transformation on the multidimensional recommendation factor matrix to obtain a standard factor matrix; calculating the inner product of the standard factor matrix and a preset weight vector to obtain a comprehensive recommendation value; and sorting the unique flight set in descending order according to the comprehensive recommendation value to determine the final recommendation result of the international flight itinerary search results.

[0016] A multi-factor recommendation and ranking system for international flight itinerary search results, comprising the steps of the multi-factor recommendation and ranking method for international flight itinerary search results, the system including: The data acquisition module acquires flight data from multiple sources concurrently, and extracts the original attribute features and source identifiers of the flight data through parsing and processing to obtain a heterogeneous data set; The field mapping module uses a random forest algorithm to classify the original attribute features based on the heterogeneous data set, identifies attribute combinations under different source identifiers, and obtains field mapping relationships containing dynamic differences. The first data alignment module converts the time format and encoding standard in the heterogeneous data set through field mapping relationships, unifies the attribute combination under different source identifiers, and obtains the first data alignment result; The second data alignment module, if there are missing values ​​in the first data alignment result, uses a support vector machine to interpolate the missing values, completes the attribute combinations under different source identifiers, and determines the second data alignment result. The common factor extraction module extracts take-off and landing time and flight number features across sources based on the second data alignment results, constructs a basic feature vector for stable cross-source comparison, and obtains common factors. The common factor standardization module standardizes common factors if there is extraction bias, eliminating the dimensional differences in the basic feature vectors and obtaining standard common factors. The duplicate flight detection module compares similarity using standard common factors, calculates the distance between basic feature vectors under different source identifiers, and determines whether duplicate flights exist. The duplicate flight removal module removes duplicate flights based on the distance between the basic feature vectors, retains flight records under a single source identifier, and obtains a unique set of flights. The recommendation and ranking module extracts the departure time, arrival time, price, flight duration, and number of stopovers of each flight record from a unique flight set as recommendation factors. It standardizes each recommendation factor and calculates a comprehensive recommendation value for each flight record based on the preset weights corresponding to each recommendation factor. The flight records in the unique flight set are then ranked in real time based on the comprehensive recommendation value to determine the final recommendation result of the international flight itinerary search results.

[0017] As can be seen from the above technical solution, the present invention has the following beneficial effects: This multi-factor recommendation and ranking method and system for international flight itinerary search results improves the uniformity of heterogeneous flight data in terms of fields, time formats, and encoding standards by uniformly parsing flight data acquired concurrently from multiple sources and establishing corresponding field mapping relationships based on attribute differences from different data sources. Furthermore, it completes missing flight attributes, enhancing the integrity and comparability of cross-source flight data. By extracting and standardizing common flight factors that can be stably compared across sources, it reduces the impact of different data sources and attribute scales on flight similarity judgment, thereby improving the accuracy of duplicate flight identification and removal and reducing the repeated display of the same actual flight in search results. Simultaneously, after obtaining a unique set of flights, it comprehensively considers multiple recommendation factors such as departure time, arrival time, price, flight duration, and number of stopovers, and ranks flight records in real time based on the comprehensive recommendation value of each factor. This allows the ranking results to more comprehensively reflect the overall differences between different international flight itineraries, thereby improving the accuracy, rationality, and recommendation efficiency of international flight itinerary search results. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a bar chart comparing the field alignment accuracy of the fixed field mapping and the adaptive random forest mapping of this invention. Figure 3 This is a bar chart comparing the missing arrival time completion errors of mean interpolation and support vector machine interpolation in this invention. Figure 4 This is a comparison chart showing the average satisfaction rate changes with the cumulative recommendation ranking under two strategies: sorting by price and sorting by overall recommendation value. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1:

[0021] like Figure 1 and Figure 4 As shown, this invention provides a technical solution: a multi-factor recommendation and ranking method for international flight itinerary search results, comprising: S1. Obtain concurrent flight data from multiple sources, and extract the original attribute features and source identifiers of the flight data through parsing and processing to obtain a heterogeneous data set; S2. Based on the heterogeneous dataset, the random forest algorithm is used to classify the original attribute features, identify attribute combinations under different source identifiers, and obtain field mapping relationships containing dynamic differences. S3. Convert the time format and encoding standard in the heterogeneous data set through field mapping relationship, unify the attribute combination under different source identifiers, and obtain the first data alignment result; S4. If there are missing values ​​in the first data alignment result, a support vector machine is used to interpolate the missing values, complete the attribute combinations under different source identifiers, and determine the second data alignment result. S5. Based on the second data alignment results, extract the take-off and landing times and flight numbers from different sources, construct a basic feature vector, and obtain common factors; S6. If there is an extraction bias in the common factors, the common factors are standardized to eliminate the dimensional differences in the basic feature vectors and obtain standard common factors. S7. Compare similarity using standard common factors, calculate the distance between basic feature vectors under different source identifiers, and determine whether duplicate flights exist; S8. If duplicate flights exist, remove duplicate flights based on the distance of the basic feature vector and retain flight records under the single source identifier to obtain a unique set of flights. S9. Based on the unique flight set, extract the departure time, arrival time, price, flight duration, and number of stopovers of each flight record as recommendation factors. Standardize each recommendation factor and calculate the comprehensive recommendation value of each flight record according to the preset weights corresponding to each recommendation factor. Sort the flight records in the unique flight set in real time according to the comprehensive recommendation value to determine the final recommendation result of the international flight itinerary search results.

[0022] In this embodiment, for international flight itinerary search requests, flight records returned by multiple flight data sources are received simultaneously, and a source identifier corresponding to each flight record is configured.

[0023] Different flight data sources are allowed to use different field names, field types, time formats, and encoding forms to represent the same business attributes. Therefore, before entering the recommendation sorting, the data returned by each source is parsed and uniformly processed.

[0024] Furthermore, after acquiring flight data, a single field is used as the parsing object to extract the field name, field data type, field character length, character composition, numerical value range, date and time format, and the correspondence between different fields within the same flight record. This information is then used as the original attribute features.

[0025] For fields consisting of letters and numbers, record the letter part, the number part, and their arrangement; for time fields, record the date composition, time composition, and time zone information; for price, flight duration, and number of stops fields, record their numerical type and value structure; and associate and store the original flight records, original attribute features, and source identifiers to form a heterogeneous data set.

[0026] Furthermore, after obtaining the heterogeneous dataset, the original attribute features corresponding to each field are input into the pre-trained random forest classification model.

[0027] Specifically, the random forest classification model consists of multiple decision trees. Each decision tree classifies the current field based on the field name, field type, character composition, date and time format, and the correspondence between fields, and outputs the corresponding field category. The classification results output by all decision trees for the current field are counted, and the field category that appears most frequently is determined as the unified target field category corresponding to that field.

[0028] For example, when a field consists of an airline code and a number, and appears simultaneously with the departure and arrival times in the same flight record, the random forest classification model can identify the field as a flight number field based on its character composition, field type, and field relationships. For fields with different names from different data sources but all actually representing departure times, they are uniformly identified as departure time fields based on their date and time format, data type, and relationships with other fields.

[0029] Furthermore, after completing the field classification, the source identifier is used as the distinguishing condition to establish the correspondence between the original fields of each source and the unified target field.

[0030] Specifically, each field mapping relationship records the data source, original field, and unified target field. When the field name, field type, or field combination of a certain data source changes, the original attribute features of the changed field are re-extracted and input into the random forest classification model for field classification. The field mapping relationship corresponding to the source is updated according to the new classification result, thus forming a field mapping relationship that can be updated as the source data structure changes.

[0031] Furthermore, after completing the field mapping, fields with the same business meaning from different sources are written into a unified data structure based on the field mapping relationship.

[0032] For takeoff and arrival times, the date, time, and time zone information contained in the original time are first identified, and then converted into the time value corresponding to Coordinated Universal Time (UTC) while retaining the original time zone information. Through this process, takeoff and arrival times in different time zones can be compared according to the same time base.

[0033] For coded fields such as flight numbers, character processing is performed according to unified rules, including removing leading and trailing spaces, removing spaces used as separators within the field, standardizing the capitalization of English letters, and standardizing the form of connectors. After completing the time format conversion and encoding standard conversion, fields with the same business meaning all adopt the same data structure and expression form, forming the first data alignment result.

[0034] Subsequently, integrity checks were performed on each flight record in the first data alignment result.

[0035] Specifically, the system pre-sets the set of necessary fields for subsequent data processing and checks whether there is valid data in the corresponding fields of each flight record; when there is no valid data in a certain necessary field, the field is marked as a missing field; when there is no missing field, the first data alignment result is directly used as the second data alignment result; when there is a missing field, a support vector machine is used to perform interpolation calculation on the missing field.

[0036] For fields such as price, time, flight duration, and others that can be represented by continuous numerical values, support vector regression is used for completion.

[0037] Specifically, training samples are selected from historical flight data with complete fields. Known fields that correspond to the field to be completed are used as input, and the actual values ​​of the field to be completed are used as output to train the support vector regression model. After training, the known fields from flight records with missing fields are input into the model, and the model outputs the estimated values ​​of the corresponding missing fields, which are then written into the missing positions.

[0038] In addition, for fields represented by discrete categories, support vector classification is used for completion.

[0039] Specifically, the known attributes in the complete historical data are used as input, and the category to which the field to be completed belongs is used as the classification result to build a model; for flight records with missing fields, their known attributes are input into the model, and the field value corresponding to the missing field is determined according to the category output by the model; after completing the completion of all missing fields, the second data alignment result is obtained.

[0040] Furthermore, based on the second data alignment results, the departure time, arrival time, and flight number are extracted from each flight record.

[0041] Specifically, the departure and arrival times are converted into numerical times under a unified time base, and the flight numbers are converted into coded results that can participate in numerical comparisons according to fixed coding rules. These are then combined in a fixed order to form a basic feature vector. All flight records use the same field order and conversion rules to construct the basic feature vector, thereby forming a common factor for stable cross-source comparison.

[0042] Because the numerical ranges and measurement methods corresponding to departure time, arrival time, and flight number are inconsistent, each feature dimension of the common factors is standardized before performing similarity comparison.

[0043] Specifically, first, calculate the mean and standard deviation of the same feature dimension in the current set of flights to be compared. Then, subtract the mean of the dimension from the original value of each flight record in that feature dimension. Finally, divide the difference by the standard deviation of the dimension. After the above processing, all data in the same feature dimension are represented according to a uniform scale. When the standard deviation of a certain feature dimension is equal to zero, the standardization result corresponding to all flight records in that dimension is uniformly set to zero. After completing the processing of all feature dimensions, the standard common factor is obtained.

[0044] Furthermore, after obtaining the standard common factors, pairwise similarity comparisons are performed on flight records with different source identifiers.

[0045] Specifically, for any two flight records to be compared, the differences between their standardized departure time features, standardized arrival time features, and standardized flight number features are calculated respectively. The three differences are squared respectively, the three squared results are added together, and the square root of the sum is taken as the basic feature vector distance between the two flight records.

[0046] As a preferred embodiment, the system pre-sets a threshold for determining duplicate flights.

[0047] Specifically, when the distance between the basic feature vectors of two flight records with different source identifiers is less than or equal to the duplicate flight determination threshold, the two flight records are determined to be duplicate flights; when the distance between the basic feature vectors is greater than the duplicate flight determination threshold, the two flight records are determined to be different flights; the duplicate flight determination threshold is predetermined based on historical cross-source flight data with known matching relationships and remains unchanged during a flight search and sorting process.

[0048] In addition, for records that are determined to be duplicate flights, flight records that mutually meet the duplicate judgment criteria will be grouped into the same duplicate flight group.

[0049] For each flight record in the duplicate flight group, calculate the basic feature vector distance between it and other flight records in the group, and add these distances together to obtain the total distance corresponding to that flight record.

[0050] Compare the total distance of each flight record within the same duplicate flight group, identify the flight record with the smallest total distance as the retained record, and delete the duplicate records corresponding to the remaining source identifiers.

[0051] When there are two or more flight records with the exact same total distance in the same duplicate flight group, the flight record that appears first in the data source priority order set by the system will be selected as the retained record.

[0052] After processing all duplicate flight groups, the retained records of each duplicate flight group and the records that were not identified as duplicate flights are combined to form a unique flight set, so that only one flight record is retained for the same flight in this set.

[0053] Furthermore, after obtaining a unique set of flights, the departure time, arrival time, price, flight duration, and number of stopovers are extracted from each flight record, and these five types of data are used as recommendation factors.

[0054] Since the five recommendation factors have different numerical units and ranges, they are standardized separately.

[0055] For each recommendation factor, the maximum and minimum values ​​of that recommendation factor in the current unique flight set are first obtained.

[0056] Specifically, for any flight record, the original value of the recommendation factor corresponding to that flight record is subtracted from the minimum value of that recommendation factor, and the difference is divided by the difference between the maximum and minimum values ​​of that recommendation factor, thus obtaining a standardized result within a uniform numerical range; when the maximum and minimum values ​​of a certain recommendation factor are equal, the standardized result of that recommendation factor in all flight records is uniformly set to zero.

[0057] For price, flight duration, and number of stopovers, the system adopts a sorting rule that the smaller the value, the higher the recommendation order. Therefore, the system uses a method of subtracting the corresponding standardized result from one to perform direction conversion, so that flights with smaller original values ​​receive larger converted recommendation values.

[0058] The system pre-sets the corresponding time sorting direction for takeoff and arrival times.

[0059] Specifically, when the preset rule is that earlier times are ranked higher, the direction is reversed by subtracting the standardized result from the preset rule; when the preset rule is that later times are ranked higher, the standardized result is used directly; during a sorting process, the sorting direction of takeoff time and arrival time remains unchanged.

[0060] Furthermore, after standardizing and unifying the direction of all recommendation factors, the system assigns corresponding preset weights to departure time, arrival time, price, flight duration, and number of stopovers, ensuring that the sum of the five preset weights equals one. For each flight record, the system multiplies each of the five standardized recommendation factors by its corresponding preset weight, then adds the five products together, and the final sum is determined as the comprehensive recommendation value for that flight record.

[0061] In addition, when the system is set to prioritize flights based on their overall recommendation value, all flight records in the unique flight set are sorted from highest to lowest based on their overall recommendation value.

[0062] When two or more flight records have the same overall recommendation value, they are compared in a fixed order: price, flight duration, number of stopovers, departure time, and arrival time. Once the order of the previous data can be determined, the next data will not be compared. Only when the previous data is exactly the same will the next data be compared, thus determining a unique sorting order for all flight records.

[0063] When any data source adds a new flight record, or when the departure time, arrival time, price, flight duration, and number of stopovers in an existing flight record change, the updated flight data will be re-entered into the process of field classification, field mapping, format unification, missing value completion, common factor extraction, standardization, duplicate flight detection, and comprehensive recommendation value calculation.

[0064] The flight ranking is updated based on the recalculated comprehensive recommendation value, and the updated ranking is determined as the final recommendation result for international flight itinerary search results.

[0065] To verify this application, an experiment was designed and conducted. Experimental environment: Python 3.12; NumPy, Pandas, Scikit-learn (random forest / support vector machine / standardization), SciPy (cluster evaluation); Matplotlib plotting. The random seed was fixed at 42.

[0066] Data source: Simulated multi-source concurrent flight data, covering three typical sources: direct sales airlines (source A), OTA platforms (source B), and GDS distribution (source C), totaling 10 international routes.

[0067] Data scale: The flight pool contains 150 real flights, from which 120 are selected as the test itinerary pool; each flight is released in parallel from 1-3 sources, resulting in a total of 293 multi-source flight records before alignment, with each actual flight being repeatedly released by 2.44 sources on average.

[0068] Data characteristics: There are differences in field naming, time format, encoding standards and price caliber among different sources. Some arrival times are missing (missing rate is about 10%). The take-off and landing times, prices and number of stopovers of the same actual flight vary across different sources, which naturally constitutes the scenario of duplicate flights.

[0069] The experiment was divided into four sub-experiments according to the method of processing the link, and control groups and experimental groups were set up in each sub-experiment to verify the gain of each link.

[0070] Table 1 Experimental Processing Link

[0071] Experiment 3 Description: Construct basic feature vectors using departure time, arrival time, flight duration, price, and number of stopovers; for the unstandardized group, calculate the vector distance directly using the original dimensional features; for the standard common factor group, first standardize to eliminate dimensional differences before comparing similarity; use connected component clustering to determine duplicate flights, use the Adjusted Rand Index (ARI) to measure the consistency between the deduplicated clusters and the actual flight affiliation, and count the number of unique flights after deduplication.

[0072] Experiment 1: Alignment effect of multi-source heterogeneous fields In 293 multi-source records with 8 attribute fields (flight number, departure time, arrival time, flight duration, price, number of stopovers, cabin class code, and route), a total of 2344 fields were aligned using both fixed field mapping and random forest adaptive mapping. The accuracy of the field mapping was statistically analyzed, and the results are shown in Table 2. Table 2 Comparison of Alignment Accuracy for Different Field Mapping Strategies

[0073] like Figure 2 As shown in Table 2, the Random Forest Adaptive Mapping uses the frequency of attribute combinations under different source identifiers to identify dynamic differences and update the mapping table. The alignment accuracy increased from 83.11% to 97.99%, an increase of about 14.88 percentage points, which effectively improved the uniformity of heterogeneous flight data in terms of fields, time formats and encoding standards.

[0074] Experiment 2: Missing Arrival Time Completion Effect Approximately 10% of the 293 records were randomly introduced with missing arrival times (37 missing samples). Mean interpolation and support vector machine interpolation were used to complete the missing times. The mean absolute error (MAE) was calculated based on the actual arrival times, and the results are shown in Table 3. Table 3 Comparison of missing completion errors for different interpolation methods

[0075] like Figure 3 As shown in Table 3, the support vector machine constructs the optimal hyperplane for interpolation based on the correlation attributes such as departure time and flight duration. The completion error is reduced from 197.9 minutes to 28.5 minutes, a reduction of about 85.6%, which significantly improves the integrity and comparability of cross-source flight data.

[0076] Experiment 3: Results of Duplicate Flight Identification and Removal Duplicate flight identification and deduplication were performed on 293 multi-source records (120 actual flights and 226 pairs of actual duplicate records). Two methods, unstandardized direct comparison and standard common factor comparison, were compared. The results are shown in Table 4. Table 4 Comparison of duplicate flight identification effects under different deduplication schemes

[0077] As shown in Table 4, without standardization, differences in dimensions such as price and flight distance dominated the feature distance, leading to severe oversegmentation (a large number of records that should have been merged into the same flight were broken up, with only 12 unique flights, far lower than the actual 120). After introducing standardization to eliminate the differences in dimensions, the RAND index was adjusted from 0.001 to 0.551, and the number of unique flights was restored to 146 (close to the actual value of 120). The accuracy of identifying and removing duplicate flights was significantly improved, effectively reducing the repeated display of the same actual flight in the search results.

[0078] Experiment 4: The effect of multi-factor comprehensive recommendation ranking After deduplication, the unique set of flights was used for ranking by single price and multi-factor comprehensive recommendation value (weights: price 0.40, flight duration 0.35, number of stopovers 0.25, and combined with departure / arrival time). The recommendation quality of the top 20 flights was statistically analyzed, and the results are shown in Table 5. Table 5. Comparison of recommendation quality for different ranking strategies (Top 20)

[0079] like Figure 4As shown in Table 5, the average satisfaction score of the top 20 comprehensive recommendations increased from 75.48 to 82.01, an increase of approximately 8.65%. The Spearman rank correlation coefficient between the ranking results and users' actual satisfaction increased from 0.690 to 0.902. This indicates that by comprehensively considering recommendation factors such as departure / arrival time, price, flight duration, and number of stopovers, the overall differences between different international flight itineraries can be more fully reflected, and the ranking results are closer to user preferences.

[0080] Example 2:

[0081] S1 includes acquiring multi-source concurrent flight data streams, separating the underlying data packets from the flight data streams through packet decomposition and processing; extracting packet structure features based on the underlying data packets to determine field mapping rules; extracting features from the underlying data packets using the field mapping rules to obtain an original attribute feature set; if the original attribute feature set is missing, completing the original attribute feature set to obtain a complete attribute feature set; assigning corresponding source identifiers based on the complete attribute feature set; and obtaining a heterogeneous data set by concatenating the complete attribute feature set with the source identifiers.

[0082] In this embodiment, for the same international flight itinerary search request, the data streams returned by each flight data source are received in parallel, and written into the receiving buffer according to the communication channel corresponding to the data source.

[0083] Specifically, when decomposing the flight data stream entering the receive buffer, the message boundary information used by the current data stream is read first. For data streams containing a message length field, the length value in the message header is read, and the corresponding number of bytes are continuously truncated according to the length value, starting from the beginning position of the message header. The truncated data is used as one underlying data message. After truncating, the next byte position is used as the starting point for reading the next underlying data message, and the decomposition continues in the same way until the current data stream is completely processed. For data streams that use start and end markers to define message boundaries, reading begins from the position where the start marker is detected and continues until the end marker corresponding to the start marker is detected. The data content between the two is determined as one underlying data message.

[0084] Therefore, the continuously transmitted flight data stream is converted into underlying data packets with independent message boundaries, so that subsequent field parsing can be performed on a message-by-message basis.

[0085] Furthermore, after obtaining the underlying data message, the message structure features are analyzed by parsing the field hierarchy, field position, field name, field data type, field length, and field value composition.

[0086] Specifically, the field hierarchy is recorded according to the access order of the field from the message root node to the node where the current field is located; the field position is recorded according to the arrangement number of the field in the current hierarchy; the field length is determined according to the actual number of characters or bytes occupied by the field value; the composition of the field value is determined according to the actual arrangement of numeric characters, alphanumeric characters, date separators, time separators and other encoded characters in the field; for nested messages, the hierarchical correspondence between parent and child fields is also recorded; through the above processing, each field forms a structural feature record corresponding to the structure of its message.

[0087] When determining field mapping rules based on message structure characteristics, the field level, field position, field name, data type, and field value composition of the current field are compared item by item with the pre-saved target attribute identification conditions.

[0088] Specifically, the target attributes include the flight number, departure time, arrival time, price, flight duration, number of stopovers, and corresponding attribute description information required for subsequent flight processing; when all the structural features of the current field meet the recognition conditions corresponding to a certain target attribute, a mapping relationship is established between the field and the target attribute.

[0089] Specifically, for the flight number field, first determine whether the field value contains both the letter characters corresponding to the airline code and the numeric characters corresponding to the flight number, and then determine whether the level of the field belongs to the basic flight information level; if both conditions are met, the current field is mapped to the flight number attribute; for the departure time and arrival time fields, read the date, time, and time zone portions of the field, and determine the corresponding attribute based on the departure node or arrival node where the field is located; for the price field, confirm that the field value is data that can be converted into a numerical value, and read the currency code within the same price node to establish the correspondence between the price value and the currency type; after completing the correspondence between each target attribute and the source field in the same way, the field mapping rules applicable to the current underlying data message are obtained; when the message structure changes, the above matching process is re-executed with the characteristics of the changed message structure, and the original field positions that are inconsistent with the current message structure are not used.

[0090] In this embodiment, when extracting features through field mapping rules, the source fields in the underlying data packets are located sequentially according to each mapping relationship, and the actual data of the fields are read.

[0091] Specifically, for flight number, the airline code, numeric code, and complete flight number are extracted; for departure and arrival time, the date, time, and time zone are extracted respectively; for price, the price value and currency code are extracted; for flight duration, the duration value and corresponding time unit are extracted; for number of stops, the integer value representing the number of stops is extracted; in addition to the actual data of the field, the data type, character length, and structural position of the corresponding field are also retained, so that the actual data of the field and the data characteristics used to describe the field form a corresponding record; all attributes extracted from the current underlying data message are collected in a predetermined attribute order, thereby obtaining the original attribute feature set.

[0092] Furthermore, after the original attribute feature set is formed, the attributes in the set are checked one by one using a pre-set list of necessary attributes to see if they have valid values.

[0093] During the inspection, the attribute names in the necessary attribute list are matched sequentially with the attribute names in the original attribute feature set; if a corresponding field exists and the field value can be parsed according to its target data type, the attribute is considered complete; if a corresponding field does not exist, the field is empty, or the field value cannot be converted to the target data type, the attribute is considered missing.

[0094] In addition, for any missing original attribute feature sets, the system first re-searches for alternative fields corresponding to the missing attributes in the current underlying data message according to the field mapping rules. If an alternative field that meets the target attribute identification conditions exists in the same message, it is read and added to the corresponding position. If no corresponding alternative field exists in the current underlying data message, records with the same flight identifier are searched in other underlying data messages from the same data source based on the obtained flight number, departure date, and departure airport information. When the search result is unique, the corresponding attribute in that record is read and added to the current original attribute feature set.

[0095] For missing attributes that can be directly calculated based on existing fields, the calculation is performed according to the defined relationships between the fields.

[0096] Specifically, when the flight duration is transmitted in hour and minute fields respectively, the hour number is multiplied by 60 to get the corresponding minute number, which is then added to the minute field value. The result is used as the flight duration uniformly expressed in minutes. When the number of stops is not directly given but the number of complete flight segments is obtained, the number of flight segments is subtracted by 1, and the resulting non-negative integer is used as the number of stops. When the number of flight segments is 1, the number of stops is determined to be 0. After completing the missing attribute supplementation, each item in the list of necessary attributes is checked again. When each necessary attribute has a valid value, the current set is determined as the complete attribute feature set.

[0097] Furthermore, after the complete set of attribute features is formed, the source identifier is determined based on the data source corresponding to the current underlying data packet when it enters the receive buffer.

[0098] Specifically, each data source is pre-configured with a unique source identifier. All underlying data packets generated by the same data source use the same source identifier, while different data sources use different source identifiers. The determined source identifier is written into the source field of the complete set of attribute features, and the source identifier, flight number, departure time, arrival time, price, flight duration, number of stopovers, and corresponding attribute features are continuously written into the same data record in a unified record order.

[0099] After completing the above processing of all underlying data messages corresponding to this international flight itinerary search request, the complete attribute feature records of each source identifier that has been spliced ​​together are collected to obtain a heterogeneous data set.

[0100] Example 3:

[0101] S2 includes extracting the original attribute features and source identifiers of the heterogeneous data set to construct a feature sample matrix. The feature sample matrix is ​​processed by a random forest algorithm to extract a feature splitting threshold. The original attribute features are divided by the feature splitting threshold to obtain attribute combinations. The frequency of the attribute combinations is statistically analyzed to generate a frequency distribution matrix. If the value of the frequency distribution matrix is ​​greater than the frequency threshold, association rules are extracted and dynamic differences in the association rules are identified. The initial mapping table is updated according to the dynamic differences to obtain the field mapping relationship containing the dynamic differences.

[0102] In this embodiment, flight records are first read one by one from the heterogeneous data set, and the original attribute features and source identifiers corresponding to each flight record are written into the same sample row; the numerical features in the original attribute features are directly written as the original values; the field type, character composition, time format, field level and encoding form are converted into unique integer codes according to the pre-established discrete encoding table; each sample is arranged in the same field order so that the same column always corresponds to the same original attribute feature, and a feature sample matrix is ​​formed after all samples are arranged.

[0103] Furthermore, the feature sample matrix is ​​input into the random forest for processing.

[0104] Specifically, for the current node to be split in each decision tree, the values ​​of the candidate features in all samples of that node are read and the different values ​​are arranged in ascending order of value. For two adjacent values ​​with different values, the sum of the two values ​​and the result divided by 2 is taken as a candidate split threshold. Then, the samples of the current node are divided according to each candidate split threshold. Samples with feature values ​​not greater than the candidate split threshold are assigned to the first subset, and samples with feature values ​​greater than the candidate split threshold are assigned to the second subset.

[0105] For each candidate split threshold, the number of samples for each field category in the two subsets is counted. The number of samples for each category is divided by the total number of samples in the subset to obtain the category proportion. The proportions of each category are multiplied by themselves and then added together. The Gini impurity of the corresponding subset is obtained by subtracting the sum from 1.

[0106] The subset weights are obtained by dividing the number of samples in each subset by the total number of samples in the node before the split. The Gini impurity of each subset is multiplied by the corresponding weight and then summed to obtain the weighted impurity corresponding to the candidate split threshold. The calculation of all candidate split thresholds for the current candidate feature is completed in sequence, and the candidate split threshold with the smallest weighted impurity value is selected as the split threshold for the feature. The minimum weighted impurity corresponding to each candidate feature of the current node is compared, and the feature with the smallest value and its split threshold are used to complete the split of the current node. The features that actually participated in the node split in the random forest and their corresponding split thresholds are collected to obtain the feature split thresholds used for the original attribute feature split.

[0107] Furthermore, after obtaining the feature splitting threshold, the original attribute features are processed into intervals.

[0108] Specifically, for features with one splitting threshold, the feature value is divided into two intervals: one not greater than the splitting threshold and one greater than the splitting threshold. For features with two or more splitting thresholds, all splitting thresholds are first arranged in ascending order of value, and then adjacent splitting thresholds are used as interval boundaries to assign the original feature value to the corresponding interval. Discrete attributes do not perform continuous interval calculations; instead, the category branches formed by the decision tree are directly used as the attribute state. The field name feature, field type feature, field structure feature, and corresponding interval state in the same flight record are connected in a fixed order to form the attribute combination corresponding to the flight record.

[0109] Furthermore, after the attribute combinations are formed, their frequency of occurrence is counted according to the source identifier.

[0110] Specifically, under the same source identifier, first count the total number of flight records participating in the current analysis, then count the number of flight records that have the specified attribute combination, divide the number of records with the specified attribute combination by the total number of flight records participating in the analysis under that source, and use the result as the frequency of occurrence of the attribute combination under that source; with the source identifier as the row and the attribute combination as the column, write the frequency of occurrence of the corresponding attribute combination for each source into the intersection position to form a frequency distribution matrix.

[0111] As a preferred embodiment, the frequency threshold is determined by historical field mapping samples.

[0112] In practice, historical heterogeneous flight data that has undergone manual field verification is selected, and attribute combinations that can be correctly mapped to a unified target field are marked as valid samples, while attribute combinations that cannot be mapped to the unified target field are marked as invalid samples. The frequency of occurrence of each historical attribute combination is calculated, and all different frequency values ​​are sorted in ascending order. Every two adjacent different frequency values ​​are added together and divided by 2 to form a candidate frequency threshold. For each candidate frequency threshold, attribute combinations with frequencies greater than the candidate value are judged as valid associations, and the remaining attribute combinations are judged as invalid associations. The judgment results are then compared with the historical manual verification results one by one, and the number of correctly judged samples is counted. The candidate frequency threshold corresponding to the largest number of correctly judged samples is selected as the actual frequency threshold. When there are more than two candidate values ​​corresponding to the same number of correct samples, the candidate value with the largest value is used to restrict low-frequency attribute combinations from entering the subsequent association rule extraction process. After the frequency threshold is determined, it is written into the configuration data and remains unchanged in the field mapping processing of the same version.

[0113] Furthermore, the frequency distribution matrix is ​​read item by item. When the frequency of a certain attribute combination under a certain source is greater than the frequency threshold, the association rule corresponding to that attribute combination is extracted.

[0114] Specifically, the association rule is determined based on the co-occurrence relationship between the attribute combination and the unified target field: the number of flight records containing the current attribute combination is counted, and then the number of flight records actually mapped to the specified unified target field is counted. The latter number is divided by the former number to obtain the association confidence result between the attribute combination and the specified unified target field. This calculation is performed on the current attribute combination and each unified target field respectively, and the unified target field with the largest association confidence result is taken as the mapping field corresponding to the attribute combination. When the association confidence results of two mapping fields are the same, the mapping field with a larger number of co-occurring records in the historical samples is taken as the current mapping field.

[0115] Furthermore, after obtaining the association rules, attribute combinations pointing to the same unified target field are compared according to the source identifier.

[0116] If the source identifiers are different, and the corresponding attribute combinations differ in any one of the following: field name, field level, data type, character composition, or value range, then the difference is recorded as a cross-source dynamic difference. For the same source identifier, the association rules formed in the current batch are compared item by item with the valid association rules of the previous batch. When the unified target field remains consistent, but the field name, field level, data type, character composition, or value range changes, the changed attribute item is recorded as a time-series dynamic difference.

[0117] Finally, the initial mapping table is read, and the corresponding mapping record is found using the source identifier and the unified target field as the location conditions.

[0118] If no corresponding record is found, the current source identifier, attribute combination, unified target field, and dynamic difference content are added to the initial mapping table. If a corresponding record already exists and the current attribute combination is consistent with the original record, the original mapping record is kept unchanged. If a corresponding record already exists but the attribute combination has changed, the changed attribute combination is written to the current mapping record, and the changed attribute items are recorded synchronously. After processing all valid attribute combinations, a field mapping relationship containing cross-source differences and changes in fields from the same source is obtained.

[0119] Example 4:

[0120] S3 includes obtaining field mapping relationships and heterogeneous data sets; parsing the heterogeneous data sets through field mapping relationships; converting them to obtain target time format and target encoding standard; reconstructing the heterogeneous data sets according to the target time format and target encoding standard; extracting reconstructed attribute combinations under different source identifiers; constructing an alignment matrix using the reconstructed attribute combinations; and unifying the reconstructed attribute combinations through the alignment matrix to obtain the first data alignment result.

[0121] In this embodiment, the field mapping relationship and heterogeneous data set obtained in Embodiment 3 are read, and the corresponding field mapping record is determined according to the source identifier carried by each flight record in the heterogeneous data set.

[0122] For a flight record to be processed, first retrieve the field mapping relationship by the source identifier, and then, based on the correspondence between the source field, field level and unified target field corresponding to the source, obtain the departure time, arrival time, flight number, airport code, airline code and field values ​​required for subsequent processing from the flight record in sequence; thus, locate the original fields used by different sources to the unified target field.

[0123] For takeoff and arrival times, first parse the year, month, date, hour, minute, second, and time zone offset from the original time field.

[0124] When the original time field does not record a second value, the second value will be filled with 0. When the original time is in 12-hour format, it will first be converted to 24-hour format based on the AM or PM mark. 12 AM will be converted to 0 PM, and 1 PM to 11 PM will be added to the original hours by 12. After the time format conversion is completed, the time zone unification process will be performed.

[0125] In addition, time zones are unified with Coordinated Universal Time as the target time base.

[0126] Specifically, if the original time zone is east of Coordinated Universal Time (UTC), the time zone offset is converted into minutes and subtracted from the original local time; if the original time zone is west of UTC, the time zone offset is converted into minutes and added to the original local time.

[0127] For example, when the time zone is shifted to UTC+8, multiply 8 by 60 to get 480 minutes, then subtract 480 minutes from the local time. If the conversion involves crossing days, adjust the date accordingly based on the actual number of minutes added or subtracted. If the conversion involves crossing months or years, continue adjusting according to the actual number of days in the corresponding month and the order of the Gregorian calendar year. After the conversion is complete, the time is written in a fixed structure of 4-digit year, 2-digit month, 2-digit day, 2-digit hour, 2-digit minute, and 2-digit second, thus obtaining the target time format.

[0128] For encoded fields, conversion is performed according to unified character rules and code correspondence.

[0129] Specifically, for flight numbers, first, leading and trailing spaces are removed, then the separating spaces between the airline code and the numerical designation are removed, and all English letters are converted to uppercase. Airport codes and airline codes are also uniformly converted to consecutive uppercase characters. If a data source uses internal airport codes or internal airline codes, a pre-saved code mapping table is read, and the corresponding unified code is obtained by using the source code as the search key. The original source code is then replaced with the unified code. After character format unification and code replacement, the target encoding standard is formed. For the same business object, different sources use the same encoding content and arrangement after conversion.

[0130] Furthermore, after obtaining the target time format and target encoding standard, the heterogeneous data set is reconstructed.

[0131] During reconstruction, a fixed field order is pre-set according to the unified target field, and each source field is written to the corresponding target field position according to the field mapping relationship; time-type fields are written with time values ​​that have been converted to Coordinated Universal Time, encoding-type fields are written with encoding values ​​that conform to the target encoding standard, and numerical fields such as price, flight duration, and number of stops are written with their corresponding valid values; target fields that do not exist in the original source retain their corresponding field positions and are marked as null values, and are not replaced by subsequent fields.

[0132] In this embodiment, after each flight record is reconstructed, the data type, time format status, encoding status and source identifier corresponding to each unified target field in the record are extracted in a fixed field order, and these attributes are combined in sequence to obtain the reconstruction attribute combination corresponding to the source.

[0133] Reconstructed attribute combinations from different sources use the same field order, ensuring that the same unified target field has the same structural position in each source.

[0134] Furthermore, an alignment matrix is ​​constructed after obtaining the combination of reconstructed attributes from various sources.

[0135] In specific processing, the source identifier is used as the row index of the matrix, and the unified target field is used as the column index of the matrix. The reconstruction results of the target field corresponding to each source are written to the intersection of the row and column. When a source has a corresponding target field, the transformation result of that field is written to the corresponding position. When there is no corresponding target field, a null value is reserved at that position. In this way, each matrix row corresponds to one source, and each matrix column always corresponds to the same unified target field.

[0136] Then check the alignment matrix column by column.

[0137] For the departure and arrival time columns, verify that the corresponding values ​​from each source all use Coordinated Universal Time (UTC) and a fixed year-month-day-hour-minute-second structure; for fields that do not meet this structure, re-execute time parsing and time zone conversion; for the flight number, airport code, and airline code columns, verify the results of letter case handling, space processing, and code table conversion; for fields that do not conform to the target encoding standard, re-execute encoding conversion; for cases where the data types of fields in the same column are inconsistent, perform conversion according to the data type preset by the unified target field, for example, convert the number of stopovers to integers and the flight duration to minutes.

[0138] When flight duration is recorded separately in hours and minutes, the number of hours is multiplied by 60 and then added to the number of minutes to obtain a uniform number of minutes. If the original field is already expressed in minutes, its number of minutes is directly retained. In this way, the same target field uses the same unit of measurement under different sources.

[0139] Furthermore, after completing the format, encoding, and data type checks of all columns, the flight records from each source are regenerated according to the order of the unified target fields in the alignment matrix; the same unified target field is located in the same field position in each source flight record and uses the same time format, encoding standard, and data type; fields that still do not have valid values ​​are left in a null state and are not estimated or filled in this step.

[0140] After all flight records that have undergone unified processing are aggregated along with their source identifiers, the first data alignment result is obtained.

[0141] Example 5:

[0142] S4 includes obtaining the missing values ​​and corresponding attribute combinations in the first data alignment result, and constructing an initial feature vector; constructing a hyperplane using a support vector machine based on the initial feature vector, and determining the predicted value for the missing value; if the predicted value meets the preset confidence threshold, then interpolating the missing value using the predicted value to obtain the completed record; and completing the attribute combinations under different source identifiers based on the completed record to determine the second data alignment result.

[0143] In this embodiment, the integrity of each flight record in the first data alignment result is first checked according to a uniform field order.

[0144] For any target field, if the corresponding field position is empty, the field content cannot be parsed according to the data type specified by the field, or the field content does not meet the pre-defined valid value conditions of the field, the field is identified as a missing value, and the remaining valid attributes that have been aligned in the record containing the missing value are obtained.

[0145] Furthermore, for each missing field, a set of input attributes for predicting that field is pre-determined, and the same attribute order is used in the training phase and the actual completion phase.

[0146] Taking the missing price field as an example, the departure time, arrival time, flight duration, number of stops, flight number code, and source identifier already obtained in the flight record are used as input attributes in sequence; for the missing flight duration, the departure time, arrival time, price, number of stops, flight number code, and source identifier are used as input attributes; the input attributes are arranged in a fixed order to form the initial feature vector corresponding to the missing field.

[0147] In addition, continuous numerical attributes undergo scaling uniformity before entering the support vector machine.

[0148] Specifically, the maximum and minimum values ​​of the same attribute in the training samples are read, the minimum value is subtracted from the value of the attribute to be processed, and then divided by the difference between the maximum and minimum values. The result is used as the standardized value of the attribute. When the maximum and minimum values ​​are the same, the standardized value of the attribute is uniformly set to 0. The source identifier, flight number code and other discrete attributes are converted into integer values ​​according to the pre-established one-to-one correspondence coding table, without changing the correspondence between the same code in the training data and the data to be completed.

[0149] Subsequently, records with no missing target fields and valid values ​​for all input attributes are selected from the first data alignment results and historical complete flight data as training samples; the input attributes in each training sample are used as inputs to the support vector machine, and the actual values ​​of the corresponding target fields are used as training results; support vector regression models are established for different fields to be completed to avoid mutual influence of the numerical distributions of different fields.

[0150] In this embodiment, the support vector regression model uses the radial basis kernel function to establish the prediction relationship.

[0151] During training, the position of the hyperplane is adjusted based on the training samples, so that training samples falling within the allowable error range are not penalized, and samples exceeding the allowable error range are penalized according to the excess portion, while limiting the complexity of the hyperplane itself; by solving for the parameter combination that minimizes the combined result of model complexity and prediction error, the optimal hyperplane for prediction of the current field is obtained; training samples located on or beyond the error boundary and that have an actual constraint effect on the position of the hyperplane are retained as support vectors.

[0152] The penalty coefficient, kernel width, and allowable error width are determined using validation samples.

[0153] In practice, the complete sample is divided into a training part and a validation part. Support vector regression models are built for each set of pre-set candidate parameters. The validation part is then input into the corresponding model one by one to obtain the predicted value. For each validation record, the absolute value of the difference between the predicted value and the actual value of the record is taken to obtain the prediction error of that record. The prediction errors of all validation records are accumulated and then divided by the total number of validation records to obtain the mean absolute error corresponding to that set of parameters. All candidate parameter combinations are compared, and the set of parameters with the smallest mean absolute error is selected to build the final model. When the mean absolute errors of two or more sets of parameters are exactly the same, the set of parameters with the fewest support vectors is used to reduce the number of samples involved in the calculation during prediction.

[0154] Furthermore, after determining the hyperplane, the initial feature vectors of the records to be completed are standardized according to the same rules used in the training phase, and then input into the corresponding support vector regression model.

[0155] During prediction, the numerical difference between the initial feature vector and each support vector for each input attribute is calculated. The numerical differences corresponding to each attribute are squared and summed to obtain the squared distance between the record and the corresponding support vector. Then, the squared distance is radially transformed according to the kernel function width determined during the training phase to obtain the similarity between the record to be completed and each support vector. Each similarity is multiplied by the weight determined during the training process of the corresponding support vector, and all products are accumulated. Combined with the bias value corresponding to the optimal hyperplane, the result is the predicted value of the missing field.

[0156] To avoid the prediction results deviating from the effective distribution of the training data, the confidence level of the predicted value is further calculated after obtaining the predicted value.

[0157] Specifically, the maximum value among the similarities of the aforementioned support vectors is first taken as the feature proximity between the current record to be completed and the training data; then, the prediction stability of the model itself is determined based on the prediction error of the model on the validation samples.

[0158] The calculation process for the model prediction stability is as follows: First, subtract the minimum value from the maximum value of the target field in the training samples to obtain the effective value range of the field. Then, divide the average absolute error of the model in the validation part by the value range to obtain the standardized error. Finally, subtract the standardized error from 1 to obtain the model prediction stability.

[0159] Finally, the feature similarity of the current record is multiplied by the stability of the model prediction, and the result is used as the confidence level of the predicted value.

[0160] As a preferred implementation, the confidence threshold is determined before the model is formally used for missing value completion.

[0161] Specifically, the target field in the validation section is masked one by one, making the field temporarily a missing field, and then re-predicted using the already trained support vector regression model to obtain the predicted value and confidence score for each validation record. Subsequently, the absolute difference between the predicted value and the original actual value is calculated, and the allowable error width determined during the model training phase is used as the judgment boundary for whether the prediction result is valid. If the absolute difference is not greater than the allowable error width, the validation record is determined as a valid prediction; if the absolute difference is greater than the allowable error width, it is determined as an invalid prediction.

[0162] Furthermore, all different confidence values ​​appearing in the verification records are arranged in ascending order, and the average of two adjacent different confidence values ​​is taken as the candidate confidence threshold.

[0163] Specifically, for each candidate confidence threshold, records with a confidence level that reaches or exceeds the candidate value are determined to be allowed to use the predicted value, while records with a confidence level below the candidate value are determined not to use the predicted value. The determination result is then compared with the aforementioned effective and invalid prediction results one by one, and the number of records with consistent judgments is counted. The candidate value corresponding to the largest number of consistent judgments is used as the preset confidence threshold. When there are two or more candidate values ​​corresponding to the same maximum number of consistent records, the candidate value with the largest value is used. The determined confidence threshold is associated with and saved with the corresponding field to be completed, and remains unchanged in the current model version.

[0164] In addition, when the confidence level of the predicted value corresponding to the missing field reaches or exceeds the preset confidence threshold, the predicted value is used for interpolation.

[0165] For the number of stopovers, the predicted value is taken as the nearest non-negative integer; for the flight duration, the predicted value is converted into an integer uniformly expressed in minutes; for the price, the number of decimal places is retained according to the price field specified in the first data alignment result; for the time field, the predicted result is converted into a uniform time format consistent with the first data alignment result; after completing the type conversion, the obtained value is written into the original missing field position to form a complete record.

[0166] Finally, the source identifier and flight record identifier are used to locate the corresponding position of the completion record in the first data alignment result, and the completion value is written into the original empty field without changing the already unified field name, field order, time format and encoding standard.

[0167] The missing fields that meet the confidence level conditions are processed sequentially as described above, and the completed flight records are reassembled according to the original source identifier to obtain the second data alignment result.

[0168] Example 6:

[0169] S5 includes extracting flight identifiers and departure and arrival times based on the second data alignment results; matching flight routes based on flight identifiers and departure and arrival times to obtain cross-source features; orthogonally transforming the cross-source features to obtain a base vector; if the variance of the base vector is greater than a threshold, extracting the base vector to construct a deviation matrix; decomposing the singular values ​​of the deviation matrix to determine a stable comparison set; extracting cross-source takeoff and landing times and flight number features from the stable comparison set to construct a base feature vector for cross-source stable comparisons and obtain common factors.

[0170] In this embodiment, flight records are read one by one from the second data alignment result, and the flight number, departure date, departure time and arrival time are obtained according to the unified field position.

[0171] The flight number uses the field value after the coding has been unified, and the departure time and arrival time use the time value under the unified time base; the flight number and departure date are combined to form a flight identifier, which is used to distinguish the flight records corresponding to the same flight number on different execution dates.

[0172] When matching flight routes based on flight identifiers and departure and arrival times, the corresponding records are first retrieved from a pre-established route trajectory database using flight number and departure date.

[0173] Specifically, each record in the route trajectory database includes at least the flight number, execution date, planned departure time, planned arrival time, and trajectory nodes arranged in flight sequence. For each candidate trajectory retrieved with the same flight number and execution date, the absolute time difference between the planned departure time of the candidate trajectory and the departure time of the current flight record is calculated, and the absolute time difference between the planned arrival time of the candidate trajectory and the arrival time of the current flight record is also calculated. The two absolute time differences are then added together, and the total time difference is used as the matching value of the current candidate trajectory. The route trajectory with the smallest matching value is selected as the route trajectory corresponding to the current flight record. When there are the same minimum matching value, the departure time difference and arrival time difference are compared in turn, and the trajectory with the smallest departure time difference is selected first. When the departure time differences are still the same, the trajectory with the smallest arrival time difference is selected.

[0174] Furthermore, after completing trajectory matching, flight number codes, departure times, arrival times, departure airports, arrival airports, and trajectory node sequences are extracted from flight records and corresponding route trajectories to form cross-source features.

[0175] Specifically, departure and arrival times are converted into minutes accumulated from a unified time starting point; flight numbers, departure airports, arrival airports, and trajectory nodes are converted into numerical codes according to a fixed coding table; and flight records from all sources are arranged in the same feature order so that the same position always corresponds to the same cross-source feature.

[0176] Then, an orthogonal transformation is performed on the cross-source features.

[0177] First, calculate the average value of all flight records for each feature. Then, subtract the average value from the original feature value of each record to obtain the centered feature value. For any two features, multiply the two centered feature values ​​corresponding to the same flight record, sum the products obtained for all records, and then divide by the number of flight records involved in the calculation minus 1 to obtain the covariance between the two features. Form a covariance matrix according to the arrangement relationship of the variance of each feature and the covariance between features.

[0178] Furthermore, the covariance matrix is ​​decomposed into eigenvalues ​​and eigenvectors corresponding to each characteristic direction.

[0179] Specifically, the square root of the sum of the squares of each component in each feature vector is taken to obtain the length of the feature vector. Then, each component is divided by the length to make the length of each feature vector uniformly 1. The normalized feature vectors are arranged in descending order of their corresponding feature values, thus forming mutually orthogonal transformation directions. For each centered cross-source feature record, each feature value is multiplied by the corresponding component in one orthogonal transformation direction, and all products are added together. The result is used as the projection value of the record in that orthogonal direction. After projecting in all orthogonal directions, the resulting projection values ​​are arranged in order to obtain the base vector corresponding to the flight record.

[0180] Furthermore, after the basic vectors are formed, the variance is calculated for each dimension.

[0181] Specifically, for a single basic vector dimension, the values ​​of all flight records for that dimension are summed and divided by the number of records to obtain the average value of that dimension. Then, the average value is subtracted from the value of each record for that dimension, and the resulting difference is squared. The sum of all squared results is then divided by the number of records minus 1 to obtain the variance of that dimension. When the variance of any basic vector dimension is greater than a preset variance threshold, the basic vector corresponding to that dimension is included in the bias analysis.

[0182] The variance threshold is predetermined based on historical samples from which cross-origin flight consistency verification has been completed.

[0183] Specifically, the variance of each basic vector dimension of the historical samples is first obtained according to the aforementioned calculation process, and the corresponding dimension is marked according to the manual verification results to indicate whether there is a data offset that affects the stable comparison across sources. All different variance values ​​in the historical samples are arranged in ascending order, and the sum of two adjacent different variance values ​​is divided by 2 to obtain the candidate variance threshold. For each candidate variance threshold, historical records with variance greater than the candidate value are judged to have data offset, and the remaining records are judged to have no data offset. Then, they are compared with the manual verification results one by one. The number of records with consistent judgment corresponding to each candidate variance threshold is counted, and the candidate value with the largest number of consistent records is used as the preset variance threshold. When there are more than two candidate values ​​corresponding to the same maximum number of consistent records, the candidate value with the smallest value is used. The determined variance threshold remains unchanged in the current feature processing version.

[0184] In addition, for the base vector whose variance exceeds the threshold, records corresponding to the same flight from different sources are grouped into the same comparison group according to flight number and execution date, and a deviation matrix is ​​constructed accordingly.

[0185] Specifically, for each basic vector dimension within the comparison group, the values ​​of each source in that dimension are first added together and then divided by the number of source records to obtain the average value within the group; then, the average value within the group is subtracted from the actual value of each source to obtain the deviation value of each source in that dimension; with the source flight records as rows and the basic vector dimensions as columns, each deviation value is written into the corresponding position to form a deviation matrix.

[0186] When performing singular value decomposition on the deviation matrix, the deviation matrix is ​​decomposed into the deviation direction of the source record, the singular value corresponding to each deviation direction, and the deviation direction of the basic vector dimension.

[0187] The singular values ​​are arranged from largest to smallest. The singular value values ​​represent the degree of data dispersion caused by the corresponding deviation direction between records from different sources.

[0188] In this embodiment, the singular value bounds used to determine the stable comparison set are also determined based on historical cross-source flight samples that have completed consistency verification.

[0189] Specifically, the same singular value decomposition is first performed on the historical deviation matrix. The different singular values ​​obtained are arranged in ascending order, and the sum of two adjacent different singular values ​​is divided by 2 to obtain the candidate singular value bounds. For each candidate bound, deviation directions not greater than the bound are determined as stable directions, and deviation directions greater than the bound are determined as unstable directions. The results are compared with the stability results of historical manual verification item by item. The candidate value with the largest number of consistent judgments is used as the singular value bound. When there are candidates with the same maximum number of consistent judgments, the candidate value with the smallest value is used.

[0190] Furthermore, based on the determined singular value limit, deviation directions with singular values ​​not greater than the limit are retained, deviation directions with singular values ​​greater than the limit are removed, and the retained deviation directions are used to recombine the basic vectors.

[0191] Subsequently, an inverse transformation is performed based on the orthogonal feature vectors saved in the orthogonal transformation stage. The recombined base vectors are projected back to the original cross-source feature space, and the average value of each cross-source feature calculated before the orthogonal transformation is added back to obtain the cross-source feature record after bias removal. The cross-source feature record constitutes a stable comparison set.

[0192] Finally, the departure time, arrival time, and flight number features corresponding to each source flight record are extracted sequentially from the stable comparison set.

[0193] Departure and arrival times are kept in numerical form under a unified time base, and flight numbers are kept in numerical form after unified encoding. They are arranged in a fixed order of departure time, arrival time, and flight number to form a basic feature vector for stable comparison across sources. After collecting the basic feature vectors corresponding to each flight record, a common factor is obtained for subsequent standardization processing and duplicate flight determination.

[0194] Example 7:

[0195] S6 includes obtaining the common factors, calculating the mean vector and covariance matrix, and obtaining the Mahalanobis distance; if the Mahalanobis distance is greater than a preset threshold, it is determined that there is an extraction bias in the common factors and the basic feature vector is extracted; the extreme values ​​of the basic feature vector are calculated to obtain the dimensional differences; a normalization matrix is ​​constructed based on the dimensional differences; the common factors are standardized using the normalization matrix to eliminate the dimensional differences in the basic feature vector and obtain the standard common factors.

[0196] In this embodiment, the common factors obtained in Embodiment 6 are read, and the basic feature vectors corresponding to each flight record are obtained in the fixed field order of departure time, arrival time, and flight number.

[0197] For each feature dimension, the values ​​of all flight records for that dimension are summed, and then divided by the number of flight records involved in the calculation to obtain the average value for that dimension. After obtaining the average values ​​for each feature dimension in turn, they are arranged in the field order of the basic feature vector to form the mean vector.

[0198] In this embodiment, the covariance matrix is ​​calculated according to the common variation relationship between each feature dimension.

[0199] Specifically, for any two feature dimensions, the average value of each feature dimension is subtracted from the actual feature value of each flight record to obtain two centered feature values. The two centered feature values ​​corresponding to the same flight record are then multiplied together. The product of all flight records is summed and divided by the number of flight records minus 1. The result is used as the covariance between the two feature dimensions. For the same feature dimension, the centered feature values ​​are squared and summed, and then divided by the number of flight records minus 1. The result is used as the variance of that dimension. The variances and covariances are written in a fixed order according to the feature dimensions to form a covariance matrix.

[0200] When calculating the Mahalanobis distance for each basic feature vector, first subtract the average value of the corresponding position in the mean vector from the values ​​of each dimension in the basic feature vector to form the deviation vector; then invert the covariance matrix, multiply the deviation vector, the inverse of the covariance matrix, and the transpose of the deviation vector in the order of matrix multiplication, and take the square root of the scalar result to obtain the Mahalanobis distance corresponding to the flight record.

[0201] Matrix multiplication is performed by multiplying the values ​​at corresponding positions in rows and columns one by one and then accumulating them, so that the correlation between departure time, arrival time and flight number can be used to calculate the deviation simultaneously. When there is a feature dimension with zero variance in the covariance matrix, the dimension is removed from the Mahalanobis distance calculation and the invertible covariance matrix is ​​constructed with the remaining feature dimensions with non-zero variance.

[0202] In this embodiment, the preset threshold for judging extraction deviation is determined based on historical cross-source flight samples that have been manually verified.

[0203] Specifically, Mahalanobis distances are calculated for historical samples using the method described above, and the presence of extraction bias is marked based on manual verification results. Different Mahalanobis distances in the historical samples are arranged in ascending order of value. The sum of any two adjacent different Mahalanobis distances is divided by 2 to obtain candidate thresholds. For each candidate threshold, samples with Mahalanobis distances greater than the threshold are considered to have extraction bias, while the remaining samples are considered to have no extraction bias. The judgment results are then compared with the manual verification results one by one. The number of consistent records corresponding to each candidate threshold is counted, and the candidate value with the largest number of consistent records is used as the preset threshold. When two or more candidate values ​​correspond to the same maximum number of consistent records, the candidate value with the smallest value is used. The determined preset thresholds remain unchanged in the current version of the public factor processing.

[0204] Specifically, when the Mahalanobis distance corresponding to any basic feature vector is greater than the preset threshold, the current common factor is determined to have an extraction bias, and all basic feature vectors constituting the common factor are extracted and entered into dimensional processing.

[0205] For the three feature dimensions of departure time, arrival time, and flight number, the maximum and minimum values ​​of all basic feature vectors of each dimension are read, and the difference between the maximum and minimum values ​​is used as the value span of the feature dimension. The value span of each feature dimension represents the dimensional range of the corresponding basic feature in the current public factor.

[0206] The range of values ​​for each feature dimension is then compared. When the range of values ​​corresponding to departure time, arrival time, and flight number are not the same, the difference between the ranges of values ​​for each dimension is determined as a difference in dimensionality.

[0207] For example, when departure time is expressed in minutes, arrival time is expressed in minutes, and flight number is expressed in numerical code, the range of values ​​corresponding to the three features is directly determined by the difference between their respective maximum and minimum values, without directly comparing across dimensions using the original units of measurement.

[0208] Furthermore, a normalized matrix is ​​constructed based on the minimum value and value range of each feature dimension.

[0209] Specifically, the normalization matrix sets corresponding columns according to the field order of the basic feature vector. Each column records the minimum value and value range of the feature dimension, so that the departure time, arrival time and flight number correspond to fixed normalization parameters. When the maximum value and minimum value of a feature dimension are equal, the value range of the dimension is recorded as 0, and the standardization result of the dimension is uniformly set to 0, without performing division operation.

[0210] When standardizing common factors using a normalized matrix, the basic feature vectors are read one by one.

[0211] For any feature value, first subtract the minimum value recorded in the corresponding feature dimension in the normalization matrix, then divide the difference by the value range of that dimension, and use the result as the standardized value of that feature; process departure time, arrival time and flight number in the same way to convert all features with non-zero value ranges to a unified value range of 0 to 1.

[0212] Furthermore, after completing all feature processing for a single basic feature vector, the standardized values ​​are rearranged according to the original field order of departure time, arrival time, and flight number.

[0213] The same processing is performed on all basic feature vectors in the common factor, thereby eliminating the dimensional differences caused by the different value ranges and encoding scales of each basic feature, and obtaining the standard common factor.

[0214] Example 8:

[0215] S7 includes obtaining flight times and route trajectories under different source identifiers; mapping flight times and route trajectories using standard common factors to obtain a first basic feature vector; reducing the dimensionality of the first basic feature vector using aircraft type coding to obtain a second basic feature vector; constructing a comparison matrix based on the second basic feature vector; performing similarity comparison on the comparison matrix to obtain vector distance and determine whether there are duplicate flights.

[0216] In this embodiment, departure time, arrival time, and flight path are extracted from flight records corresponding to different source identifiers. The departure time and arrival time use the time values ​​under the aforementioned unified time benchmark, and the flight path uses a sequence of trajectory nodes arranged in order from the departure airport to the arrival airport. Each flight record also retains the corresponding flight number, aircraft type code, and source identifier for subsequent cross-source comparison.

[0217] When using standard common factors for mapping, the departure time, arrival time and flight number that have been standardized in the standard common factors are used as the baseline features, and the route trajectory node corresponding to the current flight record is mapped to the fixed feature position.

[0218] In specific processing, each trajectory node is read according to its order in the flight route, and each node is converted into a corresponding numerical code according to a unified trajectory node encoding table. For flight records with fewer than a fixed length of trajectory nodes, zeros are added sequentially at the end. For flight records with more than a fixed length of trajectory nodes, nodes within the aforementioned fixed length range are retained according to their original node order. The fixed length is determined based on the maximum number of valid trajectory nodes in the historical flight route trajectory samples and remains unchanged in the current trajectory feature version. Subsequently, the features are arranged in a fixed order of departure time, arrival time, flight number, and trajectory node code to obtain the first basic feature vector.

[0219] When performing dimensionality reduction using aircraft type codes, each flight record is first grouped according to its aircraft type code; historical flight records corresponding to the same aircraft type code are subject to the same feature filtering rules.

[0220] Specifically, for each trajectory feature dimension in the first basic feature vector, the values ​​of historical samples within the aircraft group in that dimension are read one by one; if the values ​​of that dimension are completely consistent in all historical samples, then that dimension cannot be used to distinguish different flights and is deleted; for any two trajectory feature dimensions, if the corresponding values ​​of the two dimensions are always consistent in all historical samples, then the dimension that appears first is retained and the other dimension is deleted.

[0221] After removing fixed and duplicate dimensions, the standardized departure time, standardized arrival time, standardized flight number, and remaining trajectory features are retained and rearranged according to the original feature order to obtain the second basic feature vector.

[0222] This reduces the number of features that do not participate in flight difference identification, while retaining features that can reflect differences in flight times, flight identifiers, and flight routes.

[0223] Specifically, when constructing the comparison matrix based on the second basic feature vector, flight records are used as matrix rows, and each feature dimension in the second basic feature vector is used as matrix columns. The feature value corresponding to each flight record is written into the corresponding position.

[0224] Specifically, flight records with different source identifiers are included in the same cross-source comparison set; two records with the same source identifier are not subject to duplicate flight determination in this step; for flight records using the same aircraft type code, they are directly compared according to the same dimensionality-reduced feature position; for records with different aircraft type codes, cross-aircraft type trajectory feature distance relationship is not established to avoid dimensional misalignment caused by different trajectory feature screening results corresponding to the aircraft type.

[0225] In addition, when performing similarity comparison, two flight records with different source identifiers but the same aircraft type code are read from the comparison matrix.

[0226] Specifically, for the values ​​of two records in the first feature dimension, the feature value of one record is subtracted from the feature value of the other record, and the difference is squared. The same process is performed on all other feature dimensions. The squared differences of each feature dimension are added together, and the square root of the sum is taken as the vector distance between the two flight records. When the vector distance is 0, it means that the values ​​of the two records are completely consistent in all feature dimensions participating in the comparison. When the vector distance increases, it means that the feature differences participating in the comparison increase synchronously.

[0227] As a preferred implementation, the distance threshold used for duplicate flight determination is determined based on historical samples of flights that have already undergone cross-origin flight identity verification.

[0228] First, the vector distance of each historical sample pair is obtained according to the aforementioned distance calculation process. Sample pairs that actually belong to the same flight are marked as duplicate flights, and sample pairs that actually belong to different flights are marked as non-duplicate flights. Then, all different vector distances are arranged in ascending order, and the adjacent two different distance values ​​are added together and divided by 2 to obtain the candidate distance threshold.

[0229] Furthermore, for each candidate distance threshold, historical sample pairs with a vector distance not greater than the candidate value are judged as duplicate flights, and historical sample pairs with a vector distance greater than the candidate value are judged as non-duplicate flights. The judgment results are compared with the historical verification results one by one. The number of consistent records corresponding to each candidate distance threshold is counted, and the candidate value with the largest number of consistent records is used as the duplicate flight judgment threshold. When there are more than two candidate values ​​corresponding to the same maximum number of consistent records, the candidate value with the smallest value is used.

[0230] The determined distance threshold is written into the parameter configuration of the current comparison version and remains unchanged during the execution of this version.

[0231] In addition, during the actual flight comparison process, if the vector distance between two flight records with different source identifiers but the same aircraft type code is not greater than the duplicate flight determination threshold, the two records are determined to be duplicate flights; if the vector distance is greater than the threshold, the two records are determined to be different flights.

[0232] By comparing and processing cross-source flight records in the matrix one by one in the above manner, the vector distance between each flight record and the corresponding duplicate flight determination results are obtained.

[0233] Example 9:

[0234] S8 includes, if duplicate flights exist, obtaining the duplicate flights and their corresponding basic feature vector distances and source identifiers; matching preset priority weights and data confidence levels according to the source identifiers to obtain a source evaluation score; if the basic feature vector distance is less than a preset distance threshold, generating removal markers and retained records according to the source evaluation scores; filtering the duplicate flights through removal markers, extracting retained records to construct a target set; and using the target set to reorganize flight records under a single source identifier to obtain a unique flight set.

[0235] In this embodiment, the flight records that are determined to be duplicates are extracted based on the aforementioned duplicate flight determination results, and duplicate flight groups are divided according to the same flight number, the same execution date and the corresponding cross-source matching relationship.

[0236] Each duplicate flight group contains at least two flight records with different source identifiers. For each flight record in the group, its source identifier is read, and the basic feature vector distance between the record and other source records in the group is obtained, thus forming the data to be screened for the current duplicate flight group.

[0237] For each source identifier, read the pre-configured priority weight.

[0238] Specifically, priority weights are allocated according to the priority order of data sources determined during business access. Sources with higher priority are assigned larger weights, and the weights of all sources are converted to the range of 0 to 1. The same source maintains the same weight in the current configuration version. When the priority order of sources is adjusted, the corresponding priority weight table is regenerated, and subsequent duplicate flights are processed with the updated weight table.

[0239] In this embodiment, the data confidence level is calculated based on the historical valid records of the corresponding source.

[0240] For a given source, count the number of records with complete fields in the historical data, the number of records that do not require further correction after the aforementioned field mapping and data alignment, and the number of records that were updated within the specified update time. Divide the number of records with complete fields by the total number of historical records for that source to obtain the completeness. Divide the number of records that do not require further correction by the total number of historical records to obtain the consistency. Divide the number of records that were updated within the specified update time by the total number of historical update records to obtain the timeliness. Finally, add the completeness, consistency, and timeliness together and divide by 3. The result is taken as the data confidence of that source.

[0241] Furthermore, after obtaining the priority weight and data confidence level, the priority weight and data confidence level of the same source are multiplied together, and the result is used as the source evaluation score of that source. Each duplicate flight record obtains a unique corresponding source evaluation score according to its own source identifier, thereby enabling all records in the duplicate flight group to have a unified source evaluation basis.

[0242] The preset distance threshold is determined based on historical cross-origin flight samples that have already undergone manual verification.

[0243] First, using the same basic feature vector distance calculation method as described above, obtain the distance value of each historical sample pair, and mark the sample pairs that are confirmed to belong to the same actual flight as duplicate samples, and mark the sample pairs that are confirmed to belong to different actual flights as non-duplicate samples; arrange all different distance values ​​in ascending order, and take the average of two adjacent different distance values ​​to form candidate distance thresholds.

[0244] For each candidate distance threshold, historical sample pairs with a distance less than the candidate value are judged as duplicate samples, and historical sample pairs with a distance not less than the candidate value are judged as non-duplicate samples. Then, the judgment results are compared with the manual verification results one by one.

[0245] Count the number of correctly judged records corresponding to each candidate distance threshold, and select the candidate value with the most correctly judged records as the preset distance threshold; if there are more than two candidate values ​​corresponding to the same maximum number of correctly judged records, select the candidate value with the smallest value; the determined preset distance threshold is written into the current duplicate flight processing version and remains unchanged in this version.

[0246] In addition, for actual duplicate flight groups, when the distance between the basic feature vectors of two flight records from different sources is less than a preset distance threshold, they are identified as meeting the removal criteria, and the corresponding source evaluation scores are compared.

[0247] The flight record with the highest source evaluation score is determined as the retained record, and the remaining records are marked for deletion. The deletion mark is stored in association with the flight record identifier, source identifier, and the duplicate flight group to which it belongs, so as to ensure that the record to be deleted can be accurately located during subsequent filtering.

[0248] When there are two or more flight records with the exact same source assessment score in the same duplicate flight group, distance comparison is performed on these records.

[0249] For one record, the basic feature vector distances between that record and the other records in the group are summed sequentially to obtain the total distance within the group for that record. After performing the same calculation on all records with the same score, the record with the smallest total distance is selected as the retained record. If there are still records with the same total distance, the source record that is ranked first according to the pre-stored source priority order is selected as the retained record, and the remaining records are marked for removal.

[0250] Furthermore, after processing all duplicate flight groups, duplicate flights are filtered out based on the removal flag.

[0251] Flight records with a removal flag are removed from the current unsorted data, while flight records without a removal flag that have been identified as retained records are retained. The retained records in each duplicate flight group are aggregated and then the original records that were not identified as duplicate flights are added to form the target set. Only one cross-source record is retained for each actual flight in the target set.

[0252] When reorganizing the target set, the flight number, departure time, arrival time, price, flight duration, number of stopovers, and source identifier are rewritten in a unified field order; the retained records in the duplicate flight group retain their original source identifiers and no other source identifiers are concatenated; the flight records that have not been duplicated retain their original source identifiers and original field data.

[0253] This ensures that each reorganized flight record corresponds to a single source identifier.

[0254] After reorganizing the fields of all records in the target set, deduplication is performed according to the flight record identifier, and only one record is retained for the same flight record identifier; the verified flight records are then aggregated to form a unique flight set.

[0255] Example 10:

[0256] S9 includes extracting the departure time, arrival time, flight price, flight duration, and number of stopovers for each flight record in the unique flight set to generate a multidimensional recommendation factor matrix; performing a linear transformation on the multidimensional recommendation factor matrix to obtain a standard factor matrix; calculating the inner product of the standard factor matrix and the preset weight vector to obtain a comprehensive recommendation value; and sorting the unique flight set in descending order according to the comprehensive recommendation value to determine the final recommendation result of the international flight itinerary search results.

[0257] In this embodiment, all flight records in the unique flight set are read first, and the departure time, arrival time, flight price, flight duration and number of stopovers are extracted in a fixed field order.

[0258] Departure and arrival times are converted to cumulative minutes relative to 00:00 on the search date; flight prices are standardized according to the target currency used in the current search request. If the original price is in another currency, it is converted to the target currency price according to the exchange rate at the time the search request was generated; flight duration is uniformly converted to minutes; the number of stopovers is directly used as an integer value; each flight record forms one row, and the five recommendation factors occupy fixed column positions. After all flight records are written in sequence, a multi-dimensional recommendation factor matrix is ​​formed.

[0259] When performing a linear transformation on the multidimensional recommendation factor matrix, each of the five recommendation factors is processed separately.

[0260] Taking flight prices as an example, first read all price values ​​in the current unique flight set, determine the maximum and minimum prices, subtract the minimum price from the price of each flight record, and then divide the difference by the difference between the maximum and minimum prices to obtain the standardized result corresponding to the flight price. Flight duration, number of stops, departure time, and arrival time are processed in the same way. When the maximum and minimum values ​​of a certain recommendation factor are equal, all standardized results of that factor are uniformly set to 0 to avoid division by zero.

[0261] Since the recommended order shifts backward as the original values ​​of flight price, flight duration, and number of stopovers increase, after the above standardization is completed, the corresponding standardization result is subtracted from 1, and the resulting value is used as the final standard value of the factor. Departure time and arrival time are processed according to the time sorting direction pre-saved in the current recommendation strategy. When the departure time adopts the sorting direction of earlier is better, the standardization result is also subtracted from 1. When the sorting direction of later is better is adopted, the standardization result is directly retained. Arrival time is processed in the same way.

[0262] Therefore, the values ​​of the five recommendation factors are all between 0 and 1, with larger values ​​corresponding to higher recommendation order. All processing results are written in the original flight record order to obtain the standard factor matrix.

[0263] The preset weight vector stores five weight values ​​in the order of departure time, arrival time, flight price, flight duration, and number of stopovers. Each weight is a non-negative number, and the sum of the five weights equals 1.

[0264] In this embodiment, the weights are determined using historical flight samples that have been sorted and confirmed. Specifically, five candidate weight values ​​are generated with a step size of 0.05, and all candidate combinations whose weights sum to 1 are selected. For each candidate combination, the five standard factors of the historical flight sample are multiplied by their corresponding weights, and the five products are summed to obtain the recommended candidate value for each historical flight record. The recommended candidate values ​​are then arranged from largest to smallest. The arrangement is then compared with the confirmed target ranking in the historical samples, and the number of flight records with the same ranking position is counted. The candidate combination with the most consistent records is selected as the preset weight vector. When two candidate combinations have the same number of consistent records, the combination with the larger flight price weight is selected. If the flight price weights are still the same, the weights of flight duration, number of stops, departure time, and arrival time are compared in turn, and the candidate combination with the larger value when the values ​​first appear different is selected.

[0265] When calculating the overall recommendation value, each flight record in the standard factor matrix is ​​read one by one.

[0266] Specifically, the standard value of the departure time is first multiplied by the departure time weight, then the standard value of the arrival time is multiplied by the arrival time weight, the standard value of the flight price is multiplied by the flight price weight, the standard value of the flight duration is multiplied by the flight duration weight, and the standard value of the number of stopovers is multiplied by the number of stopovers weight. Finally, the five products are added together, and the result is used as the comprehensive recommendation value for the flight record. All flight records in the unique flight set are calculated in the same way.

[0267] Then, the flight records in the unique flight set are arranged from largest to smallest according to the comprehensive recommendation value, so that the flight with the larger comprehensive recommendation value is placed first. For flight records with the same comprehensive recommendation value, the standard value of the flight price is compared first, and the record with the larger value is placed first. If the standard values ​​of the flight price are the same, the standard values ​​of the flight duration, number of stops, departure time, and arrival time are compared in turn until the order is determined.

[0268] Furthermore, after sorting, the flight number, departure time, arrival time, flight price, flight duration, number of stopovers, and corresponding source identifier are output in the determined order to form the final recommended results of international flight itinerary search results.

[0269] When any flight record in the unique flight set is updated in terms of price, time, flight duration, or number of stopovers, the corresponding recommendation factor is regenerated, and the comprehensive recommendation value is updated according to the same linear transformation, weight calculation, and threshold judgment process. The ranking order is then re-determined based on the updated comprehensive recommendation value.

[0270] Example 11: A multi-factor recommendation and ranking system for international flight itinerary search results is also provided, for implementing the steps of the multi-factor recommendation and ranking method for international flight itinerary search results, the system comprising: The data acquisition module acquires flight data from multiple sources concurrently, and extracts the original attribute features and source identifiers of the flight data through parsing and processing to obtain a heterogeneous data set; The field mapping module uses a random forest algorithm to classify the original attribute features based on the heterogeneous data set, identifies attribute combinations under different source identifiers, and obtains field mapping relationships containing dynamic differences. The first data alignment module converts the time format and encoding standard in the heterogeneous data set through field mapping relationships, unifies the attribute combination under different source identifiers, and obtains the first data alignment result; The second data alignment module, if there are missing values ​​in the first data alignment result, uses a support vector machine to interpolate the missing values, completes the attribute combinations under different source identifiers, and determines the second data alignment result. The common factor extraction module extracts take-off and landing time and flight number features across sources based on the second data alignment results, constructs a basic feature vector for stable cross-source comparison, and obtains common factors. The common factor standardization module standardizes common factors if there is extraction bias, eliminating the dimensional differences in the basic feature vectors and obtaining standard common factors. The duplicate flight detection module compares similarity using standard common factors, calculates the distance between basic feature vectors under different source identifiers, and determines whether duplicate flights exist. The duplicate flight removal module removes duplicate flights based on the distance between the basic feature vectors, retains flight records under a single source identifier, and obtains a unique set of flights. The recommendation and ranking module extracts the departure time, arrival time, price, flight duration, and number of stopovers of each flight record from a unique flight set as recommendation factors. It standardizes each recommendation factor and calculates a comprehensive recommendation value for each flight record based on the preset weights corresponding to each recommendation factor. The flight records in the unique flight set are then ranked in real time based on the comprehensive recommendation value to determine the final recommendation result of the international flight itinerary search results.

[0271] In this embodiment, the data acquisition module receives flight data returned concurrently from multiple flight data sources, performs message parsing on the data from each source, extracts field names, field types, time formats, encoding forms, and field values, and writes the corresponding source identifiers into the flight records to form a heterogeneous data set.

[0272] The field mapping module reads heterogeneous data sets, inputs the original attribute features into the random forest model for classification, determines the corresponding unified target field based on the attribute combinations of fields from different sources, and updates the field mapping content based on the structural changes of each source field to obtain the field mapping relationship containing information on source differences and field changes.

[0273] The first data alignment module locates fields with the same meaning from different sources based on field mapping relationships, converts departure and arrival times to a unified time base, converts flight numbers, airport codes, and airline codes to a unified encoding format, and reorganizes flight records according to the unified target field order to obtain the first data alignment result.

[0274] The second data alignment module checks the integrity of fields in the first data alignment result.

[0275] When missing values ​​exist, the valid attributes of the record containing the missing field are extracted to form input features, and then input into the corresponding support vector machine model to obtain the predicted value. After the predicted value reaches the preset confidence condition, the predicted value is written into the position of the missing field, thus obtaining the second data alignment result.

[0276] The common factor extraction module extracts the departure time, arrival time and flight number of flight records from different sources from the second data alignment results, and performs cross-source feature processing in combination with the corresponding route trajectory. After filtering out bias features that affect stable comparison, it constructs a basic feature vector according to a fixed field order to form common factors.

[0277] The common factor standardization module calculates the value range of each basic feature in the common factor and performs normalization processing on basic features with extraction bias, so that the takeoff time, arrival time and flight number are converted to a uniform numerical scale to obtain the standard common factor.

[0278] The duplicate flight identification module compares the similarity of flight records from different sources based on standard common factors, calculates the difference between corresponding basic features one by one, and forms a basic feature vector distance based on each difference. This distance is then compared with a preset distance threshold to determine whether records from different sources correspond to the same flight.

[0279] When duplicate flights are detected, the duplicate flight removal module reads the source identifier, basic feature vector distance, source priority weight, and data confidence level corresponding to the duplicate record, determines the source evaluation score of each source record, retains one flight record whose source evaluation score meets the retention condition, and filters the remaining duplicate records to form a unique set of flights.

[0280] The recommendation ranking module extracts departure time, arrival time, price, flight duration, and number of stopovers from a unique set of flights. It standardizes each recommendation factor and unifies the recommendation direction, multiplies each factor by its corresponding preset weight, and then adds the products to obtain a comprehensive recommendation value for each flight record. The flight records are then sorted in descending order based on the comprehensive recommendation value. After the flight data is updated, the recommendation factor processing and comprehensive recommendation value calculation are re-executed to obtain the final recommendation result for international flight itinerary search results.

[0281] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-factor recommendation ranking method for international flight itinerary search results, characterized in that, include: S1. Obtain concurrent flight data from multiple sources, extract the original attribute features and source identifiers of the flight data, and obtain a heterogeneous data set; S2. Based on the heterogeneous data set, classify the original attribute features to obtain field mapping relationships that include dynamic differences; S3. Convert the time format and encoding standard in the heterogeneous data set through field mapping relationship, unify the attribute combination under different source identifiers, and obtain the first data alignment result; S4. If there are missing values ​​in the first data alignment result, interpolation calculation is performed on the missing values ​​to complete the attribute combinations under different source identifiers and determine the second data alignment result. S5. Based on the second data alignment results, extract the take-off and landing times and flight numbers from different sources, construct a basic feature vector, and obtain common factors; S6. If there is an extraction bias in the common factors, the common factors are standardized to eliminate the dimensional differences in the basic feature vectors and obtain standard common factors. S7. Compare similarity using standard common factors, calculate the distance between basic feature vectors under different source identifiers, and determine whether duplicate flights exist; S8. If duplicate flights exist, they are eliminated based on the distance between the basic feature vectors to obtain a unique set of flights. S9. Based on the unique flight set, sort the flight records in the unique flight set in real time to determine the final recommended results of the international flight itinerary search results.

2. The method for multi-factor recommendation ranking of international flight itinerary search results according to claim 1, characterized in that: S1 includes: Acquire multi-source concurrent flight data streams and separate the underlying data packets in the flight data streams through message decomposition and processing; Extract message structure features from the underlying data messages to determine field mapping rules; By using field mapping rules, feature extraction is performed on the underlying data packets to obtain the original attribute feature set; If the original attribute feature set is missing, then the original attribute feature set is completed to obtain the complete attribute feature set; A corresponding source identifier is assigned based on the complete set of attribute features. By concatenating the complete set of attribute features and the source identifier, a heterogeneous data set is obtained.

3. The method for multi-factor recommendation ranking of international flight itinerary search results according to claim 1, characterized in that: S2 includes: The original attribute features and source identifiers of the heterogeneous data set are extracted to construct a feature sample matrix, and the feature sample matrix is ​​processed by the random forest algorithm to extract the feature splitting threshold; The original attribute features are divided by feature splitting threshold to obtain attribute combinations, and the frequency of attribute combinations is statistically analyzed to generate a frequency distribution matrix. If the frequency distribution matrix value is greater than the frequency threshold, then the association rules are extracted and the dynamic differences in the association rules are identified. Update the initial mapping table based on the dynamic differences to obtain the field mapping relationship containing the dynamic differences.

4. The multi-factor recommendation and ranking method for international flight itinerary search results according to claim 1, characterized in that: S3 includes: Obtain field mapping relationships and heterogeneous data sets, parse the heterogeneous data sets through field mapping relationships, and convert them to obtain the target time format and target encoding standard; Reconstruct heterogeneous datasets based on target time format and target coding standard, and extract reconstructed attribute combinations under different source identifiers; An alignment matrix is ​​constructed by combining reconstructed attributes. The first data alignment result is obtained by uniformly reconstructing the attribute combinations through the alignment matrix.

5. The multi-factor recommendation and ranking method for international flight itinerary search results according to claim 1, characterized in that: S4 includes: Obtain the missing values ​​and corresponding attribute combinations in the first data alignment result, and construct the initial feature vector; Based on the initial feature vectors, a hyperplane is constructed using a support vector machine to determine the predicted values ​​for missing values. If the predicted value meets the preset confidence threshold, the predicted value is used to interpolate the missing value to obtain the completed record. Based on the completion records, complete the attribute combinations under different source identifiers to determine the second data alignment result.

6. The multi-factor recommendation and ranking method for international flight itinerary search results according to claim 1, characterized in that: S5 includes: Extract flight identifiers and departure and arrival times based on the second data alignment results; Cross-source features are obtained by matching flight identifiers and departure and arrival times with flight trajectories. The fundamental vector is obtained by orthogonal transformation of cross-source features; If the variance of the fundamental vectors is greater than the threshold, then the fundamental vectors are extracted to construct the deviation matrix; Decompose the singular values ​​of the deviation matrix to determine the stable comparison set; Extract take-off and landing times and flight numbers from the stable comparison set to construct the basic feature vector for cross-source stable comparisons and obtain common factors.

7. The multi-factor recommendation and ranking method for international flight itinerary search results according to claim 1, characterized in that: S6 includes: Obtain the common factors, calculate the mean vector and covariance matrix, and obtain the Mahalanobis distance; If the Mahalanobis distance is greater than the preset threshold, it is determined that there is an extraction bias in the common factors and the basic feature vector is extracted. Dimensional differences are obtained by calculating extreme values ​​for the basic eigenvectors. Construct a normalized matrix based on the differences in dimensions; The common factors are standardized by using a normalized matrix to eliminate the dimensional differences in the basic feature vectors and obtain the standard common factors.

8. The multi-factor recommendation and ranking method for international flight itinerary search results according to claim 1, characterized in that: S7 includes: Obtain flight schedules and route trajectories from different source identifiers; The first basic feature vector is obtained by mapping flight times and flight routes using standard common factors. The first basic feature vector is reduced in dimensionality by using the machine model code to obtain the second basic feature vector; Construct a comparison matrix based on the second basic feature vector; By comparing the similarity of the comparison matrix to obtain the vector distance, it can be determined whether there are duplicate flights.

9. The multi-factor recommendation and ranking method for international flight itinerary search results according to claim 1, characterized in that: S8 includes: If duplicate flights exist, obtain the duplicate flights and their corresponding basic feature vector distances and source identifiers; The source evaluation score is obtained by matching the source identifier with the preset priority weight and data confidence level. If the distance between the basic feature vectors is less than a preset distance threshold, then a removal label and a retention record are generated based on the source evaluation score. By filtering out duplicate flights by removing markers, a target set is constructed by extracting and retaining the records. The flight records under a single source identifier are reorganized using the target set to obtain a unique set of flights.

10. The multi-factor recommendation and ranking method for international flight itinerary search results according to claim 1, characterized in that: S9 includes: Extract the departure time, arrival time, flight price, flight duration, and number of stopovers for each flight record from the unique flight set to generate a multidimensional recommendation factor matrix; A linear transformation is performed on the multidimensional recommendation factor matrix to obtain the standard factor matrix; Calculate the inner product of the standard factor matrix and the preset weight vector to obtain the comprehensive recommendation value; The unique set of flights is sorted in descending order based on the overall recommendation value to determine the final recommended results for international flight itinerary search results.

11. A multi-factor recommendation and ranking system for international flight itinerary search results, used to implement the steps of the multi-factor recommendation and ranking method for international flight itinerary search results as described in any one of claims 1-10, characterized in that, The system includes: The data acquisition module acquires flight data from multiple sources concurrently, and extracts the original attribute features and source identifiers of the flight data through parsing and processing to obtain a heterogeneous data set; The field mapping module uses a random forest algorithm to classify the original attribute features based on the heterogeneous data set, identifies attribute combinations under different source identifiers, and obtains field mapping relationships containing dynamic differences. The first data alignment module converts the time format and encoding standard in the heterogeneous data set through field mapping relationships, unifies the attribute combination under different source identifiers, and obtains the first data alignment result; The second data alignment module, if there are missing values ​​in the first data alignment result, uses a support vector machine to interpolate the missing values, completes the attribute combinations under different source identifiers, and determines the second data alignment result. The common factor extraction module extracts take-off and landing time and flight number features across sources based on the second data alignment results, constructs a basic feature vector for stable cross-source comparison, and obtains common factors. The common factor standardization module standardizes common factors if there is extraction bias, eliminating the dimensional differences in the basic feature vectors and obtaining standard common factors. The duplicate flight detection module compares similarity using standard common factors, calculates the distance between basic feature vectors under different source identifiers, and determines whether duplicate flights exist. The duplicate flight removal module removes duplicate flights based on the distance between the basic feature vectors, retains flight records under a single source identifier, and obtains a unique set of flights. The recommendation and ranking module extracts the departure time, arrival time, price, flight duration, and number of stopovers of each flight record from a unique flight set as recommendation factors. It standardizes each recommendation factor and calculates a comprehensive recommendation value for each flight record based on the preset weights corresponding to each recommendation factor. The flight records in the unique flight set are then ranked in real time based on the comprehensive recommendation value to determine the final recommendation result of the international flight itinerary search results.

Images