Training data set processing method and freight rate file analysis method

Abstract

The invention discloses a training data set processing method and a freight rate file analysis method. Relates to the field of data processing, and the method comprises the steps: obtaining air transportation price bill data, obtaining a freight rate data interface document used by an aviation institution, and extracting basic transportation price bill data from the freight rate data interface document; the air transportation price bill data is disassembled to obtain M groups of freight rate data, the M groups of freight rate data are labeled to obtain labeled freight rate data, and the labeled freight rate data at least comprises a rule type, a general rule type and an annotation type; a data set used for model training is generated according to the marked freight rate data and the basic transportation price bill data, and the model data set comprises a document set and a data set associated with the document. Through the method and the device, the technical problems of high cost, low efficiency and limited accuracy in manual marking of the training data set in the related technology are solved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method for processing a training data set and a method for parsing freight rate files. Background Art

[0002] With the current development of artificial intelligence technology, large-scale language models are profoundly changing the operational models of various industries and playing an irreplaceable role in promoting the intelligent upgrade of products and services. By learning from massive amounts of text data, these models not only grasp the complex structure and expression of language, but also perform a range of tasks such as text generation, translation, and complex question-and-answering. To achieve these goals, large and high-quality datasets are required for model training. In the process of building a price list entry model for aviation agencies, the construction of the dataset is a key factor in determining the model's performance and practical value.

[0003] Airline fare data isn't just simple fare information; it's composed of a multi-layered, multi-dimensional structure encompassing numerous different fare categories, such as base fares, surcharges, special discounts, and restrictions. This vast array of data categories, complex dimensions, and nested rules create a highly sophisticated information system. For example, rules such as "Maximum Stay," "Eligibility," and "Flight Applications" not only have numerous entries, but each rule can include dozens of attributes and details, significantly increasing the difficulty of understanding and processing the data.

[0004] During the construction of training datasets, the timeliness of price list data makes it difficult to obtain externally or use on a large scale. This directly limits the size and diversity of the dataset, impacting the effectiveness of model training and its adaptability to practical applications. Furthermore, while manual labeling can ensure data quality, it is costly and inefficient due to the complexity of freight rate data, the complexity of ATPCO (Airline Tariff Publishing Company) standards, and the scarcity of historical price list data, making it unable to meet the demand for large-scale, high-quality datasets.

[0005] Currently, no effective solution has been proposed to the technical problems of high cost, low efficiency and limited accuracy in manually annotating training data sets in related technologies. Summary of the Invention

[0006] The main purpose of this application is to provide a method for processing a training data set and a method for parsing freight rate files to solve the technical problems of high cost, low efficiency and limited accuracy in manually annotating training data sets in related technologies.

[0007] To achieve the above objectives, according to one aspect of the present application, a method for processing a training dataset is provided. The method comprises: obtaining air transport tariff data and a tariff data interface document used by an aviation agency, extracting basic transport tariff data from the tariff data interface document; decomposing the air transport tariff data to obtain M groups of tariff data, and annotating the M groups of tariff data to obtain annotated tariff data, wherein the annotated tariff data includes at least a rule type, a general rule type, and an annotation type, and M is a positive integer; generating a dataset for model training based on the annotated tariff data and the basic transport tariff data, wherein the model dataset includes a document set and a document-associated data set, and the document-associated data set includes the M groups of annotated tariff data and the basic transport tariff data.

[0008] Furthermore, before obtaining the air transport price list data, the method also includes: obtaining an air transport price list file from a fare management system associated with the aviation agency, and sending the air transport price list file to the client, wherein, when the client receives the air transport price list file, it extracts initial air transport price list data from the air transport price list file, and stores the initial air transport price list data in the fare management system according to the entry rules; obtaining the air transport price list data includes: executing a second script code, and the second script code extracts candidate air transport price list data from the fare management system based on query conditions, wherein the query conditions include at least a rule number, an effective time, and a fee rule; screening the candidate air transport price list data according to a data structure category to obtain the air transport price list data, wherein the data structure category includes at least one of the following: field name, field description, field type, field example.

[0009] Furthermore, extracting basic transport price list data from the freight data interface document includes: executing a first script code, collecting basic field information from the freight data interface document based on the first script code to obtain M groups of field codes, extracting field information from each group of field codes to obtain M groups of field information, wherein each group of field codes is associated with a data structure category, and each data structure category includes at least one of the following: field name, field description, field type, and field example; structuring the M groups of field information to obtain a structured document, and generating basic transport price list data based on the structured document.

[0010] Furthermore, generating basic transport price list data based on the structured document includes: when reading the structured document into a preset memory, obtaining the proofreading rules of the preset memory; performing mutual translation and proofreading on the fields of the structured document based on the proofreading rules to obtain initial basic transport price list data, and sending the initial basic transport price list data to the client, wherein, when the client receives the initial basic transport price list data, it reviews the initial basic transport price list data to obtain a review result; receiving the review result sent by the client, and when the review result indicates that the review is passed, determining the initial basic transport price list data as the basic transport price list data.

[0011] Furthermore, the air transport price list data is decomposed to obtain M groups of freight rate data, including: grouping the air transport price list data to obtain M groups of price list group data, wherein the data type of each group of price list group data includes at least: a rule type, a general rule type, and an annotation type; for a group of price list group data, decomposing the group of price list group data according to the freight rate standard to obtain Y groups of freight rate sub-data, wherein Y is a positive integer; respectively recombining the Y groups of freight rate sub-data associated with each group of price list group data to obtain M groups of initial freight rate data, and converting the format of each group of initial freight rate data to obtain M groups of freight rate data.

[0012] Furthermore, labeling the M groups of freight rate data to obtain the labeled freight rate data includes: obtaining data types of the M groups of freight rate data, and when the data types of the M groups of freight rate data include rule types, general rule types, and annotation types, extracting freight rate data associated with a first data labeling requirement from the M groups of freight rate data to obtain first freight rate data, and forming first coded data from the first freight rate data; when the data types of the M groups of freight rate data include rule types and general rule types, extracting freight rate data associated with a second data labeling requirement from the M groups of freight rate data to obtain second freight rate data, and forming second coded data from the second freight rate data; when the data types of the M groups of freight rate data include annotation types, extracting freight rate data associated with a third data labeling requirement from the M groups of freight rate data to obtain third freight rate data, and forming third coded data from the third freight rate data; and combining the first coded data, the second coded data, and the third coded data to obtain the labeled freight rate data.

[0013] Furthermore, generating the model data set according to the marked freight rate data and the basic transport rate data includes: assembling the marked freight rate data to obtain assembled freight rate data; and associating and combining the assembled freight rate data with the basic transport rate data to obtain the model data set.

[0014] To achieve the above objectives, according to another aspect of the present application, a method for parsing freight rate files is provided. The method comprises: obtaining a freight rate file to be parsed, inputting the freight rate file to be parsed into a price list entry model, and processing the data to obtain N freight rate data. The price list entry model is obtained by training a preset model using a model dataset, the model dataset comprising a document set and a document-associated data set, the document-associated data set being obtained using any of the aforementioned training dataset processing methods, and N is a positive integer.

[0015] To achieve the above-mentioned objectives, according to another aspect of the present application, a training dataset processing device is provided. The device includes: a first acquisition unit, configured to acquire air transport tariff data and a tariff data interface document used by an aviation agency, and extract basic transport tariff data from the tariff data interface document; a disassembly unit, configured to disassemble the air transport tariff data to obtain M groups of tariff data, and annotate the M groups of tariff data to obtain annotated tariff data, wherein the annotated tariff data includes at least a rule type, a general rule type, and an annotation type, and M is a positive integer; and a generation unit, configured to generate a dataset for model training based on the annotated tariff data and the basic transport tariff data, wherein the model dataset includes a document set and a document-associated dataset, and the document-associated dataset includes the M groups of annotated tariff data and the basic transport tariff data.

[0016] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned methods for processing training data sets.

[0017] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory stores an executable program, and the processor is used to run the program, wherein when the one or more programs are executed by one or more processors, the one or more processors implement any of the above-mentioned methods for processing a training data set.

[0018] According to another aspect of an embodiment of the present invention, a computer program product is provided. The computer program product includes a computer program, wherein when the computer program is executed by a processor, any one of the above-mentioned methods for processing a training data set is implemented.

[0019] In an embodiment of the present application, a training set is used to process the air transport price list data and the price data interface document used by the aviation agency, and extract the basic transport price list data from the price data interface document; the air transport price list data is disassembled to obtain M groups of price data, and the M groups of price data are annotated to obtain annotated price data, wherein the annotated price data at least includes: rule type, general rule type and annotation type, and M is a positive integer; a data set for model training is generated based on the annotated price data and the basic transport price list data, wherein the model data set includes A document collection and a document-associated data collection, wherein the document-associated data collection includes M groups of annotated freight rate data and basic transport price list data, which solves the technical problems of high cost, low efficiency and limited accuracy of manually annotating training data sets in related technologies. By obtaining air transport price list data and extracting basic transport price list data, the air transport price list data is disassembled to obtain freight rate data, the freight rate data is annotated to obtain annotated freight rate data, and the annotated freight rate data and basic transport price list data are used to generate a model data set, thereby achieving the technical effect of improving the quality and construction efficiency of the training data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0021] Figure 1 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for processing a training data set;

[0022] Figure 2 is a flowchart of a method for processing a training data set provided in an embodiment of the present application;

[0023] Figure 3 This is a flowchart of a method for parsing a freight rate file according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of a system for processing a training data set according to an embodiment of the present application;

[0025] Figure 5 This is an example of an optional training data set processing method provided in accordance with an embodiment of the present application. Figure 1 ;

[0026] Figure 6 This is an example of an optional training data set processing method provided in accordance with an embodiment of the present application. Figure 2 ;

[0027] Figure 7is a schematic diagram of a device for processing a training data set according to an embodiment of the present application;

[0028] Figure 8 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0032] It should be noted that the collected information used in this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse use.

[0033] Example 1

[0034] According to an embodiment of the present application, an embodiment of a method for processing a training set is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 It is a hardware structure diagram of a computer terminal (or mobile device) for implementing a method for processing a training data set, such as Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more ( Figure 1 The computer system 102 includes a processor 102 (the processor 102 may include but is not limited to a microcontroller unit (MCU) or a programmable logic device (FPGA)), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, the computer system 102 may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS), a network interface, a keyboard, a cursor control device, a power supply, and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0036] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0037] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for processing a training data set in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned method for processing a training data set. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0038] The transmission device 106 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC) and a network interface, which can be connected to other network devices via a base station to enable communication with the Internet. In one embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0039] The display may be, for example, a touch screen liquid crystal display (LCD), which enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0040] Under the above operating environment, this application provides Figure 2 The processing method of the training dataset shown. Figure 2 is a flowchart of a method for processing a training data set according to an embodiment of the present application, such as Figure 2 As shown, the method includes the following steps:

[0041] Step S201: Acquire air transport price list data, and obtain the price data interface document used by the aviation agency, and extract basic transport price list data from the price data interface document.

[0042] It should be noted that price list data, which can be a fare document or price list, is a collection of documents published by aviation agencies containing all fare information. It is usually presented in an unstructured form and contains a large amount of text descriptions and rules. These rules describe information such as fares for different routes, different dates, and different passenger types. It is a key document for exchanging price information between aviation agencies and passengers, travel service agencies, and online booking platforms. This data is released by aviation agencies through specific systems and channels.

[0043] Tariff data is structured information extracted from fare data. Tariff data refers to detailed information about air fares in the air transport industry, typically set by airlines based on factors such as routes, cabin class, time, and passenger type. Tariff data includes rules, restrictions, fee details, and possible footnotes or additional terms. Specifically, fare data can be broken down into the following sections: Rules: These refer to the specific conditions that apply to fares, such as maximum stay, minimum stay, advance purchase period, and cancellation or change fees. General Rules: These rules apply to multiple specific rules and are typically global descriptions of fares, such as the validity period of the fare, applicable routes, or geographic areas. Footnotes: These provide additional explanations or interpretations, typically supplementing a rule or fare and may involve specific terms or exceptions. Fees and Amounts: These are numerical values ​​directly related to the price, including fares, taxes, surcharges, and more.

[0044] From the above, we can see that price list data is a raw, unstructured file containing detailed text descriptions of various freight rates; while freight rate data is standardized information that can be understood and processed by computer systems after analyzing, parsing, and structuring price list data. The airline agency's price list entry model can effectively convert price list data into freight rate data to facilitate further automated processing, analysis, and freight rate entry and management. It is a key technology for airline agencies to improve the efficiency and automation level of freight rate management. Therefore, price list data serves as the raw input for the airline agency's price list entry model. In order to convert this unstructured price list data into structured freight rate data for computer system understanding and processing, the price list entry model needs to be trained. In order to train this model, it is first necessary to obtain a model dataset.

[0045] Specifically, to obtain the model dataset, we first need to obtain the fare data interface documentation used by airlines, namely the ATPCO (Airline Tariff Publishing Company) interface documentation. Since these documents contain information such as data structure, field meanings, and data transmission format, we can use the information provided in the interface documentation to authorize access to the airline's API (Application Programming Interface) interface documentation. We can then use programming languages ​​to write scripts or development tools to download the relevant data, thereby obtaining the basic freight rate data. Furthermore, to construct the model dataset, we also need to obtain the air freight rate data stored in the freight rate management system.

[0046] Step S202 : Decompose the air transport rate data to obtain M groups of rate data, and mark the M groups of rate data to obtain marked rate data, wherein the marked rate data at least includes: rule type, general rule type and annotation type, and M is a positive integer.

[0047] Specifically, since air transport price list data contains a large amount of information, which is organized according to specific logic and structure, forming a series of complex rules and terms, in order for the model to understand and learn these rules, the original price list data needs to be broken down into smaller, easier-to-process units, namely multiple sets of freight rate data.

[0048] For example, when breaking down rule types, since each rule in a price list has its specific purpose and application scenario, "Maximum Stay" belongs to a specific rule type that involves restrictions on passenger stay time. By initially classifying the price list data, it can be broken down into different rule types such as "Maximum Stay", "Eligibility", and "Flight Applications". When breaking down general rule types, for example, general rules such as those regarding payment methods and refund and change policies appear multiple times in different locations in the price list. By breaking down the general rule types, these recurring rules can be separated into independent annotation units, which helps improve the efficiency and consistency of data annotation. When breaking down annotation types, the annotation section in the price list, such as "Footnotes", is usually a further explanation or supplementary explanation of the specific rules, which can include specific conditions, exceptions, or additional guidelines. Annotation type breaking can separate these annotations from the main rules and annotate and analyze them separately to ensure that the model can fully understand all the details of the rules.

[0049] Furthermore, after obtaining multiple sets of disassembled freight rate data, these data can also be annotated. For example, each set of data in the price list is labeled to indicate which rule type it corresponds to. For example, "R3" can represent a specific category of freight rate rules. For general rules that span multiple specific rules, their type and scope of application need to be specifically annotated. The annotations in the price list are also carefully annotated, including the type of annotation, the associated rules, the content of the annotation, and its impact on the rules. The annotated freight rate data can be saved in a structured format, such as CSV (Comma-Separated Values) or database records, to facilitate call and processing during model training. In the output annotated data, each set of freight rate information contains the above-mentioned annotated information, such as rule type, general rule type, annotation type, and other key attributes.

[0050] It should be noted that when labeling freight rate data, this can be accomplished using customized labeling tools. Such tools can automatically or semi-automatically label data according to pre-defined rules and standards. For example, a freight rate labeling tool (dataset-pipeline) built using programming languages ​​and relevant data processing libraries (such as Pandas) can extract freight rate data from freight rate management systems, associate it with the original price list text, and then generate structured training samples. The core components of such labeling tools may include multiple analyzers (such as FootNoteCat03Analyzer and RuleCat04Analyzer), which can parse and label price list data for different categories.

[0051] Step S203: Generate a data set for model training based on the labeled freight rate data and basic transport rate data, wherein the model data set includes a document set and a document-associated data set, and the document-associated data set includes M sets of labeled freight rate data and basic transport rate data.

[0052] Specifically, after labeling and structuring the freight rate data, it can be integrated into a dataset suitable for model training. First, all labeled freight rate documents are categorized and organized according to factors such as freight rate type, airline, and effective date to ensure that each document can be accurately identified and classified. Document preprocessing makes it easier for the model to understand and process the document content. Since freight rate documents can be in multiple languages, preprocessing also needs to consider language translation and unification to ensure that the model can understand documents in various languages, improving the model's multilingual processing capabilities and ultimately obtaining a document collection.

[0053] The document-associated data set refers to the structured data set associated with each document in the document set, which may include annotated freight rate data and basic freight rate sheet data, that is, integrating the annotated data to ensure that each set of data accurately corresponds to the corresponding original document.

[0054] Furthermore, after obtaining the model data set, the data set can be divided according to a preset ratio (such as 70% training set, 15% validation set, and 15% test set). The training set is used for model training, the validation set is used to adjust the model parameters, and the test set is used to evaluate the final performance of the model to ensure that the model not only performs well on the training data, but can also accurately parse freight rate information on new data that has not been seen.

[0055] The training dataset processing method provided in the embodiment of the present application obtains air transport tariff data and a tariff data interface document used by an aviation agency, extracts basic transport tariff data from the tariff data interface document, decomposes the air transport tariff data to obtain M groups of tariff data, and labels the M groups of tariff data to obtain labeled tariff data, wherein the labeled tariff data at least includes: a rule type, a general rule type, and an annotation type, and M is a positive integer; generates a dataset for model training based on the labeled tariff data and the basic transport tariff data, wherein the model dataset includes a document set and a document-associated data set, and the document-associated data set includes the M groups of labeled tariff data and the basic transport tariff data. This method solves the technical problems of high cost, low efficiency, and limited accuracy of manually labeling training datasets in the related art. By obtaining air transport tariff data and extracting basic transport tariff data, decomposing the air transport tariff data to obtain tariff data, labeling the tariff data to obtain labeled tariff data, and generating a model dataset using the labeled tariff data and the basic transport tariff data, the quality of the training dataset and the construction efficiency are improved, thereby achieving the technical effect of improving the construction efficiency.

[0056] Optionally, in the method for processing a training data set provided in an embodiment of the present application, before obtaining air transport price data, the method further includes: obtaining an air transport price file from a fare management system associated with an aviation agency, and sending the air transport price file to a client, wherein, when the client receives the air transport price file, it extracts initial air transport price data from the air transport price file, and stores the initial air transport price data in the fare management system according to an entry rule; obtaining the air transport price data includes: executing a second script code, and the second script code extracts candidate air transport price data from the fare management system based on a query condition, wherein the query condition includes at least a rule number, an effective time, and a fee rule; screening the candidate air transport price data according to a data structure category to obtain the air transport price data, wherein the data structure category includes at least one of the following: field name, field description, field type, and field example.

[0057] Specifically, before constructing the model dataset for the aviation agency, it is first necessary to obtain and process the data in the fare management system. First, obtain the air transport tariff file from the fare management system related to the aviation agency. At this time, a data access channel can be established with the aviation agency to obtain the data through an API interface, a data subscription service, or direct database access. After obtaining the tariff file, it can be sent to the client. After receiving the tariff file, the client extracts the initial air transport tariff data from the file, identifies different fare rules and terms, such as "maximum stay time limit", "passenger qualification restrictions", "flight restrictions", etc., and converts this information into a data format that the system can recognize and store it in the fare management system. During this process, the tariff information can be correctly entered into the system through entry rules to ensure data accuracy and consistency. The entry rules can include data field identification, data type conversion, data format standardization, etc., to ensure that the entered data is compatible with subsequent data processing and model training processes.

[0058] Furthermore, after the initial data entry, candidate air transport tariff data is extracted from the fare management system by executing a second script. This second script, designed as a data extraction script, utilizes the fare management system's query functionality to filter out eligible fare data based on preset query criteria. The query criteria include at least key information such as the rule number, effective date, and fee rules, thereby facilitating the selection of data samples relevant to the training model from the massive dataset. For example, when the model needs to be trained to identify a specific "maximum stay time limit" rule, the second script will accurately locate and extract all relevant fare data from the fare management system as candidate data based on the rule number, effective date range, and fee rules.

[0059] It should be noted that the extracted candidate air transport price list data needs to be screened before further processing to ensure that the data structure meets the requirements of the training model. The data structure categories in the screening process include at least field name, field description, field type, field example, etc. These attributes define the structure and content of the data. The screening operation can be performed by a dedicated data processing script. The script will check the field information of each data entry to ensure that they match the ATPCO standard or other industry specifications. For example, check whether the field "Maximum Stay" contains the correct effective time, stay time limit and fee information, and whether the field format is correct. Only data that passes the screening will be finally adopted as members of the data set for model training. This process effectively removes invalid data and data that does not match the format, thereby improving the quality of the data set.

[0060] By processing data, this embodiment not only ensures the diversity and richness of data from the source, but also significantly improves the efficiency and accuracy of data processing through automated and semi-automated data extraction and screening processes. It can not only effectively support the model in learning the complex rules of air freight rates, but also provide stable and reliable data support in the training, verification and testing stages, laying a solid foundation for the successful training and application of the price list entry model for airline agencies.

[0061] Optionally, in the processing method of the training data set provided in the embodiment of the present application, extracting basic transport price list data from the freight data interface document includes: executing a first script code, collecting basic field information from the freight data interface document based on the first script code, obtaining M groups of field codes, extracting field information from each group of field codes, and obtaining M groups of field information, wherein each group of field codes is associated with a data structure category, and each data structure category includes at least one of the following: field name, field description, field type, and field example; structuring the M groups of field information to obtain a structured document, and generating basic transport price list data based on the structured document.

[0062] Specifically, since the basic data obtained from the air fare data interface document is relatively raw and requires a series of processing before it can become suitable input for machine learning model training, a first script code can be used to collect basic field information from the air fare data interface document. That is, the script sends a request to the interface document address to obtain an HTML (HyperText Markup Language) page or API response, and then parses the obtained HTML content to locate the part containing the field information. Then, from the parsed DOM (Document Object Model) tree or JSON (JavaScript Object Notation) response, the relevant field codes are obtained. Each set of field codes represents a specific data structure category, covering various freight data elements such as "rules" and "footnotes."

[0063] Furthermore, for each set of field codes, the above script code is used to further extract its associated field information to obtain multiple sets of field information. Due to the unstructured nature of the original documents, directly using this information for model training is inefficient and difficult to ensure accuracy. Therefore, structured processing is required, that is, cleaning the collected information, removing irrelevant data or parts with incorrect formats, and ensuring the quality of the information; using data processing libraries such as pandas, the different field information is integrated into a unified data framework, creating a row for each field code and each field information as a column; converting the integrated data into a structured document format to facilitate further processing and model training, thereby obtaining a structured document.

[0064] Since structured documents contain more than just detailed field information, you can standardize field names and descriptions based on the structured documents to ensure consistent formatting and naming conventions for all information. Then, use the information in the structured documents to generate basic freight rate data.

[0065] This embodiment greatly improves the speed and efficiency of data preparation and reduces labor costs by executing the first script code and performing structured processing on the collected freight rate data interface information. Structured processing ensures the integrity and accuracy of the data. Standardization and multilingual translation further improve the applicability and reliability of the data set, providing a solid data foundation for model training.

[0066] Optionally, in the processing method of the training data set provided in the embodiment of the present application, generating basic transport price list data based on the structured document includes: when the structured document is read into the preset memory, obtaining the proofreading rules of the preset memory; performing mutual translation and proofreading on the fields of the structured document based on the proofreading rules to obtain initial basic transport price list data, and sending the initial basic transport price list data to the client, wherein, when the client receives the initial basic transport price list data, it reviews the initial basic transport price list data to obtain a review result; receiving the review result sent by the client, and when the review result indicates that the review is passed, determining the initial basic transport price list data as the basic transport price list data.

[0067] Specifically, in order to significantly improve the basic quality of data and ensure the efficiency and accuracy of subsequent processing, customized proofreading rules can be used to translate and proofread the fields in the document. First, the structured document is read into the preset memory space, and the proofreading rules (such as field name, field description, field type, field format, field example, etc.) are stored in an easily accessible memory rule library to ensure the efficiency and accuracy of the proofreading process.

[0068] Furthermore, based on the proofreading rules stored in the memory, the fields in the structured document are proofread between Chinese and English. First, each field in the document can be identified to ensure that all fields that need to be proofread are correctly identified and located. For each field, the corresponding proofreading rules are retrieved from the memory rule library. According to the proofreading rules, the fields are proofread between Chinese and English to ensure that the field names and descriptions in the document remain consistent in the two languages, avoiding inaccurate information or misunderstandings due to language differences. Any anomalies found during the proofreading process, such as inaccurate field translation, incorrect field format, etc., are recorded and processed in a timely manner. After the proofreading is completed, the proofread document, that is, the initial basic freight rate data, is sent to the client for manual review.

[0069] After receiving the initial basic freight rate data, the client can conduct a detailed review of the document, checking for field translation accuracy, data format compliance, and logical consistency to ensure high data quality and credibility. For example, a business expert can check the document field by field against proofreading rules to confirm that all information is correct. After the review is complete, the client sends the review results back to the server. If the review results indicate approval, the initial basic freight rate data can be confirmed as the basic freight rate data.

[0070] This embodiment generates basic transport price list data by reading structured documents in memory and applying proofreading rules for mutual translation and proofreading, thereby enhancing the adaptability of the dataset to various business scenarios. By utilizing the high speed of memory reading and the immediacy of the rule base, the time for data proofreading is greatly shortened, the need for manual participation is reduced, and labor costs are saved.

[0071] Optionally, in the training data set processing method provided in the embodiment of the present application, the air transport price list data is disassembled to obtain M groups of freight rate data, including: grouping the air transport price list data to obtain M groups of price list group data, wherein the data type of each group of price list group data includes at least: rule type, general rule type and annotation type; for a group of price list group data, disassemble the group of price list group data according to the freight rate standard to obtain Y groups of freight rate sub-data, wherein Y is a positive integer; respectively recombine the Y groups of freight rate sub-data associated with each group of price list group data to obtain M groups of initial freight rate data, and convert the format of each group of initial freight rate data to obtain M groups of freight rate data.

[0072] Specifically, to improve dataset quality and enhance model learning, the price list data can be effectively grouped and decomposed, and reorganized into a format suitable for model training. First, different categories of information, such as specific rules and general rules, can be identified, and the identified information can be grouped so that each group of data contains the same type of freight rate rules. Each group of price list grouped data can be further decomposed according to freight rate standards to obtain a series of smaller and more specific freight rate sub-data. In other words, based on freight rate standards, such as the fields and attributes specified by ATPCO, each group of price list grouped data is refined into smaller freight rate sub-data. The decomposed freight rate sub-data will then be reorganized based on their correlation to form multiple groups of initial freight rate data. It should be noted that during the reorganization, each group of reorganized initial freight rate data needs to be formatted to conform to the specific format required for model training, such as converting the data into JSON or other structured data formats to facilitate model understanding and processing.

[0073] This embodiment disassembles air freight rate data to obtain multiple sets of freight rate data. Each set of freight rate data contains rich sub-data, which helps the model learn the details and levels of freight rate rules, thereby more accurately identifying and parsing complex freight rate information in practical applications. This greatly improves the efficiency of data processing, ensures data consistency and accuracy, and saves a lot of time and resources for model training.

[0074] Optionally, in the method for processing a training data set provided in an embodiment of the present application, M groups of freight rate data are labeled to obtain the labeled freight rate data, including: obtaining the data type of the M groups of freight rate data, and when the data type of the M groups of freight rate data includes a rule type, a general rule type, and an annotation type, extracting freight rate data associated with a first data labeling requirement from the M groups of freight rate data to obtain first freight rate data, and forming first coded data from the first freight rate data; when the data type of the M groups of freight rate data includes a rule type and a general rule type, extracting freight rate data associated with a second data labeling requirement from the M groups of freight rate data to obtain second freight rate data, and forming second coded data from the second freight rate data; when the data type of the M groups of freight rate data includes an annotation type, extracting freight rate data associated with a third data labeling requirement from the M groups of freight rate data to obtain third freight rate data, and forming third coded data from the third freight rate data; and combining the first coded data, the second coded data, and the third coded data to obtain the labeled freight rate data.

[0075] Specifically, when labeling freight rate data, the freight rate labeling tool (such as dataset-pipeline) can be used to adopt Category parsing pipeline parallelism and adopt the strategy pattern architecture design parsing script extension. When performing labeling processing, the freight rate data can first be subdivided and encoded according to different types. For example, when the data type includes rule type, general rule type and annotation type (that is, the extracted freight rate data is classified according to rule, general rule, and footnote), the information directly related to these rule types can be extracted from the freight rate data, and the associated data of R2 and TableString can be found (that is, the freight rate data associated with the first data labeling requirement is extracted) to form an Ascii encoding sample file of R2. The obtained first freight rate data is encoded to form the first encoded data.

[0076] When the data type of the freight rate data includes rule type and general rule type (i.e., the extracted freight rate data is categorized by rule and general rule), the R3 data can be found (i.e., the freight rate data associated with the second data annotation requirement is extracted), forming a JSON format sample file of R3 RULE. The obtained second freight rate data is also encoded and converted to obtain second encoded data. When the data type of the freight rate data includes comment type (i.e., the extracted freight rate data is categorized by footnote), the corresponding R3 data can be found (i.e., the freight rate data associated with the third data annotation requirement is extracted), and then a JSON format sample file of R3 FOOTNOTE is formed. The obtained third freight rate data is encoded to generate third encoded data. Finally, the first coded data, the second coded data, and the third coded data are combined to form the annotated freight rate data. It should be noted that this process needs to take into account the correlation and logic between the data to ensure that each sample data can reflect the overall structure and rule system of the freight rate file. For example, the integration process of the coded data needs to ensure that each rule or annotation can be accurately associated with its corresponding general rule and annotation type; the integrated data also needs to undergo optimization processing, including data deduplication, detection and correction of outliers, and logical consistency checks between data to ensure the quality of the data set.

[0077] This embodiment classifies, extracts, encodes, and combines different types of freight rate data, making the encoded data more suitable for the input format of the machine learning model. This can significantly accelerate the learning speed of the model, while improving the accuracy of the model in parsing complex freight rate rules, increasing the generalization ability of the model, and improving the efficiency of data preparation.

[0078] Optionally, in the training data set processing method provided in the embodiment of the present application, generating a model data set based on the labeled freight rate data and the basic transport price list data includes: assembling the labeled freight rate data to obtain assembled freight rate data; and associating and combining the assembled freight rate data and the basic transport price list data to obtain a model data set.

[0079] Specifically, after obtaining the annotated freight rate data and basic freight rate data, in order to generate the final model dataset, since the annotated freight rate data is distributed in different files and data blocks, each data block can correspond to different aspects of the freight rate, such as "rule number", "effective time", "fee rule", etc. In order to construct a complete data sample for model learning, these scattered data blocks need to be assembled and processed to ensure that each sample dataset contains all necessary information related to a specific freight rate rule. In other words, by matching the annotated freight rate data with the freight rate data samples generated by the freight rate extraction module, all information under the same freight rate rule is integrated into a data sample. For example, the "R3 RULE" and "R3 Footnote" data are matched and integrated with the corresponding "R2" data to form a complete sample containing the rule, general rule, and footnote information.

[0080] The assembled rate data is then checked for integrity to ensure that all necessary fields are included. While the assembled rate data contains rich rule information, to enable the model to gain a deeper understanding of the rate context and application scenarios, this data needs to be associated with the underlying freight rate data. This means that the assembled rate data is mapped to the underlying freight rate data, ensuring that each rate rule is associated with a specific airline, route, or region. Finally, this associated data is combined into the model dataset.

[0081] This embodiment ensures that each sample data in the model dataset contains comprehensive freight rate information and relevant basic transportation information through the assembly processing of freight rate data and its association with basic transportation rate data. This enables the model to demonstrate stronger generalization capabilities when processing freight rates from different aviation agencies and different regions, significantly improves the efficiency and quality of model training, shortens training time, and reduces the risk of overfitting.

[0082] Under the above operating environment, this application provides Figure 3 The parsing method of the freight rate file shown. Figure 3 is a flow chart of a method for parsing a freight rate file according to an embodiment of the present application. Figure 3 As shown, the method includes the following steps:

[0083] Step S301: Obtain a freight rate file to be parsed, input the freight rate file to be parsed into a price list entry model, and process it to obtain N freight rate data. The price list entry model is obtained by training a preset model with a model dataset. The model dataset includes a document set and a document-associated data set. The document-associated data set is obtained by processing any of the above training datasets. N is a positive integer.

[0084] Specifically, the fare file to be parsed can be obtained first. This file can come from the airline's internal database, a third-party data provider such as ATPCO, or other fare publication channels. The fare file to be parsed contains unstructured fare information, such as rules, footnotes, and general rules. The collected fare file to be parsed is then input into the price list entry model previously trained with the model dataset. After receiving the fare file to be parsed, the model identifies specific rules in the file, extracts relevant detailed information, and converts the extracted information into a structured data format. Based on the learned knowledge, it can then identify, parse, and extract the information in the file, ultimately outputting multiple structured fare data.

[0085] The freight rate file parsing method provided in the embodiment of the present application obtains the freight rate file to be parsed, inputs the freight rate file to be parsed into a price list entry model, and processes to obtain N freight rate data, wherein the price list entry model is obtained by training a preset model through a model data set, and the model data set includes a document set and a document-associated data set, and the document-associated data set is obtained by any of the above-mentioned training data set processing methods, and N is a positive integer. The method solves the technical problems of high cost, low efficiency and limited accuracy of manually annotating training data sets in the related art, obtains air transport price list data and extracts basic transport price list data, disassembles the air transport price list data to obtain freight data, annotates the freight data to obtain annotated freight data, and generates a model data set using the annotated freight data and basic transport price list data, thereby achieving the technical effect of improving the parsing efficiency, accuracy and reliability of freight rate files of aviation agencies.

[0086] The present application also provides a system for processing a training data set. Figure 4 is a schematic diagram of a system for processing a training data set according to an embodiment of the present application, such as Figure 4As shown in Figure 1, the system includes: an ATPCO data annotation module, a freight rate entry module, a freight rate extraction module, a sample generation module, and a data partitioning module. The ATPCO data annotation module is used to extract the Footnotes and Rules sections and basic field information covering 17 categories from the interface document, including field name, field description, field type, field format, field example, etc.; in addition, this module can also collect basic freight rate information; the freight rate entry module can accurately enter freight rates into the freight rate management system after manually understanding the original price list file; the freight rate extraction module is used to obtain structured data generated by manually entered freight rate information from the freight rate management system; the sample generation module is used to automatically annotate samples based on the extracted freight rate information; the data partitioning module is used to scientifically and rationally divide the freight rate sample data into training, validation, and test sets according to preset standards and proportions. In addition, this module can flexibly adjust the partitioning strategy based on data characteristics and business needs, such as chronological partitioning, random partitioning, or stratified sampling, to meet the specific requirements of different scenarios, thereby providing more accurate and reliable data support for model training and evaluation.

[0087] Figure 5 This is an example of an optional training data set processing method provided in accordance with an embodiment of the present application. Figure 1 , Figure 6 This is an example of an optional training data set processing method provided in accordance with an embodiment of the present application. Figure 2 ,like Figure 5 、 Figure 6 As shown, the method is applied to the processing system of the above training data set, and the method includes:

[0088] First, the original price lists of multiple scenarios collected from the aviation agencies are manually processed and entered into the fare management system through the fare entry module. That is, business personnel manually understand the original price list files in depth and accurately enter the fares into the fare management system. For example, business personnel understand and identify fare rules from the airline's historical price list files, such as: Maximum Stay (maximum stay time limit), Eligibility (passenger qualification limit), Flight Applications (flight restrictions), etc.; then enter the specific rules into the corresponding Category in the fare management system. For example, Maximum Stay (maximum stay time limit) belongs to Category 07 and includes return regulations, return start time, ticket identification, geographic location and other information.

[0089] Furthermore, the decomposed original price list data is extracted from the freight management system through the freight extraction module, that is, the freight extraction tool is used to filter through conditions such as airline, rule number, and effective time to find the freight data in the freight management system corresponding to the original price list; then it is classified according to rule, general rule, and footnote, and the data is decomposed into formats such as INTALLFARE, R0, R1, R2C according to the freight standard, and then structured data is generated. Among them, the freight extraction tool can be built based on Python+APScheduler. By connecting to the freight management system, the pipeline parallelism is adopted to accurately extract the manually entered original price data in a timed increment.

[0090] Furthermore, the sample generation module parses and converts the price list data into standard ATPCO interface request data, and associates it with the original price list text to form training instruction pair data. This module also connects with the ATPCO document to obtain the definitions and field meanings of Footnotes and Rules. Based on the existing dataset and ATPCO document, a labeling template paradigm is developed to ensure the accuracy of the generated samples. Then, a freight rate labeling tool (dataset-pipeline) is built to extract freight rate data from the freight rate management system and generate structured data as model training output samples. The data is then automatically associated with the original price list text as the model training input sample.

[0091] The original price list data is then converted into corresponding freight rate data (rule, footnote) through the freight rate extraction module. The corresponding analyzer is used to perform labeling for different categories, and the labeled data is then assembled to generate training samples.

[0092] When annotating ATPCO documents, the ATPCO data annotation module can be used to incrementally collect ATPCO interface documents through scripts, obtain interface field names, field descriptions, field types, field formats, field examples, and other content, and structure them into documents. Then, using the pandas data processing tool library, the generated documents are read into memory, and business terminology proofreading and Chinese-English translation are performed on the fields based on rules. After manual review, a high-quality basic price list dataset is formed.

[0093] Finally, after all the sample data are generated, the two data sets are reasonably combined through the data partitioning module, and the combined data set is divided into training set, validation set and test set, and stored in a high-performance file server. At this time, the data partitioning module can formulate three partitioning strategies according to the actual scenario requirements, namely random partitioning, chronological partitioning and hierarchical partitioning.

[0094] This embodiment ensures the adequacy and accuracy of model training by constructing a data set for training the model. It greatly improves the efficiency of data preparation through automated labeling tools and data partitioning strategies, saves the cost and time of manual processing, and achieves comprehensive coverage, efficient processing and secure storage of freight rate data, providing a solid foundation for the training and application of price list entry models for airline agencies.

[0095] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0096] Example 2

[0097] The present application also provides a training data set processing device. It should be noted that the training data set processing device of the present application embodiment can be used to execute the training data set processing method provided in the present application embodiment. The following describes the training data set processing device provided in the present application embodiment.

[0098] According to an embodiment of the present application, a device for implementing the above-mentioned method for processing a training data set is also provided. Figure 7 is a schematic diagram of a processing device for a training data set provided in an embodiment of the present application, such as Figure 7 As shown, the device includes: a first acquisition unit 70, a disassembly unit 71, and a generation unit 72.

[0099] The first acquisition unit 70 is used to acquire air transport price list data and obtain the price data interface document used by the aviation agency, and extract the basic transport price list data from the price data interface document;

[0100] The disassembling unit 71 is configured to disassemble the air transport price list data to obtain M groups of freight rate data, and mark the M groups of freight rate data to obtain marked freight rate data, wherein the marked freight rate data at least includes: a rule type, a general rule type, and a comment type, and M is a positive integer;

[0101] The generating unit 72 is used to generate a data set for model training based on the labeled freight rate data and the basic transport price list data, wherein the model data set includes a document set and a document-associated data set, and the document-associated data set includes M groups of labeled freight rate data and basic transport price list data.

[0102] The training data set processing device provided in the embodiment of the present application obtains air transport price list data through a first acquisition unit 70, obtains the price list data interface document used by the aviation agency, and extracts the basic transport price list data from the price list data interface document; the disassembly unit 71 disassembles the air transport price list data to obtain M groups of price list data, and annotates the M groups of price list data to obtain annotated price list data, wherein the annotated price list data at least includes: rule type, general rule type and annotation type, and M is a positive integer; the generation unit 72 generates a data set for model training based on the annotated price list data and the basic transport price list data, wherein In the method, the model data set includes a document set and a data set associated with the document. The data set associated with the document includes M groups of labeled freight rate data and basic transport price list data, which solves the technical problems of high cost, low efficiency and limited accuracy of manually labeling training data sets in related technologies. By obtaining air transport price list data and extracting basic transport price list data, the air transport price list data is disassembled to obtain freight data, the freight data is labeled to obtain labeled freight data, and the labeled freight data and basic transport price list data are used to generate a model data set, thereby achieving the technical effect of improving the quality and construction efficiency of the training data set.

[0103] Optionally, in the processing device of the training data set provided in the embodiment of the present application, the device also includes: a second acquisition unit, which is used to obtain an air transport price list file from the freight rate management system associated with the aviation agency before obtaining the air transport price list data, and send the air transport price list file to the client, wherein, when the client receives the air transport price list file, it extracts initial air transport price list data from the air transport price list file, and stores the initial air transport price list data in the freight rate management system according to the entry rules; the first acquisition unit 70 includes: a first execution module, which is used to execute the second script code, and the second script code extracts candidate air transport price list data from the freight rate management system based on the query condition, wherein the query condition includes at least the rule number, the effective time and the fee rule; a screening module, which is used to screen the candidate air transport price list data according to the data structure category to obtain the air transport price list data, wherein the data structure category includes at least one of the following: field name, field description, field type, field example.

[0104] Optionally, in the processing device of the training data set provided in the embodiment of the present application, the first acquisition unit 70 includes: a second execution module, used to execute the first script code, collect basic field information from the freight data interface document based on the first script code, obtain M groups of field codes, extract field information from each group of field codes, and obtain M groups of field information, wherein each group of field codes is associated with a data structure category, and each data structure category includes at least one of the following: field name, field description, field type, field example; a first processing module, used to structure the M groups of field information to obtain a structured document, and generate basic transport price list data based on the structured document.

[0105] Optionally, in the processing device of the training data set provided in the embodiment of the present application, the first acquisition unit 70 includes: a first acquisition module, used to obtain the proofreading rules of the preset memory when the structured document is read into the preset memory; a proofreading module, used to perform mutual translation and proofreading on the fields of the structured document based on the proofreading rules to obtain initial basic transport price list data, and send the initial basic transport price list data to the client, wherein, when the client receives the initial basic transport price list data, it reviews the initial basic transport price list data to obtain an audit result; a receiving module, used to receive the audit result sent by the client, and when the audit result indicates that the audit is passed, determine the initial basic transport price list data as the basic transport price list data.

[0106] Optionally, in the processing device of the training data set provided in the embodiment of the present application, the disassembly unit 71 includes: a second processing module, used to group the air transport price list data to obtain M groups of price list group data, wherein the data type of each group of price list group data includes at least: rule type, general rule type and annotation type; a disassembly module, used to disassemble a group of price list group data according to the freight rate standard to obtain Y groups of freight rate sub-data, wherein Y is a positive integer; a first combination module, used to recombine the Y groups of freight rate sub-data associated with each group of price list group data to obtain M groups of initial freight rate data, and convert the format of each group of initial freight rate data to obtain M groups of freight rate data.

[0107] Optionally, in the processing device of the training data set provided in the embodiment of the present application, the disassembling unit 71 includes: a second acquisition module, used to obtain the data type of M groups of freight rate data, and when the data type of the M groups of freight rate data includes a rule type, a general rule type, and an annotation type, extract the freight rate data associated with the first data annotation requirement from the M groups of freight rate data to obtain the first freight rate data, and form the first coded data from the first freight rate data; a first extraction module, used to extract the freight rate data associated with the second data annotation requirement from the M groups of freight rate data to obtain the second freight rate data, and form the second coded data from the second freight rate data, when the data type of the M groups of freight rate data includes the rule type and the general rule type; a second extraction module, used to extract the freight rate data associated with the third data annotation requirement from the M groups of freight rate data to obtain the third freight rate data, and form the third coded data from the third freight rate data, when the data type of the M groups of freight rate data includes the annotation type; a second combination module, used to combine the first coded data, the second coded data, and the third coded data to obtain the labeled freight rate data.

[0108] Optionally, in the processing device of the training data set provided in the embodiment of the present application, the generation unit 72 includes: a third processing module, used to assemble the labeled freight rate data to obtain assembled freight rate data; a third combination module, used to associate and combine the assembled freight rate data with the basic transport price list data to obtain a model data set.

[0109] It should be noted that the first acquisition unit 70, disassembly unit 71, and generation unit 72 described above correspond to steps S201 to S203 in Example 1. The examples and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above units can also be part of a device and can be run in the computer terminal 10 provided in Example 1.

[0110] Example 3

[0111] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal or an electronic device.

[0112] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0113] In this embodiment, the computer terminal can execute the program code of the following steps in the training data set processing method: obtaining air transport price list data and obtaining a freight rate data interface document used by the aviation agency, and extracting basic transport price list data from the freight rate data interface document; disassembling the air transport price list data to obtain M groups of freight rate data, and labeling the M groups of freight rate data to obtain labeled freight rate data, wherein the labeled freight rate data at least includes: a rule type, a general rule type, and an annotation type, and M is a positive integer; generating a data set for model training based on the labeled freight rate data and the basic transport price list data, wherein the model data set includes a document set and a document-associated data set, and the document-associated data set includes the M groups of labeled freight rate data and the basic transport price list data.

[0114] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the method for processing the training data set: before obtaining the air transport price list data, the method also includes: obtaining the air transport price list file from the freight rate management system associated with the aviation agency, and sending the air transport price list file to the client, wherein, when the client receives the air transport price list file, it extracts initial air transport price list data from the air transport price list file, and stores the initial air transport price list data in the freight rate management system according to the entry rules; obtaining the air transport price list data includes: executing the second script code, and the second script code extracts candidate air transport price list data from the freight rate management system based on the query conditions, wherein the query conditions include at least the rule number, the effective time and the fee rule; screening the candidate air transport price list data according to the data structure category to obtain the air transport price list data, wherein the data structure category includes at least one of the following: field name, field description, field type, field example.

[0115] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the method for processing the training data set: extracting basic transport price list data from the freight data interface document includes: executing the first script code, collecting basic field information from the freight data interface document based on the first script code, obtaining M groups of field codes, extracting field information from each group of field codes, and obtaining M groups of field information, wherein each group of field codes is associated with a data structure category, and each data structure category includes at least one of the following: field name, field description, field type, and field example; structuring the M groups of field information to obtain a structured document, and generating basic transport price list data based on the structured document.

[0116] Optionally, the computer terminal may execute the program code of the following steps in the method for processing a training data set: generating basic transport price list data based on a structured document includes: obtaining proofreading rules from a preset memory when the structured document is read into a preset memory; performing mutual translation and proofreading on the fields of the structured document based on the proofreading rules to obtain initial basic transport price list data, and sending the initial basic transport price list data to a client, wherein, when the client receives the initial basic transport price list data, the client reviews the initial basic transport price list data to obtain a review result; receiving the review result sent by the client, and determining the initial basic transport price list data as basic transport price list data when the review result indicates that the review is passed.

[0117] Optionally, the computer terminal may execute the program code of the following steps in the method for processing a training data set: disassembling the air transport price list data to obtain M groups of freight rate data, including: performing grouping processing on the air transport price list data to obtain M groups of price list group data, wherein the data type of each group of price list group data includes at least: rule type, general rule type and annotation type; for a group of price list group data, disassembling the group of price list group data according to the freight rate standard to obtain Y groups of freight rate sub-data, wherein Y is a positive integer; respectively recombining the Y groups of freight rate sub-data associated with each group of price list group data to obtain M groups of initial freight rate data, and converting the format of each group of initial freight rate data to obtain M groups of freight rate data.

[0118] Optionally, the computer terminal may execute the program code of the following steps in the method for processing a training data set: labeling M groups of freight rate data to obtain the labeled freight rate data, including: obtaining the data type of the M groups of freight rate data, and when the data type of the M groups of freight rate data includes a rule type, a general rule type, and an annotation type, extracting freight rate data associated with a first data labeling requirement from the M groups of freight rate data to obtain first freight rate data, and forming first coded data from the first freight rate data; when the data type of the M groups of freight rate data includes a rule type and a general rule type, extracting freight rate data associated with a second data labeling requirement from the M groups of freight rate data to obtain second freight rate data, and forming second coded data from the second freight rate data; when the data type of the M groups of freight rate data includes an annotation type, extracting freight rate data associated with a third data labeling requirement from the M groups of freight rate data to obtain third freight rate data, and forming third coded data from the third freight rate data; and combining the first coded data, the second coded data, and the third coded data to obtain the labeled freight rate data.

[0119] Optionally, the above-mentioned computer terminal can execute the program code of the following steps in the training data set processing method: generating a model data set based on the labeled freight rate data and the basic transport price list data includes: assembling the labeled freight rate data to obtain assembled freight rate data; associating and combining the assembled freight rate data and the basic transport price list data to obtain the model data set.

[0120] Optionally, Figure 8 This is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 8 As shown, the electronic device may include: one or more ( Figure 8 Only one is shown) processor 802, memory 804, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0121] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for processing the training data set in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method for processing the training data set. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0122] The processor may call the information and application programs stored in the memory through the transmission device to execute the above steps in the above method for processing the training data set.

[0123] According to an embodiment of the present application, a solution for processing a training set is provided. By obtaining air transport tariff data and a tariff data interface document used by an aviation agency, basic transport tariff data is extracted from the tariff data interface document; the air transport tariff data is decomposed to obtain M groups of tariff data, and the M groups of tariff data are labeled to obtain labeled tariff data, wherein the labeled tariff data at least includes: a rule type, a general rule type, and an annotation type, and M is a positive integer; a dataset for model training is generated based on the labeled tariff data and the basic transport tariff data, wherein the model dataset includes a document set and a document-associated data set, and the document-associated data set includes the M groups of labeled tariff data and the basic transport tariff data, thereby solving the technical problems of high cost, low efficiency, and limited accuracy of manually labeling training datasets in related technologies. By obtaining air transport tariff data and extracting basic transport tariff data, decomposing the air transport tariff data to obtain tariff data, labeling the tariff data to obtain labeled tariff data, and using the labeled tariff data and the basic transport tariff data to generate a model dataset, the technical effect of improving the quality and construction efficiency of the training dataset is achieved.

[0124] It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only, and the electronic device may also be a terminal device such as a smart phone, a tablet computer, a PDA, a mobile Internet device (MID), or a PAD. Figure 8 It does not limit the structure of the above electronic device. For example, the electronic device may also include Figure 8 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 8 Different configurations shown.

[0125] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0126] Example 4

[0127] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the method for processing the training data set provided in the first embodiment.

[0128] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0129] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining air transport price list data, and obtaining a freight rate data interface document used by the aviation agency, and extracting basic transport price list data from the freight rate data interface document; disassembling the air transport price list data to obtain M groups of freight rate data, and labeling the M groups of freight rate data to obtain labeled freight rate data, wherein the labeled freight rate data at least includes: a rule type, a general rule type, and an annotation type, and M is a positive integer; generating a data set for model training based on the labeled freight rate data and the basic transport price list data, wherein the model data set includes a document set and a document-associated data set, and the document-associated data set includes M groups of labeled freight rate data and basic transport price list data.

[0130] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the method for processing a training data set.

[0131] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0132] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0134] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0135] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0136] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0137] The above is only a preferred embodiment of the present application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present application, and such improvements and modifications should also be considered within the scope of protection of the present application.

Claims

1. A method for processing a training data set, characterized in that: include: Obtain air transport price list data and obtain a freight rate data interface document used by the aviation agency, and extract basic transport price list data from the freight rate data interface document; Decomposing the air transport rate data to obtain M groups of rate data, and labeling the M groups of rate data to obtain labeled rate data, wherein the labeled rate data at least includes: a rule type, a general rule type, and an annotation type, and M is a positive integer; A data set for model training is generated based on the labeled freight rate data and the basic transport rate data, wherein the model data set includes a document set and a document-associated data set, and the document-associated data set includes the M groups of labeled freight rate data and the basic transport rate data.

2. The method according to claim 1, characterized in that Before obtaining the air transport price list data, the method further includes: Obtaining an air transport price list file from a fare management system associated with the airline organization, and sending the air transport price list file to the client, wherein upon receiving the air transport price list file, the client extracts initial air transport price list data from the air transport price list file, and stores the initial air transport price list data in the fare management system according to an input rule; Obtaining air transport price list data includes: executing a second script code, wherein the second script code extracts candidate air transport rate data from the rate management system based on a query condition, wherein the query condition includes at least a rule number, an effective time, and a fee rule; The candidate air transport price list data are screened according to data structure categories to obtain the air transport price list data, wherein the data structure categories include at least one of the following: field name, field description, field type, and field example.

3. The method according to claim 1, characterized in that Extracting basic transport rate data from the freight rate data interface document includes: Executing a first script code, based on the first script code, collecting basic field information from the freight rate data interface document to obtain M groups of field codes, extracting field information from each group of field codes to obtain M groups of field information, wherein each group of field codes is associated with a data structure category, and each data structure category includes at least one of the following: field name, field description, field type, and field example; The M groups of field information are structured to obtain a structured document, and the basic transport price list data is generated according to the structured document.

4. The method according to claim 3, characterized in that Generating the basic transport price list data according to the structured document includes: When the structured document is read into a preset memory, obtaining proofreading rules of the preset memory; Performing mutual translation and proofreading on the fields of the structured document based on the proofreading rules to obtain initial basic transport price data, and sending the initial basic transport price data to the client, wherein upon receiving the initial basic transport price data, the client reviews the initial basic transport price data to obtain a review result; The audit result sent by the client is received, and when the audit result indicates that the audit is passed, the initial basic transport price list data is determined as the basic transport price list data.

5. The method according to claim 1, wherein The air transport price list data is disassembled to obtain M groups of freight rate data including: performing grouping processing on the air transport price list data to obtain M groups of price list group data, wherein the data type of each group of price list group data includes at least: the rule type, the general rule type, and the annotation type; For a set of price list group data, the set of price list group data is disassembled according to the freight rate standard to obtain Y sets of freight rate sub-data, where Y is a positive integer; The Y groups of freight rate sub-data associated with each group of price list group data are respectively recombined to obtain M groups of initial freight rate data, and each group of initial freight rate data is format-converted to obtain the M groups of freight rate data.

6. The method according to claim 1, wherein The M groups of freight rate data are labeled, and the labeled freight rate data obtained include: Obtaining data types of the M sets of freight rate data, and if the data types of the M sets of freight rate data include the rule type, the general rule type, and the annotation type, extracting freight rate data associated with the first data annotation requirement from the M sets of freight rate data to obtain first freight rate data, and forming first coded data from the first freight rate data; When the data types of the M sets of freight rate data include the rule type and the general rule type, extracting the freight rate data associated with the second data annotation requirement from the M sets of freight rate data to obtain second freight rate data, and forming second coded data from the second freight rate data; When the data type of the M sets of freight rate data includes the annotation type, extracting the freight rate data associated with the third data annotation requirement from the M sets of freight rate data to obtain third freight rate data, and forming third coded data from the third freight rate data; The first coded data, the second coded data and the third coded data are combined to obtain the marked freight rate data.

7. The method according to claim 1, characterized in that Generating a model data set according to the labeled freight rate data and the basic transport rate sheet data includes: Assembling the marked freight rate data to obtain assembled freight rate data; The assembled freight rate data and the basic transport rate sheet data are associated and combined to obtain the model data set.

8. A method for parsing a freight rate file, characterized in that: include: Obtain a freight rate file to be parsed, input the freight rate file to be parsed into a price list entry model, and process it to obtain N freight rate data, wherein the price list entry model is obtained by training a preset model through a model data set, the model data set includes a document set and a document-associated data set, and the document-associated data set is obtained by processing the training data set according to any one of claims 1 to 7, and N is a positive integer.

9. A device for processing a training data set, characterized in that: include: A first acquisition unit is configured to acquire air transport price list data and a freight rate data interface document used by an aviation agency, and extract basic transport price list data from the freight rate data interface document; a disassembling unit configured to disassemble the air transport rate data to obtain M groups of rate data, and mark the M groups of rate data to obtain marked rate data, wherein the marked rate data at least includes: a rule type, a general rule type, and an annotation type, and M is a positive integer; A generating unit is used to generate a data set for model training based on the labeled freight rate data and the basic transport rate data, wherein the model data set includes a document set and a document-associated data set, and the document-associated data set includes the M groups of labeled freight rate data and the basic transport rate data.

10. An electronic device, characterized in that: include: a memory storing an executable program; A processor, configured to run the program, wherein the program, when running, executes the method for processing a training data set according to any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the method for processing a training data set according to any one of claims 1 to 7 are implemented.

Images