Field processing knowledge base construction method and field processing auxiliary method and device

Abstract

The invention discloses a construction method of a field processing knowledge base and an auxiliary method and device of field processing, which are applied to the technical field of data processing. In the construction process of the field processing knowledge base, field basic information and a field blood relationship are firstly obtained; on the basis of the field basic information and the field blood relationship, the field processing mode can be fully mined from the field basic information of the fields and the association relationship between the fields, and the comprehensive and high-quality field processing mode is obtained. And writing the field processing knowledge describing the field processing mode into a field processing knowledge base. And in an auxiliary stage of field processing, according to the field processing knowledge base and the obtained query problem, querying the field processing mode meeting the field processing requirement, and generating a query result including information of the field processing mode. And displaying a query result for auxiliary field processing. On the basis of the field processing knowledge base, the field processing mode meeting the field processing requirement can be inquired and obtained, and the field processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a method for constructing a field processing knowledge base, an auxiliary method for field processing, and an apparatus therefor. Background Art

[0002] In the process of data processing, various operations need to be performed on fields to meet specific data processing requirements. The process of operating on fields can be referred to as field processing. As an example, field processing includes various operations on fields such as field cleaning, field conversion, field splitting, field merging, and field filtering. The specific manner of operating on fields can be referred to as a field processing mode.

[0003] When technicians perform field processing, there are many optional field processing modes. It is difficult for technicians to quickly select a field processing mode that suits the field processing requirements from numerous field processing modes. How to determine a field processing mode that more conforms to the field processing requirements has become a technical problem to be solved. Summary of the Invention

[0004] In view of this, this application provides a method for constructing a field processing knowledge base, an auxiliary method for field processing, and an apparatus therefor, which can generate field processing knowledge describing field processing modes based on the information of the fields themselves and the association relationships between fields, construct a field processing knowledge base, and the field processing modes obtained by querying based on the field processing knowledge base are more in line with the user's field processing requirements and have a higher accuracy.

[0005] The technical solutions provided by this application are as follows:

[0006] In a first aspect, this application provides a method for constructing a field processing knowledge base, the method including:

[0007] Obtain field basic information and field lineage, where the field basic information is used to describe the definition and attributes of the fields, and the field lineage is used to describe the association relationships between fields;

[0008] Extract field processing knowledge based on the field basic information and the field lineage, where the field processing knowledge is used to describe field processing modes;

[0009] Write the field processing knowledge into a field processing knowledge base, where the field processing knowledge base is used to assist field processing.

[0010] In a second aspect, this application provides an auxiliary method for field processing, the method including:

[0011] Obtain a query problem for field processing, where the query problem is used to describe field processing requirements;

[0012] Generate a query result corresponding to the field processing requirement according to the field processing knowledge included in the field processing knowledge base and the query problem. The field processing knowledge base is generated according to the field processing knowledge base construction method described in the first aspect. The field processing knowledge is used to describe the field processing mode. The query result includes information about the field processing mode, and the query result is used to assist field processing;

[0013] Display the query result.

[0014] In a third aspect, the present application provides a field processing knowledge base construction device, which includes:

[0015] A first acquisition module, configured to acquire field basic information and field blood relationship. The field basic information is used to describe the definition and attributes of the field, and the field blood relationship is used to describe the association relationship between fields;

[0016] An extraction module, configured to extract field processing knowledge based on the field basic information and the field blood relationship. The field processing knowledge is used to describe the field processing mode;

[0017] A construction module, configured to write the field processing knowledge into a field processing knowledge base, and the field processing knowledge base is used to assist field processing.

[0018] In a fourth aspect, the present application provides a field processing assistance device, which includes:

[0019] An acquisition module, configured to acquire a query problem for field processing, and the query problem is used to describe the field processing requirement;

[0020] A query module, configured to generate a query result corresponding to the field processing requirement according to the field processing knowledge included in the field processing knowledge base and the query problem. The field processing knowledge base is generated according to the field processing knowledge base construction method described in the first aspect. The field processing knowledge is used to describe the field processing mode. The query result includes information about the field processing mode, and the query result is used to assist field processing;

[0021] A display module, configured to display the query result.

[0022] In a fifth aspect, the present application provides an electronic device, including:

[0023] One or more processors;

[0024] A storage device, on which one or more programs are stored,

[0025] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect or the method described in the second aspect.

[0026] In a sixth aspect, the present application provides a computer-readable medium, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect or the method described in the second aspect is implemented.

[0027] In a seventh aspect, the present application provides a computer program product. When the computer program product runs on a device, the device executes the method described in the first aspect or implements the method described in the second aspect.

[0028] Thus, the present application has the following beneficial effects:

[0029] The present application provides a method for constructing a field processing knowledge base. Field basic information for describing the definition and attributes of fields and field lineage relationships for describing the association relationships between fields are obtained. Based on the field basic information and field lineage relationships, field processing patterns can be fully mined from the field basic information of the fields themselves and the association relationships between the fields, and relatively comprehensive field processing patterns can be obtained. The field processing knowledge describing the field processing patterns is written into the field processing knowledge base to realize the precipitation of knowledge related to the field processing patterns.

[0030] In addition, the present application also provides an auxiliary method for field processing. First, a query problem for field processing input by a user is obtained. The query problem is used to describe the user's field processing requirements. According to the field processing knowledge included in the field processing knowledge base and the query problem, a field processing pattern that meets the field processing requirements is queried, and a query result including information on the field processing pattern is generated. On the one hand, the field processing knowledge base includes field processing knowledge describing field processing patterns. Based on the field processing knowledge base, a field processing pattern that more conforms to the field processing requirements can be queried. On the other hand, based on the field processing knowledge base, a field processing pattern can be automatically queried, reducing the cost of manually selecting and learning field processing patterns. Finally, the query result is displayed. The user can select a field processing pattern according to the information on the field processing pattern included in the query result to implement field processing. Assisting field processing based on the field processing knowledge base can improve the efficiency of field processing and reduce the cost of the user's operations of selecting a field processing pattern and performing field processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0032] Figure 2Schematic flowchart of the method for constructing a field processing knowledge base and the auxiliary method for field processing provided by the embodiments of the present application;

[0033] Figure 3 Flowchart of a method for constructing a field processing knowledge base provided by the embodiments of the present application;

[0034] Figure 4 Flowchart of an auxiliary method for field processing provided by the embodiments of the present application;

[0035] Figure 5 Schematic structural diagram of a device for constructing field processing knowledge provided by the embodiments of the present application;

[0036] Figure 6 Schematic structural diagram of an auxiliary device for field processing provided by the embodiments of the present application;

[0037] Figure 7 Schematic structural diagram of an electronic device provided by the embodiments of the present application. Detailed implementation manners

[0038] To facilitate the understanding and interpretation of the technical solutions provided by the embodiments of the present application, the background technology of the present application will be described first below.

[0039] In the process of data processing, field processing is a relatively important link. Field processing can optimize data, and the processed data is more in line with the needs of using the data. The specific manner of field processing can be referred to as the field processing mode. When technicians process fields, they can determine the field processing mode used based on personal experience or by other auxiliary methods for determining the field processing mode.

[0040] Technicians, such as data engineers or business personnel, can select the field processing mode based on personal experience, manually write scripts or use data processing tools for field processing. By selecting the field processing mode based on personal experience, problems such as low processing efficiency and errors are likely to occur. Moreover, due to limited personal experience, the adaptation rate of the selected field processing mode to the field processing requirements may not be high enough. For example, with the rapid change of business, the field processing modes mastered by technicians may not include the latest field processing modes, resulting in a limited range of field processing modes selected based on personal experience and being unable to reasonably select a more appropriate field processing mode. Therefore, selecting the field processing mode based on personal experience has problems such as low efficiency, being prone to errors, and being difficult to handle large-scale data processing and relatively complex business scenarios.

[0041] In addition to selecting the field processing mode based on personal experience, other ways of auxiliary field processing mode can also be adopted. Currently, there are three commonly used methods to assist in determining the field processing mode. The following will introduce these three auxiliary methods and their existing defects respectively.

[0042] First, assist field processing through a rule engine.

[0043] Based on experience in advance, generate processing rules for executing the field processing mode. When it is necessary to process a field, the processing rules that meet the field processing requirements can be selected to achieve automatic field processing. The rule maintenance cost of the processing rules is relatively high, and the constructed processing rules are limited, making it difficult to cover a wide range of field processing modes. Moreover, it is also difficult for the processing rules to be updated quickly with the change of the business scenario. The method of using processing rules to achieve field processing has the problem that it is difficult to select a more suitable field processing mode for the field processing requirements.

[0044] Second, assist field processing based on a metadata management tool.

[0045] The metadata management tool records the relevant information of the fields. The metadata management tool can provide the function of field lineage analysis for technical personnel. Among them, field lineage refers to the flow and transfer between fields. Based on the results of field lineage analysis, technical personnel can understand the source and processing process of the fields. However, most metadata management tools can only provide the function of field lineage analysis and cannot directly recommend suitable field processing modes to technical personnel.

[0046] Third, use a machine learning model to assist in querying the field processing mode.

[0047] By performing supervised training on the machine learning model, a machine learning model for determining the field processing mode based on the query problem can be trained. Supervised training requires a large amount of labeled data of <query problem, field processing mode>, and the labeling cost is relatively high. In the case where the number of labeled data is small or the labeled data does not cover enough query scenarios, the model accuracy of the trained machine learning model may not be high. The query results obtained by using the machine learning model are not accurate enough. Moreover, the generalization ability of the machine learning model is limited and it is difficult to adapt to different business scenarios of data processing.

[0048] Based on the above introduction of the three existing methods for assisting in determining the field processing mode, it can be seen that the currently commonly used methods for assisting in determining the field processing mode all have certain defects, resulting in the inability to recommend a field processing mode that is more in line with the field processing requirements.

[0049] Based on this, the embodiments of the present application provide a method for constructing a field processing knowledge base. Obtain the field basic information for describing the definition and attributes of the field. The field basic information includes information related to the field. Obtain the field blood relationship for describing the association relationship between fields. The field blood relationship includes information between fields. Then analyze the field basic information and the field blood relationship, and the field processing knowledge for describing the field processing mode can be obtained. Based on the field basic information and the field blood relationship, the field processing mode can be fully mined from the field basic information of the field itself and the association relationship between fields, and a relatively comprehensive field processing mode covering a wide range of processing scenarios can be obtained. Write the field processing knowledge for describing the field processing mode into the field processing knowledge base to realize the precipitation of the relevant knowledge of the field processing mode. The field processing knowledge base can provide the relevant knowledge of the field processing mode, which is convenient for analyzing the field processing mode suitable for the field processing requirements based on the field processing knowledge, so as to select a relatively accurate field processing mode that meets the field processing requirements. The field processing knowledge can also provide an effective selection basis for selecting the field processing mode, and provide a more reliable knowledge basis for data processing and business decision-making.

[0050] In addition, the embodiments of the present application also provide an auxiliary method for field processing. First, obtain the query question for field processing input by the user. The query question is used to describe the user's field processing requirements. According to the field processing knowledge included in the field processing knowledge base and the query question, query the field processing mode that meets the field processing requirements, and generate a query result including the information of the field processing mode. On the one hand, the field processing knowledge base includes the knowledge for describing the field processing mode. Based on the field processing knowledge base, the field processing mode that relatively meets the field processing requirements can be queried. On the other hand, based on the field processing knowledge base, the field processing mode can be automatically queried, reducing the cost of manual selection and learning the field processing mode. Finally, display the query result. The user can select the field processing mode according to the information of the field processing mode included in the query result to implement the field processing. Assisting in selecting the field processing mode based on the field processing knowledge base can improve the efficiency of field processing and reduce the cost of the user's operation of selecting the field processing mode and performing the field processing.

[0051] To facilitate the understanding of the technical solutions provided by the embodiments of the present application, the application scenarios of the method for constructing the field processing knowledge base and the auxiliary method for field processing provided by the embodiments of the present application will be described below with reference to the accompanying drawings.

[0052] The method for constructing the field processing knowledge base and the auxiliary method for field processing provided by the embodiments of the present application can be applied to a field processing assistance system. See Figure 1As shown, the field processing assistance system 100 obtains field basic information and field lineage. The field basic information is used to describe the definition and attributes of the field. The field lineage is used to describe the association relationship between fields. The field processing assistance system 100 extracts field processing knowledge based on the field basic information and field lineage. The field processing knowledge is used to describe the field processing mode. The field processing assistance system 100 writes the field processing knowledge into the field processing knowledge base.

[0053] Among them, as an example, the field processing assistance system 100 can call a large language model (LLM) 101 to process the field basic information and field lineage to obtain field processing knowledge. The LLM 101 can be deployed in the field processing assistance system 100. Alternatively, the LLM 101 is deployed in a computing system independent of the field processing assistance system 100, and provides an invocation interface externally to provide an interactive service externally.

[0054] In the stage of assisting field processing, the field processing assistance system 100 obtains a query question describing the field processing requirement, and generates a query result corresponding to the field processing requirement according to the field processing knowledge and the query question. The query result includes information about the field processing mode. The query result is used to assist field processing. The field processing assistance system 100 displays the query result to the user so that the user can process the field based on the query result.

[0055] As an example, the field processing assistance system 100 can call a large language model 102 to process the query question based on the field processing knowledge base to obtain a query result including information about the field processing mode. The large language model 101 used for constructing the above field processing knowledge base and the large language model 102 used for assisting data processing can be the same large language model or different large language models, which can be flexibly selected according to processing needs, and this application does not make a limitation on this.

[0056] It should be noted that the above application scenarios are only a possible implementation method and do not limit the application scenarios of the method provided by the embodiments of this application.

[0057] Next, the method for constructing the field processing knowledge base and the method for assisting field processing provided by the embodiments of this application are introduced respectively.

[0058] It should be noted that, as shown in Figure 2 The field processing knowledge base can be pre-constructed when the field processing assistance system is in an offline state, so as to provide auxiliary services for field processing externally when the field processing assistance system is in an online state. Figure 2 For the specific content included, please refer to the following introduction.

[0059] For the sake of easy understanding, the construction method of the field processing knowledge base will be introduced first.

[0060] See Figure 3 As shown, this figure is a flowchart of a construction method of a field processing knowledge base provided by an embodiment of the present application. The method includes:

[0061] S301: Obtain field basic information.

[0062] The field basic information includes information related to the field. The field basic information is used to describe the definition and attributes of the field. The sources of the field basic information are, for example, data dictionary documents, business knowledge bases, field naming specifications, and field annotation descriptions, etc., which are used to define or explain fields. Among them, the data dictionary document is used to record the fields corresponding to the data. The business knowledge base includes the knowledge of the fields related to the business. The field naming specification is used to define the field naming method. The field annotation description is used to explain the field.

[0063] In some possible implementation manners, standardizing the obtained field basic information to obtain standardized field basic information can improve the quality of the field basic information, reduce the error of the description, eliminate ambiguity, facilitate subsequent analysis of the standardized field basic information, and improve the efficiency of analyzing the field basic information.

[0064] Standardization includes two aspects: content and format. First, the content included in the field basic information will be introduced.

[0065] The field basic information includes one or more of field attribute information, business information of the field, calculation information of the field, and field quality information.

[0066] Among them, the field attribute information includes Chinese name, English name, data type, value range description, and measurement unit. The Chinese name is a standard and concise Chinese name. The Chinese name is used in display, document recording, and interaction. The English name is the English name or abbreviation of the field. The English name is used in programming, database design, and calling the Application Programming Interface (API). The data type is the data type of the field. Such as integer (INT), floating point number (FLOAT), etc. The value range description is used to describe in detail the allowable value range or enumerated values of the field. In the case where the field value is a numerical value, the measurement unit of the numerical value needs to be explained.

[0067] Business information includes business meaning, usage scenarios, importance, and business owners. Among them, the business meaning is used to explain in detail and accurately the meaning and role of the field in the business context. The usage scenarios include the business scenarios or application names where the field is usually used. The importance is used to illustrate the importance level of the field to the business. The business owner is the technical person in charge of the field.

[0068] Calculation information includes generation rules, upstream dependencies, update frequency, and data sources. Among them, the generation rules are used to describe in detail the calculation method or generation method of the field. The generation rules include specific formulas, algorithms, or business rules. The upstream dependencies include other fields or data sources on which the calculation of this field depends. The dependency relationships include direct dependencies and indirect dependencies. A direct dependency means that the field can be calculated directly using other fields or data sources. An indirect dependency means that an intermediate field can be calculated using other fields or data sources, and then the field can be calculated using the intermediate field. The update frequency is the frequency of field updates. The data source is the original data source of the field.

[0069] Field quality information includes accuracy, integrity, timeliness, and consistency. Accuracy is the specific requirement for the accuracy of the field value. For example, the allowable error range, precision, number of decimal places, etc. of the field value. Integrity is the requirement for the integrity of the field data. For example, whether the field value is allowed to be null (NULL), whether the field value has a default value, whether the field value has a uniqueness constraint, etc. Timeliness is the requirement for the timeliness of the field data. For example, the maximum acceptable time for field value delay, the deadline for field value update, etc. Consistency is the requirement for data consistency between the field and other related fields.

[0070] The above are the standardized requirements for the content of the field basic information. The standardized field basic information can be represented in the JavaScript Object Notation (JSON) format.

[0071] As an example, this application provides a format style for the field basic information:

[0072] {

[0073] "Field Identifier": "comment_level",

[0074] "Chinese Name": "Evaluation Level",

[0075] "English Name": "commentLevel",

[0076] "Data Type": "VARCHAR(10)",

[0077] "Value Range Description": {

[0078] "Good review": "12 - 15 points",

[0079] "Medium review": "7 - 11 points",

[0080] "Bad review": "0 - 6 points"

[0081] },

[0082] "Business meaning": "The evaluation level of users on goods or services",

[0083] "Application scenarios": ["Commodity quality assessment", "Service quality monitoring"],

[0084] "Importance": "Important",

[0085] "Generation rule": "Calculated based on user scores",

[0086] "Upstream dependencies": ["user_score"],

[0087] "Update cycle": "Real-time",

[0088] "Data source": "User evaluation system",

[0089] "Accuracy": "No error",

[0090] "Integrity": "Not allowed to be empty",

[0091] "Timeliness": "Real-time update",

[0092] "Measurement unit": "points",

[0093] }

[0094] The basic information of the fields after standardization processing can improve the format consistency of the basic information of the fields and the comprehensibility of the content. In terms of format, it is convenient for subsequent analysis of the basic information of the fields. In terms of content, it is relatively comprehensive, facilitating a comprehensive understanding of the field meanings and attributes. Furthermore, it realizes the effective analysis of the field association relationships and the extraction of the field processing patterns. The standardized basic information of the fields is convenient for more fully extracting the information of the fields and improving the extraction efficiency of the field processing knowledge.

[0095] S302: Obtain the field blood relationship.

[0096] The field blood relationship is used to describe the association relationship between fields. The field blood relationship can reflect the flow and transmission between fields.

[0097] This application embodiment does not limit the method of obtaining the field blood relationship.

[0098] In a possible implementation, field blood relationship is obtained through the already constructed field blood relationship network. The field blood relationship network includes nodes and the association relationships between the nodes. Among them, the nodes of the field blood relationship network are used to represent fields. The association relationship between the nodes is the field blood relationship.

[0099] In another possible implementation, the field blood relationship is extracted from one or more of the field basic information and the field processing records.

[0100] The field basic information is the information of the field itself. The field basic information includes the information describing the relationship between fields. For example, in the above example, "upstream dependency" includes the dependency relationship between a field and the dependent field. The dependency relationship belongs to the field blood relationship.

[0101] The field processing record is used to record the field processing process. Based on the field processing record, it is possible to analyze the fields involved before and after the field processing, so as to extract the field blood relationship.

[0102] In a possible implementation, the field blood relationship includes one or more of the vertical blood relationship, the horizontal association relationship, and the time series change relationship. Among them, the vertical blood relationship, the horizontal association relationship, and the time series change relationship are all sub-relationships included in the field blood relationship.

[0103] Among them, the vertical blood relationship is used to describe the evolution process of the field from the upstream data processing stage to the downstream data processing stage. That is to say, the vertical blood relationship reflects the evolution process of the field in the data processing process. The upstream data processing stage is an earlier data processing stage. The downstream data processing stage is later than the upstream data processing stage. The horizontal blood relationship is used to describe the type of association relationship between fields. The time series change relationship is used to describe the change process of the field over time.

[0104] The extraction processes of the vertical blood relationship, the horizontal association relationship, and the time series change relationship will be introduced separately below.

[0105] First, the vertical blood relationship.

[0106] The vertical blood relationship can include one or more of the following five contents.

[0107] 1. The full life cycle tracking result.

[0108] The field processing record can include the complete evolution process of each field from the original data to the derived field, for example, including various links such as data collection, cleaning, conversion, aggregation, and calculation. Based on the field processing record, it is possible to analyze the evolution process of the field from the initial stage to the current stage and generate the full life cycle tracking result.

[0109] 2. The processing process record result.

[0110] Each process of field processing includes information in multiple dimensions. For example, it includes operation type, parameter configuration, technical environment, operator, and code version, etc. Based on the field processing records, relevant information for each field processing can be analyzed to generate the result of the processing process record. The result of the processing process record can assist in understanding the field processing process and can be used to troubleshoot processing problems.

[0111] 3. Flow path.

[0112] Based on the field processing records, determine the flow path of the field between different wide tables. And, according to the association relationship between the wide tables, analyze the flow path of the field between the wide tables, which can facilitate the understanding of the data processing flow and understand the role of the field in the wide table.

[0113] 4. Field value evaluation result.

[0114] Based on the basic field information, the field value can be evaluated to obtain the change situation of the field value and generate the field value evaluation result. Specifically, the field value can be determined based on factors such as the usage frequency of the field, the downstream influence range, and the business importance. According to the field value evaluation result, core fields, key fields, and low-value fields can be identified.

[0115] 5. Data quality impact analysis result.

[0116] Analyze how the data quality problems of the upstream fields affect the downstream fields to obtain the data quality impact analysis result. Data quality problems include missing values, outliers, and delays, etc. Based on the data quality impact analysis result, the connection between the upstream fields and the downstream fields can be determined, and the data quality risk can also be evaluated.

[0117] In a possible implementation, various big data processing engines and message queues can be analyzed to obtain the vertical lineage relationship. In this way, the wide applicability and scalability of lineage tracking are ensured.

[0118] Based on the vertical lineage relationship, the source and generation method of the field, the processing steps and conversion logic passed by the field, the flow situation of the field between different systems, tables, and applications, and the transmission path of data quality problems can be determined. In addition, the value of the field can also be determined, and core fields, key fields, and low-value fields can be determined. Evaluating the field value helps to identify core fields and key metrics and guide business optimization.

[0119] Second, horizontal association relationship.

[0120] The types of association relationships between fields can include strong association relationships, moderate association relationships, weak association relationships, mutual exclusion relationships, scenario association relationships, dependency relationships, and similarity association relationships.

[0121] Among them, the strong association relationship, moderate association relationship, weak association relationship, and mutual exclusion relationship are divided based on the correlation strength between fields.

[0122] First, the strong association relationship, moderate association relationship, weak association relationship, and mutual exclusion relationship will be introduced below.

[0123] The strong association relationship usually represents a field group with high-frequency co-processing and can represent the characteristics of core business entities. For example, in the e-commerce scenario, fields such as product identifier, product name, product category, product price, and brand have a strong association relationship.

[0124] The moderate association relationship usually represents a field group related to the business process or fields with a certain degree of correlation. For example, in the order process, fields such as order identifier, order placement time, payment time, shipping time, and receipt time have a moderate association relationship.

[0125] The weak association relationship represents a field group with occasional associations, which may be used together in specific scenarios but have no association in other scenarios. For example, in the user login scenario, fields such as user identifier, registration time, and last login time may have a weak association relationship.

[0126] The mutual exclusion relationship represents a field group with business logic conflicts. Fields with a mutual exclusion relationship usually do not appear or are used together at the same time. For example, the two fields "order paid" and "order cancelled" have a mutual exclusion relationship.

[0127] The strong association relationship, moderate association relationship, weak association relationship, and mutual exclusion relationship can be determined based on the correlation coefficient between fields. As an example, the Pearson correlation coefficient is used as the correlation coefficient. The Pearson correlation coefficient is used to measure the strength and direction of the linear relationship between two variables. The value range of the Pearson correlation coefficient is from -1 to 1.

[0128] In a possible implementation, according to the field basic information, the Pearson correlation coefficient between two fields is calculated. If the Pearson correlation coefficient is greater than 0.8, it is determined that there is a strong association relationship between the fields. If the Pearson correlation coefficient is greater than 0.6 and less than or equal to 0.8, it is determined that there is a moderate association relationship between the fields. If the Pearson correlation coefficient is greater than 0.3 and less than or equal to 0.6, it is determined that there is a weak association relationship between the fields. If the Pearson correlation coefficient is less than 0.1, it is determined that there is a mutual exclusion relationship between the fields.

[0129] It should be noted that the above method of calculating the correlation coefficient and classifying the relationship types according to the threshold is only an example and is not a limitation.

[0130] In addition, the types of association relationships can also be divided by other dimensions.

[0131] Based on the information of the business scenarios included in the field basic information, the fields that are often used together in a specific business scenario are determined as scenario association relationships.

[0132] Based on the calculation methods of the fields included in the field basic information, field dependency relationships are determined. For example, if the field value of field A is calculated from the field values of field B and field C, then field A depends on field B and field C.

[0133] Based on the meanings, sources or usages of the fields included in the field basic information, multiple similar fields are determined. The association relationship between multiple similar fields is a similarity association relationship.

[0134] Based on the horizontal association relationships, the fields that are often used together, the fields with calculation dependencies, the fields used together in specific business scenarios, the fields with similarities, and the fields with conflict or mutual exclusion relationships can be determined, realizing the extraction of various types of association relationships between fields.

[0135] Third, time-series change relationships.

[0136] The time-series change relationships can include one or more of the following six contents.

[0137] 1. Short-term detection change results.

[0138] Detect the changes in various indicators of the field within a preset detection time period to generate short-term detection results. The short-term detection results belong to the field processing records. Based on the short-term detection results, the fields that change over time in the short term can be discovered in a timely manner to obtain short-term detection change results. The various indicators of the field include but are not limited to maximum value, minimum value, average value, median, standard deviation, quantile, null value rate, unique value rate, etc. The preset detection time period is, for example, one week.

[0139] 2. Long-term analysis change results.

[0140] Record the quarterly or annual business trends and field changes to obtain long-term detection results. The long-term detection results belong to the field processing records. Based on the long-term detection results, the long-term change trends of the fields can be analyzed, and then the long-term field changes can be extracted to generate long-term analysis change results. Combining with business development, the impact of business development on field changes can also be determined.

[0141] 3. Mutation results.

[0142] When the change range of each index of a field is detected to exceed 30%, an early warning is triggered. The early warning processing result includes the processing result of the field change. The early warning and the early warning processing result belong to the field processing record. Based on the early warning, the field with mutation can be determined, and based on the early warning processing result, it can be identified whether the mutation is an abnormal situation or a normal situation. By analyzing the early warning and the early warning processing result, a mutation result is obtained. The mutation result includes the change of the field mutation and the mutation attribute. The mutation attribute is normal or abnormal.

[0143] 4. Wide table structure change result.

[0144] The field processing record includes the historical change records of the fields in the wide table recorded in terms of the wide table dimension, such as the addition, deletion, type change, and renaming of the fields in the wide table. By analyzing the wide table historical change records, the change method of the fields in the wide table is determined, and the wide table structure change result is generated.

[0145] 5. Usage change result.

[0146] By analyzing the field basic information and the field processing record, the change of the usage method of the field can be determined, and the usage change result of the field is generated. For example, the field was initially used for report display and later for machine learning model training.

[0147] 6. Lineage change result.

[0148] By analyzing the field basic information and the field processing record, the change of the vertical lineage relationship or the horizontal association relationship between fields over time is determined, and the lineage change result is obtained. For example, the change of the dependency relationship between fields, or the change of the correlation strength between fields.

[0149] Based on the time series change relationship, the change process of the field usage method and the field importance over time, the change process of the field processing mode over time, the change process of the wide table structure to which the field belongs, the mutation of the field, and the change process of the vertical lineage relationship or the horizontal association relationship of the field can be determined.

[0150] The field lineage relationship helps to analyze and understand the source and processing process of the field, is conducive to field tracing and problem troubleshooting, and supports more in-depth data analysis.

[0151] For the three-dimensional field lineage relationship including the above three sub-relationships, it can support the flexible analysis and adjustment of the relationship between fields, can provide more comprehensive information on the association relationship between fields, helps to explore the field processing mode, and is convenient for recommending more appropriate and accurate field processing modes to users during subsequent user queries.

[0152] The field blood relationship, including vertical blood relationship, horizontal association relationship, and temporal change relationship, can not only be used to trace the source and destination of fields, but also be used to analyze the horizontal association and temporal evolution between fields, comprehensively reflecting the association characteristics of fields. Based on the field blood relationship, the association relationship between fields can be comprehensively analyzed, various usage methods of fields and field processing modes can be discovered, field processing knowledge can be generated, so as to provide a powerful knowledge basis for the field processing assistance system, realize more accurate recommendation of field processing modes that meet the field processing requirements, and improve the efficiency and quality of data processing.

[0153] In a possible implementation, based on the field blood relationship including vertical blood relationship, horizontal association relationship, and temporal change relationship, a three-dimensional blood relationship network can be established. The three-dimensional blood relationship network includes the link where the field is generated, which is used to describe the source, processing steps, current status, and future destination of the field. The blood relationship network is used to track the flow process and propagation process of the field to ensure the traceability of data.

[0154] S303: Extract field processing knowledge based on field basic information and field blood relationship.

[0155] The field basic information includes the information of the field itself, and the field blood relationship is the association relationship between fields. Analyzing the field basic information and the field blood relationship can identify the field processing mode from the dimension of the field itself and the dimension of the association relationship between fields, and then extract the field processing knowledge used to describe the field processing mode.

[0156] The embodiments of this application do not limit the implementation manner of extracting field processing knowledge. As an example, use a large language model to process the field basic information and the field blood relationship, identify the field processing mode, and extract the field processing knowledge.

[0157] Using a large language model to automatically mine the field processing mode can, on the one hand, reduce the cost of manual analysis and learning. On the other hand, the large language model can learn based on a large amount of information, fully mine the field processing mode, and obtain a field processing mode with better quality.

[0158] In a possible implementation, first identify the field processing mode according to the field basic information and the field blood relationship. For the identified field processing mode, extract the description information of the field processing mode from the field basic information and the field blood relationship. The description information of the field processing mode is used to describe the field processing mode. Based on the description information of the field processing mode, generate the field processing knowledge that describes the field processing mode.

[0159] As an example, based on the above introduction to field lineage, it can be known that field lineage can include one or more sub - relationships. Vertical lineage, horizontal association relationship, and temporal change relationship are all sub - relationships. Different types of sub - relationships represent association relationships with different characteristics. By adopting an identification strategy corresponding to the sub - relationship, analyzing the field basic information and the sub - relationship, the field processing pattern can be obtained.

[0160] Regarding the characteristics of the association relationship between fields, adopt the corresponding identification strategy, analyze the field basic information and the sub - relationship, and comprehensively identify the field processing pattern from different perspectives of association relationships.

[0161] Next, the possible implementation methods of identifying the field processing pattern using the three sub - relationships will be introduced first.

[0162] The first one: Vertical lineage.

[0163] Analyzing the vertical lineage, the field processing steps can be determined based on the change of the field or the change of the field value. The field processing steps are the operation steps that trigger the change of the field or the field value. For example, the field processing steps include steps such as cleaning, transformation, aggregation, and calculation. For example, cleaning includes missing value processing, outlier processing, duplicate value processing, and data type conversion, etc. Data transformation includes standardization, normalization, discretization, and binning, etc. Data aggregation includes summation, average, maximum value, minimum value, and counting, etc. Data calculation includes calculating new fields through formulas or business rules.

[0164] The field processing steps are part of the field processing pattern. Based on the processing process from the original data to the field, the various field processing steps included in the field processing process can be analyzed, thereby determining the link composed of the field processing steps and obtaining the field processing pattern. The obtained field processing pattern includes at least one field processing step.

[0165] The second one: Horizontal association relationship.

[0166] The horizontal association relationship includes information related to the usage scenario of the field. In addition, the field basic information may also include business information. Analyze the horizontal association relationship and the field basic information to determine the usage scenario of the field. Cluster the fields belonging to the same usage scenario and write the fields belonging to the same usage scenario into the same scenario field set. In this way, the fields can be divided according to the usage scenario, and at least one scenario field set can be obtained. For example, order identifier, order time, payment time, shipping time, receipt time, etc. are divided into the usage scenario of the order and written into the scenario field set of the order.

[0167] For each set of scenario fields and horizontal association relationships, analyze the processing process of fields in the same usage scenario to obtain the field processing mode in the usage scenario. In this way, the field processing mode can be identified in terms of the dimension of the usage scenario, and the field processing mode divided by the usage scenario can be obtained. Based on the field processing mode in the usage scenario, it is possible to recommend the field processing mode in the usage scenario for a specific usage scenario later, which is more accurate.

[0168] The third type: time series change relationship.

[0169] Based on the field basic information and the time series change relationship, the change law of the field and the law of the change of the field processing mode over time can be obtained, and the field processing mode in different time periods can be obtained. For example, the field processing mode of a field changes from a relatively simple aggregation to the feature extraction of a relatively complex machine learning model. In addition, based on the time series change relationship, the frequently used field processing mode can also be obtained.

[0170] In addition, after identifying the field processing mode, the identified field processing mode can be further analyzed. As an example, the frequently used field processing mode can be identified based on the occurrence frequency of the field processing mode. As another example, the sequence and dependency of the field processing mode can also be analyzed based on the field blood relationship. As still another example, determine the similarity degree of the field processing mode, cluster the similar field processing modes, and thus determine the mode category of the field processing mode, which is convenient for subsequent recommendation of the field processing mode. As yet another example, the field processing mode with infrequent use or field quality problems can also be determined, so as to identify the errors or usage risks existing in the field processing mode, and thus effectively eliminate abnormal field processing modes and improve the effectiveness of data processing.

[0171] The format of field processing knowledge can be set in advance. For example, the format of field processing knowledge adopts standardized JSON.

[0172] As an example, the field processing knowledge is as follows:

[0173] {

[0174] "Pattern ID": "P20231108_001",

[0175] "Pattern Name": "Missing Value Filling",

[0176] "Applicable Scenarios": ["Scenario 1", "Scenario 2"],

[0177] "Field Types": ["Numeric Type", "Categorical Type"],

[0178] "Processing Rule": "Rule Description",

[0179] "Parameter Configuration": {

[0180] "Filling Method": "Mean / Median / Mode / Specified Value",

[0181] "Specified Value": "0"

[0182] },

[0183] "Effect Indicators": {

[0184] "Success Rate": "95.6%",

[0185] "Cases": ["Case 1"]

[0186] },

[0187] "Advantages and Disadvantages": {

[0188] "Advantages": ["Simple to use", "Good effect"],

[0189] "Disadvantages": ["May introduce bias"]

[0190] },

[0191] "Precautions": ["Pay attention to data distribution", "Pay attention to business meaning"],

[0192] "Related Patterns": ["P20231108_002", "P20231108_003"],

[0193] "Code Examples": {

[0194] "SQL": "UPDATE table SET column = 0 WHERE column IS NULL;",

[0195] "Python": "df['column'].fillna(0, inplace = True)"

[0196] }

[0197] }

[0198] The field processing knowledge describes the field processing mode. The above field processing knowledge includes the key information of the field processing mode, including the mode identification (Identification, ID), mode name, applicable scenario, field type, processing rules, parameter configuration, effect indicators, advantages and disadvantages, precautions, related patterns, and code examples.

[0199] Among them, the processing rules are used to describe in detail the field processing steps, methods, and parameters, which are the core content of field processing knowledge. Data cleaning is used to describe how to handle missing values, outliers, and duplicate values. Feature derivation is used to describe how to generate new feature fields from existing fields through mathematical formulas or business rules. The applicable scenario is used to indicate the business scenarios to which the field processing mode applies. For example, Scenario 1 is order processing. Order processing is used to indicate that the field processing mode is applicable to scenarios such as order management and logistics tracking. By clarifying the applicable scenario, it is ensured that a field processing mode that conforms to the business scenario can be recommended, improving the accuracy of the field processing mode.

[0200] Field processing knowledge with a unified format can clearly describe the field processing mode, facilitating the subsequent selection of a field processing mode that meets the processing requirements based on the field processing knowledge.

[0201] S304: Write the field processing knowledge into the field processing knowledge base.

[0202] The field processing knowledge base is used to store and manage the field processing knowledge extracted from the field basic information and field lineage. The field processing knowledge base can provide an effective basis and reference for selecting the field processing mode.

[0203] In addition to extracting the field processing knowledge, it is also possible to evaluate the field processing mode described by the field processing knowledge.

[0204] In one possible implementation, the quality of the field processing mode itself can be evaluated. First, obtain the first evaluation information. The first evaluation information includes one or more of the usage frequency, success rate, generality value, innovation value, and business value of the field processing mode. Among them, the usage frequency is the historical usage frequency of the field processing mode. The success rate is used to measure the proportion of the historical successful use of the field processing mode. For example, the success rate includes the data quality improvement rate or business indicator improvement rate after processing through the field processing mode. The generality value is used to evaluate the scope of business scenarios applicable to the field processing mode. For example, the scope of business scenarios includes multiple business scenarios or field types applicable to the field processing mode, etc. The innovation value is used to evaluate the novelty degree of the field processing mode. For example, the novelty degree of the field processing mode can be measured by the new algorithms, technologies, or ideas adopted by the field processing mode. The business value is used to evaluate the impact degree of the field processing mode on the business. According to the first evaluation information, determine the static quality value of the field processing mode. The static quality value is used to measure the quality of the field processing mode. The static quality value is a numerical value for evaluating and measuring the value of the field processing mode itself.

[0205] As an example, the static quality value can be expressed as:

[0206] Static quality value = α1 × Usage frequency + β1 × Success rate + γ1 × General value + δ × Innovation value + ε × Business value (1)

[0207] Among them, α1, β1, γ1, δ, and ε are all weight coefficients.

[0208] The static quality value of the field processing mode can be used to assist in sorting and screening candidate field processing modes when selecting a field processing mode, so as to improve the quality of the finally selected field processing mode.

[0209] In some possible implementation manners, the field processing knowledge base is a dynamic knowledge base, that is to say, the field processing knowledge included in the field processing knowledge base is dynamically updated and not statically unupdated.

[0210] When the update condition is met, the field processing knowledge included in the field processing knowledge base can be updated. Among them, the update condition includes one or more of the following four items:

[0211] First, identify new field processing knowledge.

[0212] As an example, the field basic information and field blood relationship can be analyzed in real time or periodically, especially the updated field basic information and field blood relationship, so as to extract new field processing modes.

[0213] Second, obtain the exception check result for the field processing knowledge base.

[0214] In a possible implementation manner, the field processing knowledge included in the field processing knowledge base can be checked regularly to timely discover abnormal or invalid field processing knowledge and generate an exception check result. In this way, the invalid or abnormal field processing knowledge can be removed based on the exception check result to ensure the validity of the field processing knowledge in the field processing knowledge base, thereby helping to obtain more accurate and effective field processing modes based on the field processing knowledge base.

[0215] The check time can be determined based on the check period. The check period can be set as needed. For example, one week can be used as the check period.

[0216] In another possible implementation manner, the operation status of the field processing knowledge base is checked. If there are problems or exceptions in the operation of the field processing knowledge base, identify the cause of the exception and repair the field processing knowledge base in a timely manner. If the exception is due to the field processing knowledge, update and repair the field processing knowledge that causes the exception to reduce the impact on the field processing assistance system.

[0217] Checking the field processing knowledge or the operation of the field processing knowledge base can effectively ensure the quality of the field processing knowledge.

[0218] Third, obtain the field processing knowledge provided by the user.

[0219] The user can provide a new field processing mode. Generate field processing knowledge based on the field processing mode provided by the user and write it into the field processing knowledge base. Before writing it into the field processing knowledge base, the field processing knowledge can also be audited or verified. After passing the audit or verification, the field processing knowledge is then written into the field processing knowledge base.

[0220] Fourth, obtain the feedback result for the recommended field processing mode.

[0221] After the user obtains the query result including the information of the field processing mode through the field processing assistance system, the user can evaluate the query result to generate a feedback result. The feedback result may include the optimization information for the field processing mode. The field processing mode can be updated and optimized based on the feedback result.

[0222] Each time the field processing knowledge base is updated, the update content can also be recorded to facilitate the control of the update process of the field processing knowledge base for subsequent backtracking and management.

[0223] The dynamic field processing knowledge base can quickly adapt to the changes in data and business requirements, continuously expand new field processing modes, and thus be able to cope with the challenges of new data processing.

[0224] In addition, based on the update situation of the field processing knowledge, the quality of the field processing knowledge can be dynamically evaluated to determine the quality change of the field processing knowledge over time.

[0225] As an example, first obtain the second evaluation information. The second evaluation information includes one or more of the static quality value, usage frequency, success rate, and citation times of the field processing mode.

[0226] Among them, the static quality value is used to evaluate and measure the value of the field processing mode itself. The usage frequency is the historical usage frequency of the field processing mode. The success rate is used to measure the proportion of the historical successful use of the field processing mode. The citation times is the number of times the field processing mode is cited in the process of historically recommending the field processing mode.

[0227] Determine the dynamic quality value of the field processing mode according to the second evaluation information.

[0228] As an example, the dynamic quality value can be expressed as:

[0229] W(t)=W0e -λt+f·(α2S + β2U + γ2N) (2)

[0230] Among them, W(t) is the dynamic quality value and also the weight of the field processing mode in the time dimension of time t. W0 is the initial weight of the field processing mode, for example, it is the static quality value. t is the time from the last update moment to the current moment of the field processing mode. λ is the decay coefficient, which controls the decay speed of the dynamic quality value over time. f is the adjustment factor, which is used to balance the influence of time and effect. S is the success rate. U is the usage frequency. N is the number of citations. α2, β2, and γ2 are all weight coefficients.

[0231] Based on the above formula (2), it can be seen that the dynamic quality value gradually decays over time, but increases with the increase of the success rate, usage frequency, and number of citations. In this way, the quality of the field processing mode over time can be measured. Based on the dynamic quality value, the field processing knowledge that reflects the latest processing requirements can be selected, and the effectiveness of the selected field processing mode can be improved.

[0232] The above is an introduction to the update process of the field processing knowledge base. In another possible implementation, the field processing knowledge included in the field processing knowledge base can also be verified to ensure the effectiveness of the field processing mode described by the field processing knowledge, ensure that the field processing knowledge has high quality, and thus improve the field processing mode selected based on the field processing knowledge base.

[0233] The verification methods that the field processing knowledge base can adopt include one or more of the following six verification methods.

[0234] First: Offline verification method.

[0235] The AB test method is used to verify the effect of the new field processing mode offline. The AB test method is to test the updated field processing mode and the field processing mode before the update, evaluate the performance of the field processing mode before and after the update in different scenarios, and compare it with the effects of other field processing modes. Based on the evaluation results and comparison results, it is determined whether the updated field processing mode has higher quality. If it is determined that the updated field processing mode has higher quality, the updated field processing mode passes the verification.

[0236] Second: Online verification method.

[0237] The updated field processing mode is tested online on a small scale to achieve an online AB test and evaluate the data processing effect of the field processing mode before and after the update. If it is determined that the updated field processing mode has better data processing effect, the updated field processing mode passes the verification.

[0238] Third: Real-time detection method.

[0239] Perform real-time detection on the field processing knowledge included in the field processing knowledge base. If abnormalities occur in the usage frequency, success rate, error rate, etc. of the field processing mode described by the field processing knowledge, it is determined that the verification of the field processing knowledge fails, and the abnormal field processing knowledge is processed in a timely manner.

[0240] Fourth: Expert review and verification.

[0241] Obtain the evaluation results of experts on the field processing knowledge included in the field processing knowledge base. Determine whether the quality of the field processing knowledge meets the verification requirements based on the experts' evaluation results.

[0242] Fifth: User feedback verification.

[0243] Users can evaluate the field processing mode selected based on the field processing knowledge base to generate feedback results. The feedback results may include the adoption results, modification content, scores, or evaluation results of the field processing mode. Determine whether the quality of the field processing knowledge meets the verification requirements based on the users' evaluation results.

[0244] Sixth: Emergency handling.

[0245] If problems or abnormalities occur in the operation of the field processing knowledge base, identify the causes of the abnormalities. If the abnormality is due to the field processing knowledge, the verification of the field processing knowledge that causes the abnormality fails.

[0246] By verifying the field processing knowledge, ensure the quality, effectiveness, and reliability of the field processing knowledge included in the field processing knowledge base, realize providing users with highly accurate and high-quality field processing modes based on the field processing knowledge base, effectively reduce the errors in field processing, and improve the efficiency of users in processing data.

[0247] Based on the above method for constructing the field processing knowledge base, the experience of field processing and field processing modes can be precipitated, facilitating the sharing and reuse of knowledge, avoiding the extraction of duplicate field processing knowledge, and avoiding duplicate knowledge extraction operations. The high-quality field processing modes extracted based on the information of the fields themselves and the association relationships between fields can provide a more reliable theoretical basis for data analysis and business decision-making.

[0248] The above is an introduction to the process of constructing the field processing knowledge base. Based on the constructed field processing knowledge base, the field processing mode can be queried. The following describes the process of querying the field processing mode based on the field processing knowledge base.

[0249] In a possible implementation, the auxiliary method for field processing is executed by invoking a large language model. By using the large language model to process query questions based on the field processing knowledge base and obtain query results, it is possible to achieve automated and intelligent field processing assistance, improve the efficiency of selecting field processing modes, and reduce the cost of manual operations.

[0250] The above content introduces the construction method of the field processing knowledge base. The following is an introduction to the auxiliary method for field processing based on the field processing knowledge base.

[0251] See Figure 4 As shown in the figure, this figure shows an auxiliary method for field processing provided by an embodiment of the present application. The method includes:

[0252] S401: Obtain a query question for field processing.

[0253] The query question is a question provided by the user to describe the field processing requirements. The query question is used to query the field processing mode suitable for the user during the field processing. The query question may include information related to the processing requirements such as the business scenario, processing purpose, and field context of the field to be processed by the user. The field context refers to information related to field processing, such as including field type, business scenario, and processing operations that have been performed, etc.

[0254] The present application does not limit the language used for the query question. As an example, the query question input by the user can be described in natural language.

[0255] The query question can be a question input by the user in a single interaction, or can include questions input by the user in multiple interactions. For example, in the case of invoking a large language model to execute the auxiliary method for field processing, the large language model can interact with the user multiple times to obtain a query question including multiple rounds of conversations. In this way, the field processing requirements of the user can be gradually clarified through multiple rounds of conversations, and more accurate recommendations for field processing modes can be realized based on the clarified field processing requirements.

[0256] In addition, it should be noted that the query question can be the initial question input by the user in the interaction, or can be a follow-up question after obtaining the query result. The embodiments of the present application do not limit this.

[0257] In a possible implementation, during the interaction with the user, based on a part of the text of the query question input by the user, it is possible to predict the field processing requirements of the user and display recommended supplementary text to the user in order to achieve automatic supplementation of the query question.

[0258] As an example, first obtain the input text of the query question. The input text of the query question is a part of the query question. Based on the input text, the field processing requirements can be analyzed. For example, the input text includes "cleaning". By performing semantic analysis on "cleaning", it can be determined that the field processing requirements include cleaning the field. Based on the field processing requirements represented by the input text, recommended supplementary text is determined. The recommended supplementary text is used to form a query question with the input text after being selected. The recommended supplementary text can be determined based on historical query questions or information related to the field processing requirements included in the field processing knowledge. For example, in historical query questions, the term that co-occurs with "cleaning" with a relatively high frequency is "missing value processing", so "missing value processing" can be used as the recommended supplementary text. Another example is that the field processing knowledge includes scenarios applicable to data cleaning, and the processing rule includes "missing value filling". Based on the field processing knowledge, it can be determined that the user may need to query the processing mode for missing values, and then "missing value processing" is generated as the recommended supplementary text.

[0259] Then display the recommended supplementary text related to the input text. For example, display the recommended supplementary text in gray color. Or display the recommended supplementary text in the form of a selection control. The embodiments of the present application do not limit the display manner of the recommended supplementary text.

[0260] The user can select the appropriate recommended supplementary text by moving the pointer or triggering the arrow keys, and select the recommended supplementary text by clicking or triggering the shortcut keys. After the recommended supplementary text is selected, it can be automatically combined with the input text. The user can continue to input based on the combined text, or directly use the combined text as the text of the complete query question.

[0261] The recommended supplementary text can guide the user to express the field processing requirements, reduce the user's input operations, simplify the user operation process, and reduce the user's operation cost.

[0262] S402: Generate a query result corresponding to the field processing requirements according to the field processing knowledge included in the field processing knowledge base and the query question.

[0263] Based on the query question, the user's field processing requirements can be determined. According to the field processing knowledge included in the field processing knowledge base, query the field processing mode that meets the field processing requirements, and generate a query result including information on the field processing mode.

[0264] It should be noted that based on the field processing knowledge base, the field processing requirements of users in multiple aspects can be met. The query result includes one or more of the following types of sub-information.

[0265] Introduction text of the field processing mode, parameters included in the field processing mode, field processing results, code for implementing the field processing mode, comparison results between multiple field processing modes, and exception modification suggestions for the exception field processing mode.

[0266] Among them, the introduction text of the field processing mode is used to introduce the field processing mode. As an example, the introduction text of the field processing mode includes content such as the processing rules, historical cases, advantages and disadvantages, applicable scenarios, expected effects, reasons for recommendation, and risk warnings of the field processing mode. Based on the introduction text of the field processing mode, users can fully understand the processing logic of the field processing mode, the applicable processing scenarios, the expected effects, the basis for selecting the field processing mode, and the advantages and disadvantages of the field processing mode. This facilitates users to select the field processing mode for actual use, increases users' trust in the query results, effectively assists users in processing data, and improves the efficiency of users' data processing.

[0267] The parameters included in the field processing mode are the parameters that need to be configured in the process of implementing field processing. The parameters included in the field processing mode can be determined based on the information related to the parameters recorded in the field processing knowledge and the field processing requirements of the user. This can achieve automatic parameter configuration, eliminating the need for users to manually select parameters, reducing the operation cost of users, and reducing the error probability of manual configuration by users.

[0268] The field processing result is the result obtained after processing the fields included in the query problem according to the field processing mode. In this way, users can intuitively view the result after processing according to the field processing mode, which is convenient for users to select.

[0269] The field processing process can be implemented by executing code. For some processing steps included in the field processing mode, code for implementing the field processing mode can be automatically generated based on the information related to code generation recorded in the field processing knowledge and the field processing requirements. The language type of the code is not limited. Users can directly obtain the code for implementing the field processing steps from the query results and run it directly or after adjustment, providing a basis for users to edit code. There is no need for users to generate code completely through manual editing, greatly reducing the cost and threshold of users' editing of field processing code and improving the efficiency of users' data processing.

[0270] In some possible cases, there may be multiple field processing modes that meet the field processing requirements. The query result includes information on multiple field processing modes. The query result also includes the comparison results between multiple field processing modes. As an example, the comparison results between multiple field processing modes include the expected effects of adopting each field processing mode, as well as the similarities and differences between multiple field processing modes. Based on the comparison results between multiple field processing modes, the user can clarify the differences between multiple field processing modes and then select a more appropriate field processing mode.

[0271] When the user processes a field using a field processing mode, an exception may occur. The user can also query for problems that may exist in the abnormal field processing mode by querying questions. Correspondingly, the query result includes abnormal modification suggestions for the abnormal field processing mode. The user can identify the existing problems based on the abnormal modification suggestions and make corresponding modifications.

[0272] The above is an introduction to the possible sub-information that the query result may include. Next, the possible implementation methods for generating the query result will be introduced.

[0273] In one possible implementation, the embodiments of the present application provide a possible implementation method for generating a query result, including three steps A1 - A3.

[0274] A1: First, analyze the query question to determine the user's field processing requirements.

[0275] In one possible implementation, the query question includes a specific target field and the business scenario involved in field processing, and can be analyzed from the aspects of the semantics of the field and the business scenario to determine the field processing requirements.

[0276] For example, determine the target field included in the query question, and determine the semantics of the target field according to the information such as the field name, type, value range, business meaning, and data quality included in the field basic information of the target field.

[0277] Analyze the business scenario included in the query question. For example, analyze the field context included in the query text to understand the user's current field processing task and the usage purpose of the target field. For example, the field processing task may be a data cleaning task, a feature engineering task, a report generation task, a data mining task, a model training task, etc.

[0278] The usage purpose of the target field is related to the field processing task and the field semantics. The usage purpose of the target field may be, for example, calculating metrics, building a machine learning model, generating a report, data analysis, etc.

[0279] Based on the field processing task and the usage purpose of the target field, the field processing requirements can be analyzed. The field processing requirements are the processing requirements that the user may need to perform on the target field.

[0280] A2: In the field processing knowledge base, query the target field processing mode that matches the field processing requirements.

[0281] As an example, based on at least one retrieval dimension, the target field processing mode that matches the field processing requirements can be queried.

[0282] The retrieval dimensions include one or more of vertical retrieval, horizontal retrieval, scenario retrieval, and semantic retrieval. Among them, vertical retrieval is used to trace the field processing mode and successful experience of the field history. For example, the target field is a field obtained by aggregating multiple fields. Through vertical retrieval, a similar aggregation field processing mode can be determined.

[0283] Horizontal retrieval is used to query the processing solutions of related fields. For example, if the target field is the "order amount" field, through horizontal retrieval, the field processing mode of the "payment amount" associated with the "order amount" field can be determined. The field processing mode of the "payment amount" is used as the field processing mode of the queried "order amount" field.

[0284] Scenario retrieval is used to query the common field processing modes in the business scenario or similar business scenarios. For example, if the query question includes the "order process" business scenario, based on the field processing knowledge base, the field processing mode in the "order process" business scenario is queried.

[0285] Semantic retrieval is used to perform retrieval based on the semantic information of the target field.

[0286] In a possible implementation, the field processing mode obtained based on the above retrieval dimensions can be directly used as the target field processing mode. The number of target field processing modes can be one or more.

[0287] In another possible implementation, the field processing mode obtained based on the above retrieval dimensions can be used as the candidate field processing mode. The number of candidate field processing modes can be one or more. Further, at least one target field processing mode is selected from at least one candidate field processing mode.

[0288] As an example, based on the selection dimension, the candidate field processing modes are sorted according to the matching degree, and the candidate field processing modes that meet the selection conditions are used as the target field processing modes.

[0289] The selection dimensions include one or more of the historical common dimension, business scenario dimension, association dimension, and preference dimension.

[0290] Among them, the historical common dimension filters the target field processing mode from historical usage habits. The candidate field processing mode that is commonly used historically is used as the target field processing mode.

[0291] The business scenario dimension filters the target field processing mode from the perspective of the business scenario. For example, in the business scenario, the candidate field processing mode with the best quality is used as the target field processing mode.

[0292] The association dimension filters the target field processing mode from the dimension of the association relationship with the field. For example, the candidate field processing mode with the highest degree of association with the target field is determined as the target field processing mode. The degree of association between the candidate field processing mode and the target field can be determined based on the field type processed by the candidate field processing mode or the association relationship between the specific field and the target field.

[0293] The preference dimension filters the target field processing mode from the dimension of user preferences. For example, based on the historical feedback results of the user, analyze the field processing mode that the user prefers to use, and determine the candidate field processing mode that meets the user's preference as the target field processing mode.

[0294] Based on the above selection dimensions, it is possible to provide users with personalized field processing modes that meet the user's needs and improve the user experience.

[0295] A3: Generate a query result according to the field processing knowledge of the target field processing mode.

[0296] Finally, the field processing knowledge of the target field processing mode includes information describing the target field processing mode. Extract the information of the target field processing mode from the field processing knowledge to generate a query result.

[0297] The above implementation method of generating query results based on field processing knowledge and query questions is only an example. In another possible implementation method, before generating query results, a field knowledge graph is first obtained. The field knowledge graph includes nodes and the connection relationships between the nodes. Among them, the nodes represent fields. The connection relationships between the nodes represent the association relationships between fields. The field knowledge graph can be constructed based on the above-mentioned field blood relationship or field blood network. The field knowledge graph can reflect the association relationships between fields. Based on the field knowledge graph, the field processing knowledge base, and the query question, query results are generated. Based on the field knowledge graph, other fields associated with the target field can be determined, so that the field processing mode suitable for processing the target field can be queried based on other fields associated with the target field. The field knowledge graph can implement Retrieval-Augmented Generation (RAG), effectively supplement the association relationships between fields. During the query process, not only can a single field be understood, but also the association relationships between fields can be understood, thereby more accurately analyzing the target field to be processed by the user and querying a higher-quality field processing mode.

[0298] S403: Display the query results.

[0299] By displaying the query results, the field processing mode is visually displayed, which is convenient for users to understand the process of field processing and select and debug the field processing mode used based on the query results.

[0300] In a possible implementation method, the field processing modes included in the query results are displayed in the form of a flowchart or a decision tree, which is convenient for users to quickly understand the processing logic. By visually displaying the field processing mode, the user's trust in the query results is enhanced.

[0301] It should be noted that the above-mentioned auxiliary method for field processing can be executed by calling a large language model. By using the large language model to assist in processing fields, the efficiency and quality of field processing can be improved, the manual operation cost can be reduced, and foreseeable processing risks can be effectively avoided.

[0302] In addition, the application controls corresponding to the field processing mode can also be displayed. Users can trigger the application controls to quickly apply the corresponding field processing mode. For example, run the code included in the query results for implementing the field processing mode. This is convenient for users to quickly apply the field processing mode obtained by the query and reduces the user's operation cost.

[0303] Based on the auxiliary method for field processing described above, according to the high-quality field processing knowledge included in the field processing knowledge base, a field processing mode that more meets the user's field processing requirements can be queried, thereby effectively assisting the user in field processing. In this way, a suitable field processing mode can be selected, the field processing process can be optimized, the consumption of processing resources can be reduced, and the efficiency of field processing can be improved. In addition, the quality of field processing can also be improved, and rework and losses caused by problems with the field processing mode can be reduced.

[0304] Based on a method for constructing a field processing knowledge base provided in the above method embodiment, an embodiment of the present application also provides a device for constructing a field processing knowledge base. The device for constructing a field processing knowledge base will be described below with reference to the accompanying drawings.

[0305] See Figure 5 As shown, this figure is a schematic structural diagram of a device for constructing a field processing knowledge base provided in an embodiment of the present application. As Figure 5 shown, the device for constructing a field processing knowledge base includes:

[0306] A first acquisition module 501, configured to acquire field basic information and field blood relationship. The field basic information is used to describe the definition and attributes of the field, and the field blood relationship is used to describe the association relationship between fields;

[0307] An extraction module 502, configured to extract field processing knowledge based on the field basic information and the field blood relationship. The field processing knowledge is used to describe the field processing mode;

[0308] A construction module 503, configured to write the field processing knowledge into a field processing knowledge base, and the field processing knowledge base is used to assist field processing.

[0309] In a possible implementation manner, the extraction module 502 is specifically configured to:

[0310] Based on the field basic information and the field blood relationship, identify the field processing mode and extract the description information of the field processing mode;

[0311] Generate field processing knowledge including the description information of the field processing mode.

[0312] In a possible implementation manner, the extraction module 502 is configured to identify the field processing mode based on the field basic information and the field blood relationship, including:

[0313] The extraction module 502 is configured to determine the sub-relationships included in the field blood relationship; adopt an identification strategy corresponding to the sub-relationship, and analyze the field basic information and the sub-relationship to obtain the field processing mode.

[0314] In a possible implementation, the sub-relationship is a vertical blood relationship, and the vertical blood relationship is used to describe the evolution process of the field from the upstream data processing stage to the downstream data processing stage, where the upstream data processing stage is earlier than the downstream data processing stage; the extraction module 502 is configured to analyze the field basic information and the sub-relationship to obtain a field processing mode, including:

[0315] The extraction module 502 is configured to analyze the field basic information and the vertical blood relationship to obtain a field processing step, where the field processing step is an operation step that triggers a change in the field or the field value; based on the recognized processing steps, a field processing mode is obtained, and the field processing mode includes at least one of the field processing steps;

[0316] and / or,

[0317] The sub-relationship is a horizontal association relationship, and the horizontal blood relationship is used to describe the type of association relationship between the fields. The extraction module 502 is configured to analyze the field basic information and the sub-relationship to obtain a field processing mode, including:

[0318] The extraction module 502 is configured to determine a usage scenario according to the service information included in the field basic information and the horizontal association relationship; classify the fields according to the usage scenario to obtain a scenario field set, where the scenario field set includes fields belonging to the same usage scenario; determine the field processing mode in the usage scenario according to the scenario field set and the horizontal association relationship;

[0319] and / or,

[0320] The sub-relationship is a time-sequence change relationship, and the time-sequence change relationship is used to describe the change process of the field over time. The extraction module 502 is configured to analyze the field basic information and the sub-relationship to obtain a field processing mode, including:

[0321] The extraction module 502 is configured to determine the field processing mode in different time periods based on the field basic information and the time-sequence change relationship.

[0322] In a possible implementation, the device further includes:

[0323] A second acquisition module, configured to acquire first evaluation information, where the first evaluation information includes one or more of the usage frequency, success rate, general value, innovation value, and business value of the field processing mode. The success rate is used to measure the proportion of historical successful uses of the field processing mode. The general value is used to evaluate the scope of business scenarios applicable to the field processing mode. The innovation value is used to evaluate the novelty degree of the field processing mode. The business value is used to evaluate the impact degree of the field processing mode on the business.

[0324] A first evaluation module, configured to determine a static quality value of the field processing mode according to the first evaluation information. The static quality value is used to measure the quality of the field processing mode and is used to assist in querying the field processing mode.

[0325] And / or

[0326] A third acquisition module, configured to acquire second evaluation information, where the second evaluation information includes one or more of the static quality value, usage frequency, success rate, and citation times of the field processing mode. The citation times are the number of times the field processing mode is recommended.

[0327] A second evaluation module, configured to determine a dynamic quality value of the field processing mode according to the second evaluation information. The dynamic quality value is used to measure the quality of the field processing mode after changing with time and is used to assist in querying the field processing mode.

[0328] In a possible implementation manner, the step of extracting field processing knowledge based on the field basic information and the field lineage is performed by invoking a large language model.

[0329] In a possible implementation manner, the apparatus further includes:

[0330] An update module, configured to update the field processing knowledge included in the field processing knowledge base if an update condition is satisfied. The update condition includes one or more of the following: identifying new field processing knowledge, obtaining an exception check result for the field processing knowledge base, obtaining field processing knowledge provided by a user, and obtaining a feedback result for a recommended field processing mode.

[0331] Based on the auxiliary method for field processing provided in the foregoing method embodiment, an embodiment of the present application further provides an auxiliary apparatus for field processing. The auxiliary apparatus for field processing will be described below with reference to the accompanying drawings.

[0332] See Figure 6 As shown in the figure, the figure is a schematic structural diagram of an auxiliary apparatus for field processing provided in an embodiment of the present application. AsFigure 6 As shown, the auxiliary device for field processing includes:

[0333] An acquisition module 601, configured to acquire a query question for field processing, where the query question is used to describe the field processing requirement;

[0334] A query module 602, configured to generate a query result corresponding to the field processing requirement according to the field processing knowledge included in the field processing knowledge base and the query question. The field processing knowledge base is generated according to the above-mentioned method for constructing the field processing knowledge base. The field processing knowledge is used to describe the field processing mode. The query result includes information about the field processing mode, and the query result is used to assist field processing;

[0335] A display module 603, configured to display the query result.

[0336] In a possible implementation manner, the query module 602 is specifically configured to:

[0337] Analyze the query question to determine the field processing requirement;

[0338] In the field processing knowledge base, query for a target field processing mode that matches the field processing requirement;

[0339] Generate a query result according to the field processing knowledge of the target field processing mode.

[0340] In a possible implementation manner, the query module 602 is configured to analyze the query question to determine the field processing requirement, including:

[0341] The query module 602 is configured to analyze the query question to determine the field processing task and the usage purpose of the target field. The target field is the field included in the query question; according to the field processing task and the usage purpose of the target field, determine the field processing requirement.

[0342] In a possible implementation manner, the query module 602 is configured to query for a target field processing mode that matches the field processing requirement, including:

[0343] The query module 602 is configured to query for candidate field processing modes that match the field processing requirement; use the candidate field processing modes that meet the selection conditions under the selection dimension as the target field processing mode. The selection dimension includes one or more of the historical common dimension, business scenario dimension, association dimension, and preference dimension.

[0344] In a possible implementation manner, the query module 602 is specifically configured to:

[0345] Obtain a field knowledge graph, where the field knowledge graph is constructed according to field blood relationship, and the field blood relationship is used to describe the association relationship between fields. The field knowledge graph includes fields and the association relationship between fields;

[0346] Generate a query result corresponding to the field processing requirement according to the field knowledge graph, the field processing knowledge base, and the query question.

[0347] In a possible implementation manner, the obtaining module 601 is specifically configured to:

[0348] Obtain the input text of the query question, where the input text of the query question is a partial text of the query question;

[0349] The display module 603 is further configured to display recommended supplementary text related to the input text. The recommended supplementary text is determined based on the field processing requirement represented by the input text, and is used to form a query question with the input text after being selected.

[0350] In a possible implementation manner, it is characterized in that the query result includes one or more of the following types of sub-information, and the type of sub-information included in the query result is determined based on the field processing requirement:

[0351] Introduction text of the field processing mode, parameters included in the field processing mode, code for implementing the field processing mode, comparison results between multiple field processing modes, and abnormal modification suggestions for abnormal field processing modes.

[0352] In a possible implementation manner, the auxiliary device for field processing executes each step by calling a large language model.

[0353] Next, refer to Figure 7 , which shows a schematic structural diagram of an electronic device 700 suitable for implementing the embodiments of the present application. The terminal device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable android devices, tablet computers), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs (televisions), desktop computers, etc. Figure 7 The shown electronic device is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0354] As shown Figure 7 in FIG. 700, the electronic device 700 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 701, which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0355] Generally, the following devices may be connected to the I / O interface 705: an input device including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 7 the electronic device 700 with various devices is shown, it should be understood that it is not required to implement or include all the shown devices. Instead, more or fewer devices may be implemented or included.

[0356] Specifically, according to an embodiment of the present application, the process described above with reference to the flowchart may be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program codes for performing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above functions defined in the construction method of the field processing knowledge base or the auxiliary method of field processing of the embodiments of the present application are executed.

[0357] The electronic device provided by the embodiments of the present application and the construction method of the field processing knowledge base and the auxiliary method of field processing provided by the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment may be referred to the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0358] Based on the method for constructing a field processing knowledge base and the auxiliary method for field processing provided in the above method embodiments, an embodiment of the present application provides a computer storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the method for constructing a field processing knowledge base or the auxiliary method for field processing as described in any of the above embodiments.

[0359] It should be noted that the computer-readable medium in the present application may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program, and this program can be used by or combined with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, and this computer-readable signal medium can send, propagate, or transmit a program for use by or combined with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0360] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (Hyper Text Transfer Protocol), and can be interconnected with digital data communication in any form or medium (for example, a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (for example, the Internet), and end-to-end networks (for example, ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0361] The above computer-readable medium may be included in the above electronic device; or may exist separately without being assembled into the electronic device.

[0362] The above computer-readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to execute the method for constructing the field processing knowledge base or the auxiliary method for field processing.

[0363] Computer program code for performing the operations of this application may be written in one or more programming languages or combinations thereof. The above programming languages include, but are not limited to, object-oriented programming languages such as Java, Smalltalk, C++; and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0364] The flowcharts and block diagrams in the drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that, in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0365] The modules involved in the embodiments described in this application may be implemented in software or in hardware. Among them, the name of the unit / module does not, in some cases, constitute a limitation on the unit itself. For example, the voice data acquisition module may also be described as the "data acquisition module".

[0366] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chip (SOCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0367] In the context of this application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM or Flash memory), an optical fiber, a portable Compact Disc Read Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0368] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the systems or apparatuses disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions in the method section.

[0369] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist simultaneously. Here, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (one)" or its similar expression below refers to any combination of these items, including any combination of single items (ones) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0370] It should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0371] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing a field processing knowledge base, characterized in that The method includes: Obtaining field basic information and field lineage relationships, where the field basic information is used to describe the definition and attributes of a field, and the field lineage relationships are used to describe the association relationships between fields; Based on the field basic information and the field lineage relationships, extracting field processing knowledge, where the field processing knowledge is used to describe field processing patterns; Writing the field processing knowledge into a field processing knowledge base, where the field processing knowledge base is used to assist field processing.

2. The method according to claim 1, characterized in that, The extracting field processing knowledge based on the field basic information and the field lineage relationships includes: Based on the field basic information and the field lineage relationships, identifying field processing patterns and extracting the description information of the field processing patterns; Generating field processing knowledge including the description information of the field processing patterns.

3. The method according to claim 2, wherein The identifying field processing patterns based on the field basic information and the field lineage relationships includes: Determining the sub-relationships included in the field lineage relationships; Using an identification strategy corresponding to the sub-relationships to analyze the field basic information and the sub-relationships to obtain field processing patterns.

4. The method according to claim 3, characterized in that, The sub-relationship is a vertical lineage relationship, where the vertical lineage relationship is used to describe the evolution process of the field from an upstream data processing stage to a downstream data processing stage, and the upstream data processing stage is earlier than the downstream data processing stage; The analyzing the field basic information and the sub-relationships to obtain field processing patterns includes: Analyzing the field basic information and the vertical lineage relationship to obtain field processing steps, where the field processing steps are operation steps that trigger changes in fields or field values; Identifying field processing patterns based on the processing steps, where the field processing patterns include at least one of the field processing steps; And / or, The sub-relationship is a horizontal association relationship, where the horizontal lineage relationship is used to describe the type of association relationship between the fields, and the analyzing the field basic information and the sub-relationships to obtain field processing patterns includes: Determining a usage scenario according to the business information included in the field basic information and the horizontal association relationship; Classifying the fields according to the usage scenario to obtain a scenario field set, where the scenario field set includes fields belonging to the same usage scenario; Determining the field processing patterns in the usage scenario according to the scenario field set and the horizontal association relationship; And / or, The sub-relationship is a time-series change relationship, where the time-series change relationship is used to describe the change process of the field over time, and the analyzing the field basic information and the sub-relationships to obtain field processing patterns includes: Based on the field basic information and the time-series change relationship, determining the field processing patterns for different time periods.

5. The method according to claim 1, characterized in that, The method further includes: Obtain first evaluation information, where the first evaluation information includes one or more of the usage frequency, success rate, general value, innovation value, and business value of the field processing mode. The success rate is used to measure the proportion of historical successful uses of the field processing mode. The general value is used to evaluate the scope of business scenarios applicable to the field processing mode. The innovation value is used to evaluate the novelty degree of the field processing mode. The business value is used to evaluate the impact degree of the field processing mode on the business; Determine the static quality value of the field processing mode according to the first evaluation information. The static quality value is used to measure the quality of the field processing mode and is used to assist in querying the field processing mode; and / or, Obtain second evaluation information, where the second evaluation information includes one or more of the static quality value, usage frequency, success rate, and citation times of the field processing mode. The citation times are the number of times the field processing mode is recommended; Determine the dynamic quality value of the field processing mode according to the second evaluation information. The dynamic quality value is used to measure the quality of the field processing mode after changing over time and is used to assist in querying the field processing mode.

6. The method according to any one of claims 1-5, characterized in that, The extraction of field processing knowledge based on the field basic information and the field blood relationship is executed by calling a large language model.

7. The method according to any one of claims 1-5, characterized in that The method further includes: If an update condition is satisfied, update the field processing knowledge included in the field processing knowledge base. The update condition includes one or more of the following: Identifying new field processing knowledge, obtaining the abnormal inspection result for the field processing knowledge base, obtaining the field processing knowledge provided by the user, and obtaining the feedback result for the recommended field processing mode.

8. An auxiliary method for field processing, characterized in that, The method includes: Obtain a query problem for field processing, where the query problem is used to describe the field processing requirement; Generate a query result corresponding to the field processing requirement according to the field processing knowledge included in the field processing knowledge base and the query problem. The field processing knowledge base is generated according to the field processing knowledge base construction method described in any one of claims 1-7. The field processing knowledge is used to describe the field processing mode. The query result includes the information of the field processing mode and is used to assist in field processing; Display the query result.

9. The method according to claim 8, characterized in that, The generating a query result corresponding to the field processing requirement according to the field processing knowledge included in the field processing knowledge base and the query problem includes: Analyze the query problem to determine the field processing requirement; Query the target field processing mode that matches the field processing requirement in the field processing knowledge base; Generate a query result according to the field processing knowledge of the target field processing mode.

10. The method according to claim 9, wherein The analyzing the query problem to determine the field processing requirement includes: Analyze the query problem to determine the field processing task and the usage purpose of the target field. The target field is the field included in the query problem; Determine the field processing requirements according to the field processing task and the usage purpose of the target field.

11. The method according to claim 9, wherein The querying of the target field processing mode that matches the field processing requirements includes: Querying candidate field processing modes that match the field processing requirements; Taking the candidate field processing modes that meet the selection conditions under the selection dimension as the target field processing mode, where the selection dimension includes one or more of the historical common dimension, business scenario dimension, association dimension, and preference dimension.

12. The method according to claim 8, wherein The generating of the query result corresponding to the field processing requirements according to the field processing knowledge included in the field processing knowledge base and the query question includes: Obtaining a field knowledge graph, which is constructed according to the field blood relationship, where the field blood relationship is used to describe the association relationship between fields, and the field knowledge graph includes fields and the association relationship between fields; Generating a query result corresponding to the field processing requirements according to the field knowledge graph, the field processing knowledge base, and the query question.

13. The method according to claim 8, wherein The obtaining of the query question for field processing includes: Obtaining the input text of the query question, where the input text of the query question is a partial text of the query question; The method further includes: Displaying recommended supplementary text related to the input text, where the recommended supplementary text is determined based on the field processing requirements represented by the input text, and the recommended supplementary text is used, after being selected, to form a query question with the input text.

14. The method according to any one of claims 8 - 13, characterized in that, The query result includes one or more of the following types of sub-information, and the type of sub-information included in the query result is determined based on the field processing requirements: Introduction text of the field processing mode, parameters included in the field processing mode, code for implementing the field processing mode, comparison results between multiple field processing modes, and abnormal modification suggestions for abnormal field processing modes.

15. The method according to any one of claims 8-13, characterized in that, The auxiliary method for field processing is executed by calling a large language model.

16. An apparatus for constructing a field processing knowledge base, characterized in that, The device includes: A first obtaining module, configured to obtain field basic information and field blood relationship, where the field basic information is used to describe the definition and attributes of the field, and the field blood relationship is used to describe the association relationship between fields; An extraction module, configured to extract field processing knowledge based on the field basic information and the field blood relationship, where the field processing knowledge is used to describe the field processing mode; A construction module, configured to write the field processing knowledge into a field processing knowledge base, where the field processing knowledge base is used to assist field processing.

17. An auxiliary device for field processing, characterized in that, The device includes: An obtaining module, configured to obtain a query question for field processing, where the query question is used to describe the field processing requirements; A query module, configured to generate a query result corresponding to the field processing requirements according to the field processing knowledge included in the field processing knowledge base and the query question, where the field processing knowledge base is generated according to the field processing knowledge base construction method according to any one of claims 1-7, the field processing knowledge is used to describe the field processing mode, the query result includes information on the field processing mode, and the query result is used to assist field processing; A display module for displaying the query result.

18. An electronic device, characterized in that, Comprising: One or more processors; A storage device having stored thereon one or more programs, which when executed by the one or more processors cause the one or more processors to implement the method according to any one of claims 1-7, or to implement the method according to any one of claims 8-15.

19. A computer-readable medium, characterized in that, Having stored thereon a computer program, wherein when the program is executed by a processor, the method according to any one of claims 1-7 is implemented, or the method according to any one of claims 8-15 is implemented.

Images