Data consanguinity link determination method and device based on large model

Abstract

The invention discloses a data consanguinity link determination method and device based on a large model, relates to the technical field of big data processing, and mainly aims to solve the problems of poor accuracy and low efficiency of existing data consanguinity link generation. Comprising the steps that a knowledge base is called, and the knowledge base comprises technical knowledge information and business knowledge information; a knowledge object is obtained from the knowledge base according to a preset verification strategy, the knowledge object is analyzed based on a large model, a blood relationship link analysis result is obtained, and the knowledge object is at least one piece of knowledge key information formed based on the technical knowledge and / or the business knowledge information; and performing service verification on the blood relationship link analysis result, and generating a data blood relationship link based on a verification result.

Description

A method and apparatus for determining data lineage based on a large model Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a method and apparatus for determining data lineage links based on a large model. Background Technology

[0002] With the rapid rise of large-scale models, more and more data governance scenarios are being implemented based on large-scale models, such as the construction of data lineage, the impact analysis of changes based on data lineage, and question-answering reasoning based on data lineage.

[0003] Currently, the determination of existing data lineage is typically based on parsing tools. For example, metadata collection tools are used to collect metadata of table, view, and other types into a database, followed by full lineage parsing. The lineage is then formed using ETL parsing tools or custom-developed parsing logic to meet the needs of different business application scenarios. However, using ETL parsing tools or custom-developed parsing logic requires a large amount of customized code development, significantly increasing human resources and time consumption. Furthermore, the processing logic increases with the complexity of the scenario. Therefore, there is an urgent need for a data lineage determination method based on a large model to solve the aforementioned technical problems. Summary of the Invention

[0004] In view of this, this application provides a method and apparatus for determining data lineage links based on a large model, the main purpose of which is to solve the problems of poor accuracy and low efficiency in the existing data lineage link generation.

[0005] According to one aspect of this application, a method for determining data lineage links based on a large model is provided, comprising: retrieving a knowledge base, the knowledge base including technical knowledge information and business knowledge information; obtaining knowledge objects from the knowledge base according to a preset verification strategy, and parsing the knowledge objects based on a large model to obtain lineage link parsing results, wherein the knowledge objects are at least one key knowledge information composed of the technical knowledge and / or the business knowledge information; performing business verification on the lineage link parsing results, and generating data lineage links based on the verification results.

[0006] Furthermore, before retrieving the knowledge base, the method further includes: obtaining the association between the technical knowledge information and the business knowledge information based on business technical text; constructing the context content of the technical knowledge information and the business knowledge information; adding business scenario information of the technical knowledge information and the business knowledge information; constructing the knowledge base through the association, the context content, and the business scenario information, and storing it.

[0007] Furthermore, the preset verification strategy includes a single-dimensional verification strategy and a multi-dimensional verification strategy. Obtaining knowledge objects from the knowledge base according to the preset verification strategy includes: sequentially selecting single technical knowledge information or single business knowledge information from the knowledge base according to the single strategy, and generating key knowledge information based on the relationships, contextual content, and business scenario information corresponding to the selected single technical knowledge information or single business knowledge information; and sequentially selecting at least one of multiple technical knowledge information, multiple business knowledge information, or technical-business combination information from the knowledge base according to the multi-dimensional verification strategy, and generating key knowledge information based on the relationships, contextual content, and business scenario information corresponding to at least one of the selected multiple technical knowledge information, multiple business knowledge information, or technical-business combination information.

[0008] Furthermore, the step of parsing the knowledge object based on the large model to obtain the lineage link parsing result includes: generating a question-and-answer statement that matches the knowledge object according to the key information of the knowledge; using the question-and-answer statement as model input, parsing it through the large model to obtain the lineage link parsing result; wherein, the lineage link parsing result includes any one of the following: text parsing result, image parsing result, table parsing result, and code parsing result.

[0009] Further, the step of generating a question-and-answer statement matching the knowledge object based on the key knowledge information includes: when the key knowledge information is a code-related keyword, obtaining a code-related question template and generating a code-related question-and-answer statement based on the code-related keyword and the code-related question template; when the key knowledge information is a text-related keyword, obtaining a text-related question template and generating a text-related question-and-answer statement based on the text-related keyword and the text-related question template; when the key knowledge information is a character-related keyword, obtaining an image-related question template and generating an image-related question-and-answer statement based on the character-related keyword and the image-related question template; when the key knowledge information is a table-related keyword, obtaining a table-related question template and generating a table-related question-and-answer statement based on the table-related keyword and the table-related question template.

[0010] Furthermore, the business verification of the lineage link parsing result includes: outputting the lineage link parsing result to instruct the user to perform business verification; receiving the verification result entered in the lineage link parsing result, the verification result including a valid parsing result or an invalid parsing result; the generation of the data lineage link based on the verification result includes: when the verification result is a valid parsing result, generating the data lineage link and outputting it.

[0011] Furthermore, the method further includes: when the verification result is an invalid parsing result, obtaining the updated lineage link information of the invalid parsing result, and updating and training the large model based on the updated lineage link information.

[0012] According to another aspect of this application, a data lineage determination device based on a large model is provided, comprising: a retrieval module for retrieving a knowledge base, the knowledge base including technical knowledge information and business knowledge information; a parsing module for obtaining knowledge objects from the knowledge base according to a preset verification strategy, and parsing the knowledge objects based on a large model to obtain a lineage resolution result, wherein the knowledge objects are at least one key knowledge information composed of the technical knowledge and / or the business knowledge information; and a generation module for performing business verification on the lineage resolution result, and generating a data lineage based on the verification result.

[0013] Furthermore, the retrieval module is specifically used to obtain the association relationship between the technical knowledge information and the business knowledge information based on the business technical text; construct the context content of the technical knowledge information and the business knowledge information; add business scenario information of the technical knowledge information and the business knowledge information; construct the knowledge base through the association relationship, the context content and the business scenario information, and store it.

[0014] Furthermore, the preset verification strategy includes a single-dimensional verification strategy and a multi-dimensional verification strategy. The parsing module is specifically used to sequentially select single technical knowledge information or single business knowledge information from the knowledge base according to the single strategy, and generate key knowledge information based on the correlation, context content, and business scenario information corresponding to the selected single technical knowledge information or single business knowledge information; according to the multi-dimensional verification strategy, it sequentially selects at least one of multiple technical knowledge information, multiple business knowledge information, or technical-business combination information from the knowledge base, and generates key knowledge information based on the correlation, context content, and business scenario information corresponding to at least one of the selected multiple technical knowledge information, multiple business knowledge information, or technical-business combination information.

[0015] Furthermore, the parsing module is specifically used to generate question-and-answer statements that match the knowledge object according to the key knowledge information; based on the question-and-answer statements as model input, the large model is used for parsing to obtain the lineage link parsing result; wherein the lineage link parsing result includes any one of the following: text parsing result, image parsing result, table parsing result, and code parsing result.

[0016] Furthermore, the parsing module is specifically configured to: when the key knowledge information is a code-related keyword, obtain a code question template and generate a code-related question-and-answer statement based on the code-related keyword and the code question template; when the key knowledge information is a text-related keyword, obtain a text question template and generate a text-related question-and-answer statement based on the text-related keyword and the text question template; when the key knowledge information is a character-related keyword, obtain an image question template and generate an image-related question-and-answer statement based on the character-related keyword and the image question template; when the key knowledge information is a table-related keyword, obtain a table question template and generate a table-related question-and-answer statement based on the table-related keyword and the table question template.

[0017] Furthermore, the generation module is specifically used to output the lineage link parsing result to instruct the user to perform business verification; receive the verification result entered for the lineage link parsing result, the verification result including a valid parsing result or an invalid parsing result; when the verification result is a valid parsing result, the data lineage link is generated and output.

[0018] Furthermore, the device further includes an update module, configured to, when the verification result is an invalid parsing result, obtain updated lineage information of the invalid parsing result, and update and train the large model based on the updated lineage information.

[0019] According to another aspect of this application, a storage medium is provided, wherein at least one executable instruction is stored therein, the executable instruction causing a processor to perform operations corresponding to the above-described method for determining data lineage links based on a large model.

[0020] According to another aspect of this application, a terminal is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform the operation corresponding to the above-described method for determining data lineage links based on a large model.

[0021] By employing the above technical solutions, the technical solutions provided in this application have at least the following advantages: This application provides a method and apparatus for determining data lineage links based on a large model. Compared with the prior art, the embodiments of this application retrieve a knowledge base, which includes technical knowledge information and business knowledge information; obtain knowledge objects from the knowledge base according to a preset verification strategy, and parse the knowledge objects based on a large model to obtain lineage link parsing results. The knowledge objects are at least one key knowledge information composed of the technical knowledge and / or the business knowledge information; perform business verification on the lineage link parsing results, and generate data lineage links based on the verification results. Combined with the intelligent parsing of lineage links using a large model, this greatly reduces human resources and time consumption, and reduces the processing difficulty of business scenarios and business logic, thereby improving the accuracy and effectiveness of data lineage link determination.

[0022] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0023] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 shows a flowchart of a data lineage determination method based on a large model provided by an embodiment of this application; Figure 2 shows a schematic diagram of knowledge base information provided by an embodiment of this application; Figure 3 shows another schematic diagram of knowledge base information provided by an embodiment of this application; Figure 4 shows a flowchart of a large model query provided by an embodiment of this application; Figure 5 shows a block diagram of a data lineage determination device based on a large model provided by an embodiment of this application; Figure 6 shows a structural schematic diagram of a terminal provided by an embodiment of this application. Detailed Implementation

[0024] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0027] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0028] Based on this, in one embodiment, the present invention provides a method for determining data lineage links based on a large model. Taking the application of this method to computer devices such as servers as an example, the server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0029] This application provides a method for determining data lineage based on a large model, as shown in Figure 1. The method includes: 101. Retrieving a knowledge base.

[0030] In this embodiment, the current execution end, as the execution subject for determining the data lineage, can be a server or a terminal, including but not limited to terminal computers, cloud servers, etc., to retrieve the knowledge base for determining the data lineage. The knowledge base is pre-established for different business scenarios and includes technical knowledge information and business knowledge information. Technical knowledge information is used to represent knowledge content with a technical dimension, including but not limited to DOL statements, ETL processing scripts, and stored procedures. Business knowledge information is used to represent knowledge content with a business dimension, including but not limited to design schemes, summary reports, and system introductions. This embodiment does not impose specific limitations on these aspects.

[0031] It should be noted that the business knowledge information and technical knowledge information in the embodiments of this application are applicable to the fields of financial business, Internet technology, industrial production technology, etc., so as to build a knowledge base applicable to different scenarios.

[0032] 102. Obtain knowledge objects from the knowledge base according to the preset verification strategy, and parse the knowledge objects based on the large model to obtain the lineage link parsing result.

[0033] In this embodiment, the preset verification strategy includes a single-dimensional verification strategy and a multi-dimensional verification strategy. The single-dimensional verification strategy is a strategy that extracts information using one piece of knowledge, while the multi-dimensional verification strategy is a strategy that extracts information using at least two pieces of knowledge. This embodiment does not impose specific limitations on these strategies. Furthermore, the knowledge object is at least one key knowledge piece of information composed of the technical knowledge and / or the business knowledge information. It can be a single word, multiple statements, identifiers, etc., and this embodiment does not impose specific limitations on these concepts. Subsequently, a large model is used to parse the knowledge object to obtain the lineage chain parsing result.

[0034] It should be noted that the large model in the embodiments of this application is a large language model developed using open-source deployed artificial intelligence technology, such as DeepSeek-R1 (inference model), AnythingLLM (open-source knowledge base management platform), Ollam (model running platform), or a combination of various models. This application does not make any specific limitations.

[0035] 103. Perform business verification on the lineage link parsing results, and generate data lineage links based on the verification results.

[0036] In this embodiment, to ensure the correctness and validity of the lineage link resolution results, the current execution end can use verification to review the lineage link resolution results, thereby generating the final data lineage link fed back to the user based on the verification results. Business verification can be based on pre-defined words, identifiers, links, form fields, etc., in a business blacklist or whitelist, or it can be based on manual business verification methods; this embodiment does not impose specific limitations.

[0037] In another embodiment of this application, for further definition and explanation, before retrieving the knowledge base, the method further includes: obtaining the association relationship between the technical knowledge information and the business knowledge information based on business technical text; constructing the context content of the technical knowledge information and the business knowledge information; adding business scenario information of the technical knowledge information and the business knowledge information; constructing the knowledge base through the association relationship, the context content, and the business scenario information, and storing it.

[0038] To pre-build a knowledge base and combine knowledge information from it according to a strategy to resolve data lineage issues, the current execution end first obtains the association relationship between technical knowledge information and business knowledge information based on business technical texts. Here, business technical texts are business knowledge documents, such as detailed designs and preliminary designs. These documents cover specific implementation schemes, involved physical table structures, business processes, etc., and represent a knowledge body that integrates technology and business. In this case, the association relationship is used to characterize whether the technology and business are related. For example, if business knowledge information is recorded in the technical knowledge information, it indicates that there is an association relationship between the technical knowledge information and the business knowledge information. This embodiment does not impose specific limitations. Simultaneously, the current execution end constructs the contextual content of the technical knowledge information and business knowledge information. Here, the contextual content is used to characterize the language content that helps the large model better understand the business background, providing support for understanding the context, background, professional terminology, or language content of specific scenarios. The contextual content can include general business knowledge, such as system introductions, specification manuals, rules and regulations, etc. This embodiment does not impose specific limitations. For customized business applications, to accurately meet the parsing needs of large models in specific business scenarios, the current execution end adds business scenario information, including technical and business knowledge. This business scenario information can include, but is not limited to, daily business knowledge accumulation and specialized knowledge summaries, making the collection of business knowledge dimensions more flexible. This facilitates the expansion and application of large models without requiring customized development of specific business analysis logic. Finally, the knowledge base is constructed and stored using the aforementioned relationships, contextual content, and business scenario information.

[0039] In another embodiment of this application, for further definition and explanation, the step of obtaining knowledge objects from the knowledge base according to a preset verification strategy includes: sequentially selecting single technical knowledge information or single business knowledge information from the knowledge base according to the single strategy, and generating key knowledge information based on the association, context content, and business scenario information corresponding to the selected single technical knowledge information or single business knowledge information; sequentially selecting at least one of multiple technical knowledge information, multiple business knowledge information, or technical-business combination information from the knowledge base according to the multi-dimensional verification strategy, and generating key knowledge information based on the association, context content, and business scenario information corresponding to at least one of the selected multiple technical knowledge information, multiple business knowledge information, or technical-business combination information.

[0040] To improve the accuracy of data lineage path construction, the preset verification strategy includes a single-dimensional verification strategy and a multi-dimensional verification strategy. In a specific embodiment, the current execution end selects single technical knowledge information or single business knowledge information from the knowledge base according to the single strategy. That is, in the knowledge base shown in Figure 2, single technical knowledge information is selected sequentially according to the single strategy. For example, DOL statement 1 is selected as the knowledge object. After parsing DOL statement 1, design scheme 1 is obtained as the new knowledge object. This embodiment does not make specific limitations. Furthermore, based on the association, context content, and business scenario information corresponding to the selected single technical knowledge information or single business knowledge information, key knowledge information is generated. That is, the association, context content, and business scenario information corresponding to the single business knowledge information are retrieved from the knowledge base and identified as key knowledge information.

[0041] In one specific embodiment, the current execution end sequentially selects at least one of multiple technical knowledge information, multiple business knowledge information, or technical-business combination information from the knowledge base according to a multi-dimensional verification strategy. When selecting at least one of these multiple technical knowledge information, business knowledge information, or technical-business combination information, it does not simply seek a single key knowledge information containing multiple indicator information. Instead, it can add one piece of technical knowledge information or business knowledge information sequentially according to the storage order to serve as a new key knowledge information, thereby obtaining multiple key knowledge information. For example, in the knowledge base shown in Figure 3, the number of selected knowledge is then increased sequentially; this embodiment does not impose specific limitations. Furthermore, key knowledge information is generated based on the corresponding relationships, contextual content, and business scenario information of at least one of the selected multiple technical knowledge information, business knowledge information, or technical-business combination information. That is, the corresponding relationships, contextual content, and business scenario information of at least one of the multiple technical knowledge information, business knowledge information, or technical-business combination information are retrieved from the knowledge base and identified as key knowledge information.

[0042] In another embodiment of this application, for further definition and explanation, the step of parsing the knowledge object based on a large model to obtain the lineage link parsing result includes: generating a question-and-answer statement that matches the knowledge object according to the key knowledge information; and using the question-and-answer statement as model input, parsing it through the large model to obtain the lineage link parsing result.

[0043] To leverage the large model for lineage analysis of knowledge objects, the current execution end first generates question-and-answer statements matching the knowledge objects based on key knowledge information. This involves combining or converting key knowledge information into question statements suitable for the large language model, enabling it to understand the information. The lineage analysis results can be presented in various formats, including text, image, table, and code. This demonstrates that the output lineage analysis results can be displayed in multiple ways after the large model queries the questions based on these statements. For example, for querying key knowledge information documents, the question-and-answer statements could be something like, "Could you help me find the person in charge and their contact information for the xx system?" or "Could you please introduce the xx system and its implementation?", and the large model can be directly invoked to ask the question.

[0044] In a specific embodiment, the question-and-answer statement could be "Help me analyze the lineage at the TEMP_DATE field level." After the large model is parsed, a lineage parsing result is generated, including: "The field-level lineage of the TEMP_DATE table is as follows: LASE_MONTH_TRD_DATE; Source: The maximum value of PHYDATE is obtained from the source table and generated after date formatting; Dependency: Depends on the PHYDATE field in the source table; Purpose: Used for subsequent date condition judgments." At this time, the lineage parsing result is a text-based parsing result.

[0045] In another embodiment of this application, for further definition and explanation, the step of generating a question-and-answer statement matching the knowledge object according to the key knowledge information includes: when the key knowledge information is a code-related keyword, obtaining a code question template, and generating a code-related question-and-answer statement based on the code-related keyword and the code question template; when the key knowledge information is a text-related keyword, obtaining a text question template, and generating a text-related question-and-answer statement based on the text-related keyword and the text question template; when the key knowledge information is a character-related keyword, obtaining an image question template, and generating an image-related question-and-answer statement based on the character-related keyword and the image question template; when the key knowledge information is a table-related keyword, obtaining a table question template, and generating a table-related question-and-answer statement based on the table-related keyword and the table question template.

[0046] To enable rapid application execution based on large models for business scenarios, different types of question-and-answer statements can be generated for complex question-and-answer statements, tailored to different question models. Specifically, in this embodiment, commonly used question-and-answer statements are formed into fixed syntax templates, which may include code question templates, text question templates, image question templates, and table question templates, to correspondingly generate code-based, text-based, image-based, and table-based question-and-answer statements. Furthermore, this application can employ different settings for different types of question templates to meet the needs of different language habits and dialect usage.

[0047] In some embodiments, for the example of metadata change impact analysis, the question-and-answer form template may include: 1. "Help me analyze the field-level lineage of table xxx and generate a form," 2. "Based on the lineage form generated above, which systems will be affected by changes to table xxx?" etc. Text-based question templates may include "Please help me find information and contact details of personnel affected by XXX?" etc., and this embodiment does not impose specific limitations. Furthermore, different question-and-answer templates can be combined or asked sequentially, as shown in Figure 4, and this embodiment does not impose specific limitations.

[0048] In another embodiment of this application, for further definition and explanation, the step of performing business verification on the lineage link parsing result includes: outputting the lineage link parsing result to instruct the user to perform business verification; receiving the verification result entered on the lineage link parsing result; the step of generating a data lineage link based on the verification result includes: when the verification result is a valid parsing result, generating the data lineage link and outputting it.

[0049] To ensure the accuracy of the lineage link resolution results, the current execution terminal preferably performs manual verification during business verification. This involves outputting the lineage link resolution results to instruct the user to perform business verification. Subsequently, after verification by technical personnel, a verification result is generated. The current execution terminal receives the verification result entered in the lineage link resolution results. At this point, the verification result includes either a valid or invalid resolution result. If the verification result is valid, the data lineage link is generated and output.

[0050] In another embodiment of this application, for further definition and explanation, the steps further include: when the verification result is an invalid parsing result, obtaining the updated lineage link information of the invalid parsing result, and updating and training the large model based on the updated lineage link information.

[0051] To ensure the effectiveness of the large model's parsing, when the verification result is invalid, it indicates that there is an error in the large model's parsing. Therefore, the current execution end obtains the updated lineage information of the invalid parsing result and updates and trains the large model based on the updated lineage information, thereby improving the parsing accuracy of the large model for business scenarios.

[0052] This application provides a method for determining data lineage links based on a large model. Compared with existing technologies, this application retrieves a knowledge base, which includes technical knowledge information and business knowledge information; obtains knowledge objects from the knowledge base according to a preset verification strategy, and parses the knowledge objects based on the large model to obtain lineage link parsing results. The knowledge objects are at least one key knowledge information composed of the technical knowledge and / or the business knowledge information; performs business verification on the lineage link parsing results, and generates data lineage links based on the verification results. By combining the intelligent parsing of lineage links with the large model, the method greatly reduces human resources and time consumption, and reduces the processing difficulty of business scenarios and business logic, thereby improving the accuracy and effectiveness of data lineage link determination.

[0053] Furthermore, as an implementation of the method shown in Figure 1 above, this application embodiment provides a data lineage determination device based on a large model, as shown in Figure 5. The device includes: a retrieval module 21, used to retrieve a knowledge base, the knowledge base including technical knowledge information and business knowledge information; a parsing module 22, used to obtain knowledge objects from the knowledge base according to a preset verification strategy, and parse the knowledge objects based on the large model to obtain lineage resolution results, wherein the knowledge objects are at least one key knowledge information composed of the technical knowledge and / or the business knowledge information; and a generation module 23, used to perform business verification on the lineage resolution results, and generate data lineages based on the verification results.

[0054] Furthermore, the retrieval module is specifically used to obtain the association relationship between the technical knowledge information and the business knowledge information based on the business technical text; construct the context content of the technical knowledge information and the business knowledge information; add business scenario information of the technical knowledge information and the business knowledge information; construct the knowledge base through the association relationship, the context content and the business scenario information, and store it.

[0055] Furthermore, the preset verification strategy includes a single-dimensional verification strategy and a multi-dimensional verification strategy. The parsing module is specifically used to sequentially select single technical knowledge information or single business knowledge information from the knowledge base according to the single strategy, and generate key knowledge information based on the correlation, context content, and business scenario information corresponding to the selected single technical knowledge information or single business knowledge information; according to the multi-dimensional verification strategy, it sequentially selects at least one of multiple technical knowledge information, multiple business knowledge information, or technical-business combination information from the knowledge base, and generates key knowledge information based on the correlation, context content, and business scenario information corresponding to at least one of the selected multiple technical knowledge information, multiple business knowledge information, or technical-business combination information.

[0056] Furthermore, the parsing module is specifically used to generate question-and-answer statements that match the knowledge object according to the key knowledge information; based on the question-and-answer statements as model input, the large model is used for parsing to obtain the lineage link parsing result; wherein the lineage link parsing result includes any one of the following: text parsing result, image parsing result, table parsing result, and code parsing result.

[0057] Furthermore, the parsing module is specifically configured to: when the key knowledge information is a code-related keyword, obtain a code question template and generate a code-related question-and-answer statement based on the code-related keyword and the code question template; when the key knowledge information is a text-related keyword, obtain a text question template and generate a text-related question-and-answer statement based on the text-related keyword and the text question template; when the key knowledge information is a character-related keyword, obtain an image question template and generate an image-related question-and-answer statement based on the character-related keyword and the image question template; when the key knowledge information is a table-related keyword, obtain a table question template and generate a table-related question-and-answer statement based on the table-related keyword and the table question template.

[0058] Furthermore, the generation module is specifically used to output the lineage link parsing result to instruct the user to perform business verification; receive the verification result entered for the lineage link parsing result, the verification result including a valid parsing result or an invalid parsing result; when the verification result is a valid parsing result, the data lineage link is generated and output.

[0059] Furthermore, the device further includes an update module, configured to, when the verification result is an invalid parsing result, obtain updated lineage information of the invalid parsing result, and update and train the large model based on the updated lineage information.

[0060] This application provides a data lineage determination device based on a large model. Compared with the prior art, this application retrieves a knowledge base, which includes technical knowledge information and business knowledge information; obtains knowledge objects from the knowledge base according to a preset verification strategy, and parses the knowledge objects based on a large model to obtain lineage determination results. The knowledge objects are at least one key knowledge information composed of the technical knowledge and / or the business knowledge information; performs business verification on the lineage determination results, and generates data lineages based on the verification results. Combined with the intelligent analysis of lineages using a large model, this significantly reduces human resources and time consumption, and lowers the processing difficulty of business scenarios and business logic, thereby improving the accuracy and effectiveness of data lineage determination. According to one embodiment of this application, a storage medium is provided, which stores at least one executable instruction that can execute the data lineage determination method based on a large model in any of the above method embodiments.

[0061] Figure 6 shows a schematic diagram of a terminal according to an embodiment of this application. The specific implementation of the terminal is not limited by the specific embodiments of this application.

[0062] As shown in Figure 6, the terminal may include: a processor 302, a communications interface 304, a memory 306, and a communication bus 308.

[0063] The processor 302, communication interface 304, and memory 306 communicate with each other via communication bus 308.

[0064] Communication interface 304 is used to communicate with other network elements such as clients or other servers.

[0065] The processor 302 is used to execute program 310, which can specifically execute the relevant steps in the above-described embodiment of the data lineage determination method based on a large model.

[0066] Specifically, program 310 may include program code that includes computer operation instructions.

[0067] Processor 302 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The terminal includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0068] Memory 306 is used to store program 310. Memory 306 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0069] Specifically, program 310 can be used to cause processor 302 to perform the following operations: retrieve a knowledge base, which includes technical knowledge information and business knowledge information; obtain knowledge objects from the knowledge base according to a preset verification strategy, and parse the knowledge objects based on a large model to obtain a lineage link parsing result, wherein the knowledge object is at least one key knowledge information composed of the technical knowledge and / or the business knowledge information; perform business verification on the lineage link parsing result, and generate a data lineage link based on the verification result.

[0070] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0071] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for determining data lineage based on a large model, characterized in that, include: Retrieve a knowledge base, which includes technical knowledge information and business knowledge information; According to the preset verification strategy, knowledge objects are obtained from the knowledge base, and the knowledge objects are parsed based on the large model to obtain the lineage link parsing result. The knowledge object is at least one key knowledge information composed of the technical knowledge and / or the business knowledge information. The lineage link parsing results are verified for business purposes, and a data lineage link is generated based on the verification results.

2. The method according to claim 1, characterized in that, Before retrieving the knowledge base, the method further includes: obtaining the association between the technical knowledge information and the business knowledge information based on business technical text; constructing the context content of the technical knowledge information and the business knowledge information; adding business scenario information of the technical knowledge information and the business knowledge information; constructing the knowledge base through the association, the context content, and the business scenario information, and storing it.

3. The method according to claim 2, characterized in that, The preset verification strategy includes a single-dimensional verification strategy and a multi-dimensional verification strategy. Obtaining knowledge objects from the knowledge base according to the preset verification strategy includes: sequentially selecting single technical knowledge information or single business knowledge information from the knowledge base according to the single strategy, and generating key knowledge information based on the relationships, contextual content, and business scenario information corresponding to the selected single technical knowledge information or single business knowledge information; and sequentially selecting at least one of multiple technical knowledge information, multiple business knowledge information, or technical-business combination information from the knowledge base according to the multi-dimensional verification strategy, and generating key knowledge information based on the relationships, contextual content, and business scenario information corresponding to at least one of the selected multiple technical knowledge information, multiple business knowledge information, or technical-business combination information.

4. The method according to claim 3, characterized in that, The step of parsing the knowledge object based on the large model to obtain the lineage link parsing result includes: generating a question-and-answer statement that matches the knowledge object according to the key information of the knowledge; using the question-and-answer statement as model input, parsing it through the large model to obtain the lineage link parsing result; wherein, the lineage link parsing result includes any one of the following: text parsing result, image parsing result, table parsing result, and code parsing result.

5. The method according to claim 4, characterized in that, The step of generating a question-and-answer statement matching the knowledge object based on the key knowledge information includes: when the key knowledge information is a code-related keyword, obtaining a code-related question template and generating a code-related question-and-answer statement based on the code-related keyword and the code-related question template; when the key knowledge information is a text-related keyword, obtaining a text-related question template and generating a text-related question-and-answer statement based on the text-related keyword and the text-related question template; when the key knowledge information is a character-related keyword, obtaining an image-related question template and generating an image-related question-and-answer statement based on the character-related keyword and the image-related question template; when the key knowledge information is a table-related keyword, obtaining a table-related question template and generating a table-related question-and-answer statement based on the table-related keyword and the table-related question template.

6. The method according to any one of claims 1-5, characterized in that, The business verification of the lineage link parsing result includes: outputting the lineage link parsing result to instruct the user to perform business verification; receiving the verification result entered for the lineage link parsing result, the verification result including a valid parsing result or an invalid parsing result; the generation of the data lineage link based on the verification result includes: when the verification result is a valid parsing result, generating the data lineage link and outputting it.

7. The method according to claim 6, characterized in that, The method further includes: when the verification result is an invalid parsing result, obtaining the updated lineage link information of the invalid parsing result, and updating and training the large model based on the updated lineage link information.

8. A device for determining data lineage based on a large model, characterized in that, include: The retrieval module is used to retrieve a knowledge base, which includes technical knowledge information and business knowledge information. The parsing module is used to obtain knowledge objects from the knowledge base according to a preset verification strategy, and parse the knowledge objects based on the large model to obtain the lineage link parsing result. The knowledge object is at least one key knowledge information composed of the technical knowledge and / or the business knowledge information. The generation module is used to perform business verification on the lineage link parsing results and generate data lineage links based on the verification results.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 1.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 1.

Images