A data standard formulation method and device based on an artificial intelligence large model

Abstract

The application provides a data standard formulation method and device based on an artificial intelligence large model, and relates to the field of artificial intelligence. The method comprises: in response to an operation of formulating a data standard for a target project, acquiring a data set corresponding to the target project; taking a data table name and a data item name in the data set as input, performing a first data standard retrieval operation, the first data standard retrieval operation corresponding to a first retrieval result; if the data standard corresponding to the target project is not acquired through the first data standard retrieval operation, inputting the data set into a large language model to acquire a semantic description relationship in the data set; dividing a triple corresponding to the data item name according to the semantic description relationship through the large language model, and extracting a target keyword in the data set according to the triple; taking the target keyword, the data table name and the data item name as input, performing a second data standard retrieval operation on the target project, the second data standard retrieval operation corresponding to a second retrieval result; and sorting the first retrieval result and the second retrieval result through a preset sorting operation to acquire the data standard corresponding to the target project. The application solves the problem that a data standard formulation method based on rules or simple text matching is difficult to accurately identify the corresponding relationship between data items, thereby affecting the accuracy of the data standard.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method and apparatus for developing data standards based on large-scale artificial intelligence models. Background Technology

[0002] With the development of big data and artificial intelligence technologies, the formulation of data standards has become a key link in improving data quality, achieving data sharing, and cross-system interoperability.

[0003] Traditional methods for setting data standards mainly rely on manually defined rules or simple keyword matching, which has significant limitations. For example, in industry databases or business systems, data item names may have synonyms, abbreviations, or different expressions, making it difficult for rule-based or simple text matching-based data standard setting methods to accurately identify the correspondence between data items, thus affecting the accuracy of the data standards.

[0004] Therefore, there is an urgent need for a data standard development method and apparatus based on large-scale artificial intelligence models. Summary of the Invention

[0005] This application provides a data standard setting method and apparatus based on a large artificial intelligence model, which solves the problem that data standard setting methods based on rules or simple text matching are difficult to accurately identify the correspondence between data items, thus affecting the accuracy of the data standard.

[0006] The first aspect of this application provides a data standard formulation method based on a large artificial intelligence model. The method includes: in response to an operation of formulating data standards for a target project, obtaining a dataset corresponding to the target project; performing a first data standard retrieval operation using the data table name and data item name in the dataset as input, the first data standard retrieval operation corresponding to a first retrieval result; if the data standard corresponding to the target project is not obtained through the first data standard retrieval operation, inputting the dataset into a large language model to obtain semantic description relationships in the dataset; using the large language model, dividing the data item name into triples according to the semantic description relationships, and extracting target keywords from the dataset based on the triples; performing a second data standard retrieval operation on the target project using the target keywords, data table name, and data item name as input, the second data standard retrieval operation corresponding to a second retrieval result; and sorting the first retrieval result and the second retrieval result through a preset sorting operation to obtain the data standard corresponding to the target project.

[0007] Optionally, the dataset corresponding to the target project is obtained, specifically including: dividing data into data classes, data subclasses, and data items according to the business links and business processes corresponding to the target project, and constructing a data subset through data classes, data subclasses, and data items; and using the collection of data subsets as the dataset corresponding to the target project.

[0008] Optionally, the dataset is input into a large language model to obtain the semantic description relationship of the data item names in the dataset. Specifically, this includes: using the large language model to parse the hierarchical structure of the dataset in a progressive manner, the hierarchical structure including dataset name, data subset name, data class name, data table name, and data item name; segmenting the data item names into words, and ranking the data item names according to the accuracy of the word segmentation results based on their positional relationship in the hierarchical structure; and using the ranking result as the semantic description relationship output by the large language model.

[0009] Optionally, using a large language model, the data item names are divided into triples based on semantic description relationships, and target keywords in the dataset are extracted based on the triples. Specifically, this includes: using a large language model, the data item names are divided into triples based on semantic description relationships, where each triple includes the subject, object, and relationship; and target keywords in the dataset are extracted based on the triples.

[0010] Optionally, the second data standard retrieval operation includes a first search operation and a second search operation. Obtaining the second search result corresponding to the second data standard retrieval operation specifically includes: obtaining the first matching document corresponding to the first search operation and obtaining the second matching document corresponding to the second search operation; and using the first matching document and the second matching document as the second search result.

[0011] Optionally, obtaining the first matching document corresponding to the first search operation specifically includes: using the target keyword as a query condition, performing a full-text search on the original documents in the target project to obtain multiple first query documents; calculating the relevance score between each first query document and the original document, and sorting the multiple relevance scores; and obtaining the first query document that meets the first preset sorting threshold from the sorted results as the first matching document.

[0012] Optionally, obtaining the second matching document corresponding to the second search operation specifically includes: mapping the target keywords to query vectors using a preset text vectorization model; constructing a vector database of the target project based on the HNSW index, the vector database being used to store the correspondence between multiple second query documents and multiple document vectors, with one document vector corresponding to one second query document; calculating multiple similarity values ​​between the query vector and multiple document vectors in the vector database, with one document vector corresponding to one similarity value, and sorting the multiple similarity values; obtaining the target document vector corresponding to the similarity value that meets the second preset sorting threshold in the sorting results, and using the second query document corresponding to the target document vector as the second matching document.

[0013] Optionally, the first and second search results are sorted using a preset sorting operation to obtain the data standard corresponding to the target item. Specifically, this includes: performing a first sort on the first and second search results by weighted summation; obtaining the third matching document in the first sorting results that meets a third preset sorting threshold, and performing a second sort on the third matching document using a re-sorting model; obtaining the fourth matching document in the second sorting results that meets a fourth preset sorting threshold; calculating the second document score corresponding to the fourth matching document, and determining whether the second document score is greater than or equal to the expected threshold; if the second document score is greater than or equal to the expected threshold, then the data corresponding to the fourth matching document is used as the data standard.

[0014] A second aspect of this application provides a data standard setting apparatus based on a large artificial intelligence model. The apparatus includes an acquisition module and a processing module, wherein...

[0015] The acquisition module is used to respond to the operation of defining data standards for the target project and obtain the dataset corresponding to the target project. It takes the data table name and data item name in the dataset as input and performs the first data standard retrieval operation, which corresponds to the first retrieval result. If the data standard corresponding to the target project is not obtained through the first data standard retrieval operation, the dataset is input into a large language model to obtain the semantic description relationship in the dataset. Through the large language model, the data item name is divided into triples according to the semantic description relationship, and the target keywords in the dataset are extracted from the dataset based on the triples.

[0016] The processing module is used to perform a second data standard retrieval operation on the target item using the target keyword, data table name, and data item name as input. The second data standard retrieval operation corresponds to the second retrieval result. The first retrieval result and the second retrieval result are sorted by a preset sorting operation to obtain the data standard corresponding to the target item.

[0017] A third aspect of this application provides an electronic device including a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform any of the methods described above.

[0018] A fourth aspect of this application provides a computer-readable storage medium storing a computer program, which is executed by a processor using the method described in any of the foregoing descriptions.

[0019] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0020] 1. Obtain the dataset corresponding to the target project; using the data table name and data item name in the dataset as input, perform a first data standard retrieval operation, which corresponds to a first retrieval result; if the data standard corresponding to the target project is not obtained through the first data standard retrieval operation, input the dataset into a large language model to obtain the semantic description relationship in the dataset; using the large language model, divide the data item name into triples according to the semantic description relationship, and extract the target keywords in the dataset based on the triples; using the target keywords, the data table name, and the data item name as input, perform a second data standard retrieval operation on the target project, which corresponds to a second retrieval result; sort the first retrieval result and the second retrieval result through a preset sorting operation to obtain the data standard corresponding to the target project. Thus, by combining the dual recall mechanism of full-text search and vector search, semantic parsing and keyword extraction based on the large language model, and multi-level sorting optimization strategy, the accuracy of the data standard formulation process is significantly improved.

[0021] 2. Utilizing a large language model, the hierarchical structure of the dataset is parsed in a progressive manner, and data item names are segmented. Based on the positional relationship of the data item names within the hierarchical structure, the data item names are ranked according to the accuracy of the segmentation results. This ranking is used as the semantic description relationship output by the large language model. Furthermore, based on the hierarchical structure and semantic information of the dataset, the semantic description relationship of the data item names is accurately parsed, thereby improving the standardization of data item names.

[0022] 3. Using a large language model, the data item names are divided into triples based on semantic description relationships. Each triple includes the subject, object, and relation. Target keywords are extracted from the dataset based on the triples. This allows for in-depth analysis of the semantic structure of the data items using the large language model, thereby generating high-quality target keywords and providing accurate query conditions for subsequent full-text and vector searches. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a method for developing data standards based on a large artificial intelligence model, as provided in an embodiment of this application.

[0024] Figure 2 This is a schematic diagram of a target keyword extraction process provided in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of a data retrieval and sorting process according to an embodiment of this application;

[0026] Figure 4This is a schematic diagram of a data standard setting process provided in an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of a data standard setting device based on a large artificial intelligence model provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0029] Explanation of reference numerals in the attached figures: 51, acquisition module; 52, processing module; 601, processor; 602, communication bus; 603, user interface; 604, network interface; 605, memory. Detailed Implementation

[0030] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0031] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0032] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0033] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0034] Please refer to Figure 1 The diagram illustrates a flowchart of a data standard formulation method based on a large artificial intelligence model provided in this application embodiment. The flowchart mainly includes the following steps: S101 to S106.

[0035] Step S101: In response to the operation of defining data standards for the target project, obtain the dataset corresponding to the target project.

[0036] Specifically, when a user initiates a data standard setting request for a specific target project, the first step is to determine the type of the target project. For example, a target project could be a company's business system, an industry database, a scientific research dataset, a government open dataset, or a manufacturing product information database. Different types of target projects involve different data structures and data sources. Based on the type of the target project, and according to pre-defined business rules, datasets related to that target project are filtered and obtained.

[0037] In one possible implementation, step S101 further includes: dividing data into data classes, data subclasses, and data items according to the business links and business processes corresponding to the target project, and constructing a data subset through the data classes, data subclasses, and data items; and using the collection of data subsets as the dataset corresponding to the target project.

[0038] Specifically, data is organized hierarchically according to the business links and processes corresponding to the target project, establishing a multi-level data structure of data classes, data subclasses, and data items. Business links are typically a series of logically related steps in the operation of the target project. For example, in an education management system, business links may include enrollment management, student status management, course management, and grade management, while business processes are the specific operational sequence of business links. For instance, the grade management link may involve operations such as grade entry, grade review, and grade publication.

[0039] Next, the system constructs data subsets through data classes, data subclasses, and data items. Data classes represent a broad category of data, such as "student registration data," data subclasses refine data types, such as "grade data," and data items represent specific data fields, such as "course grades." These hierarchical levels form a progressively deeper semantic description relationship, allowing the semantic information of the data to unfold from top to bottom, facilitating the subsequent development of data standards using semantic parsing techniques. For example, in an education management information system, the dataset might be named "Education Management Information Dataset." This dataset contains multiple data subsets, such as the "student data subset." Within the "student data subset," data classes can be further subdivided, such as the "student registration data class." These data classes can then be further subdivided into data subclasses, such as the "grade data subclass." Finally, specific data information is defined by data items, such as the "course grade data item."

[0040] This hierarchical structure clarifies the hierarchical relationships of the data and provides a progressive semantic expression, which is beneficial for subsequent semantic description relationship analysis based on large language models. It also improves data query efficiency and the accuracy of data standard setting and matching. Furthermore, this method is applicable to different types of target projects. For example, a medical management system can be organized as "Medical Management Dataset -> Patient Data Subset -> Inpatient Data Class -> Diagnosis and Treatment Data Subclass -> Examination Report Data Items"; a financial risk control system can be organized as "Financial Risk Control Dataset -> Customer Data Subset -> Credit Assessment Data Class -> Transaction Record Data Subclass -> Transaction Amount Data Items"; and an education management system can be organized as "Education Management Information Dataset -> Student Data Subset -> Student Status Data Class -> Grade Data Subclass -> Course Grade Data Items," thus ensuring the universality and flexibility of the data structure.

[0041] Step S102: Using the data table name and data item name in the dataset as input, perform the first data standard retrieval operation. The first data standard retrieval operation corresponds to the first retrieval result.

[0042] Specifically, the first data standard retrieval operation is to perform a full-text search on the target item to obtain matching data standards. At this time, the retrieval results of the first data standard retrieval operation include multiple first matching documents.

[0043] In one possible implementation, step S102 further includes: performing a full-text search on the original documents in the target project through a first data standard retrieval operation to obtain multiple first query documents; calculating the relevance score between each first query document and the original document, and sorting the multiple relevance scores; and obtaining the first query document that meets the first preset sorting threshold from the sorting results as the first matching document.

[0044] Specifically, a full-text search is performed on the original document database of the target project, combining the data table names and data item names. The BM25 algorithm is used to calculate the relevance score Score(D,Q) between document D (the original document in the database) and query Q (composed of target keywords, data table names, and data item names), to measure the degree of matching between document D and query Q. The calculation formula is as follows:

[0045]

[0046] Where, q i Let f(q) be the i-th word in Q, i.e., a single term in the target keyword. i D) is the word q iThe frequency of a term in document D indicates its importance; higher frequency indicates greater importance. |D| is the total length of document D (i.e., the total number of words). Longer documents may contain more matches. avgdl is the average length of all documents in the database, ensuring comparability of match rates across documents of different lengths. k and b are adjustable parameters, k = 1.5k and b = 0.75, where k controls the impact of term frequency on the score, and b controls the impact of document length on the score. IDF(q i ) is the word q i The inverse document frequency (IVF) is used to measure the uniqueness of a word, and is calculated using the following formula:

[0047]

[0048] Where N is the total number of documents in the database, n(q i ) is a word containing q i The number of documents. Calculate the relevance score of all first query documents and sort the search results in descending order according to Score(D,Q). Set a first preset sorting threshold T1, filter out all first query documents with a relevance score greater than or equal to T1, and use them as the first matching documents.

[0049] Step S103: If the data standard corresponding to the target project is not obtained through the first data standard retrieval operation, the dataset is input into the large language model to obtain the semantic description relationship in the dataset.

[0050] Specifically, if the first matching document is the data standard corresponding to the target project, the data standard setting is complete, and the data standard setting operation ends directly. If the first matching document is not the data standard corresponding to the target project, the dataset is input into the large language model, which then obtains the semantic description relationships of the data item names in the dataset. Please refer to [reference needed]. Figure 2 It presents a schematic diagram of a target keyword extraction process provided in an embodiment of this application. Figure 2 The specific process is carried out according to the following steps.

[0051] In one possible implementation, step S103 further includes: using a large language model to parse the hierarchical structure of the dataset in a progressive manner, the hierarchical structure including dataset name, data subset name, data class name, data table name, and data item name; segmenting the data item names into words, and ranking the data item names according to the accuracy of the word segmentation results based on their positional relationship in the hierarchical structure; and using the ranking results as the semantic description relationship output by the large language model.

[0052] Specifically, by leveraging the hierarchical relationships between dataset names, data subset names, data class names, data table names, and data item names, a progressively advancing semantic description system is established to clarify the position and semantic role of data item names within the dataset structure. The hierarchical organization of the dataset is analyzed, where the dataset contains several data subsets, data subsets contain several data classes, data classes contain several data tables, data tables contain several data items, and data items describe the attribute information of the data tables and semantically inherit features from their parent structures. Natural language processing methods are used to segment data item names, analyze their internal structure, and deconstruct their core information to obtain fine-grained semantic features. Combining the descriptive role of data item names within the data tables, the segmented data item names are ranked according to the semantic accuracy of the segmentation results, ensuring that the ranking reflects the core attribute information of the data items, facilitating subsequent data standard setting, matching, and retrieval optimization. Through these methods, the system can accurately parse the semantic description relationships of data item names based on the hierarchical structure and semantic information of the dataset, thereby improving the accuracy of data standard setting for data item names and providing optimized keyword extraction schemes for subsequent data standard setting and retrieval.

[0053] Step S104: Using a large language model, divide the data item names into triples according to semantic description relationships, and extract the target keywords from the dataset based on the triples.

[0054] Specifically, a large language model is used to parse the semantic description relationship of data item names, and the core keywords of data items are extracted based on the semantic relationship.

[0055] In one possible implementation, step S104 further includes: using a large language model to divide the data item names into triples according to semantic description relationships, the triples including the subject, object, and relationship; and extracting target keywords from the dataset based on the triples.

[0056] Specifically, the large language model analyzes the internal structure of data item names based on the semantic description relationships obtained in step S103, dividing the data item names into triples [subject, object, relation] according to the subject, object, and relation framework. Here, the subject represents the core concept of the data item, the object represents a specific attribute or entity associated with the subject, and the relation describes the semantic connection between the subject and object, such as verbs, prepositions, or implicit attribute relationships. After determining the triples, the system further extracts target keywords based on semantic information to ensure that the keywords accurately summarize the semantic features of the data item. For example, the data item name "Is a student a poor student?" can be parsed as [student, poor student, whether], where "poor student" is the core descriptive information of the data item and is extracted as the target keyword. Through this process, the large language model can deeply analyze the semantic structure of data items, thereby generating high-quality target keywords and providing accurate query conditions for subsequent full-text search and vector search.

[0057] Step S105: Using the target keywords, data table name, and data item name as input, perform a second data standard retrieval operation on the target item. The second data standard retrieval operation corresponds to the second retrieval result.

[0058] Step S106: Sort the first search result and the second search result by a preset sorting operation to obtain the data standard corresponding to the target item.

[0059] Specifically, the first matching document corresponding to the first data standard retrieval operation is obtained, and the second matching document corresponding to the second data standard retrieval operation is obtained; the first matching document and the second matching document are sorted according to a preset method, the sorting of which includes two sorting operations, in order to obtain the data standard corresponding to the target item.

[0060] In one possible implementation, step S106 further includes: mapping target keywords to query vectors using a preset text vectorization model; constructing a vector database for the target project based on the HNSW index, wherein the vector database is used to store the correspondence between multiple second query documents and multiple document vectors, with one document vector corresponding to one second query document; calculating multiple similarity values ​​between the query vector and multiple document vectors in the vector database, with one document vector corresponding to one similarity value, and sorting the multiple similarity values; obtaining the target document vector corresponding to the similarity value that meets the second preset sorting threshold in the sorting results, and using the second query document corresponding to the target document vector as the second matching document.

[0061] Specifically, the search results corresponding to the secondary data standard retrieval are obtained through the following steps: multiple second-matching documents are combined with target keywords, data table names, and data item names. A pre-defined text vectorization model (such as BERT, Sentence-BERT, etc.) is used to map the query conditions into high-dimensional vector representations. An HNSW (Hierarchical Navigable Small World) index is used to construct a vector database of the target items, and the HNSW index is used for efficient retrieval of the vectorized query conditions. The cosine similarity between the query vector and the document vectors in the target item vector database is calculated, and the search results are sorted according to the similarity score. The calculation formula is as follows:

[0062]

[0063] Among them, Score cos Let x be the cosine similarity. i Let y be the i-th query vector. i Let n be the i-th document vector in the target project vector database, and n be the dimension of the vector. A second preset ranking threshold T2 is set to filter out all first query documents with a relevance score greater than or equal to T2, and these are used as the second matching documents. The first and second preset ranking thresholds need to be set according to actual needs to ensure that the retrieval results have both high accuracy and reasonable recall. For example, the first preset ranking threshold can be calculated by analyzing the distribution of BM25 scores in an existing dataset and selecting the 95th percentile as the threshold; the second ranking threshold can be calculated by statistically analyzing the cosine similarity distribution of matching documents in the training data and taking the 90th percentile as the initial threshold. The embodiments in this application do not limit the setting of the first and second preset ranking thresholds.

[0064] The final data standard in this application needs to be obtained through two sorting processes. Please refer to [reference needed]. Figure 3 The document presents a schematic diagram of a data retrieval and sorting process according to an embodiment of this application. The sorting process includes steps S1 to S8.

[0065] Step S1: Perform a first sorting on the first matching document and the second matching document by weighted summation according to their weights.

[0066] Specifically, the weight of the Score(D,Q) in the first retrieval is w1, and the weight of the Score(D,Q) in the second retrieval is w1. cos The score weight is w2, where w1 + w2 = 1. The scores are then summed according to their weights.

[0067] Step S2: Obtain the third matching document in the first sorting result that meets the third preset sorting threshold.

[0068] Specifically, from the document collection after the first sorting, the overall score is selected. mixed Documents that are greater than or equal to the third preset sorting threshold T3 are defined as the third matching documents.

[0069] Step S3: Calculate the score of the first document corresponding to the third matching document, and determine whether the score of the first document is greater than or equal to the first expected threshold.

[0070] Specifically, for each third matching document, its first document score is calculated by combining its full-text search score and vector search similarity.

[0071] Step S4: If the score of the first document is greater than or equal to the first expected threshold, then the data corresponding to the third matching document is used as the data standard.

[0072] Specifically, the data corresponding to the d1th most matching document after sorting, that is, the data corresponding to the d1th document that satisfies the first expected threshold E1, is used as the data standard.

[0073] Step S5: If the score of the first document is less than the first expected threshold, then the first matching document and the second matching document are sorted in the second sorting by the re-sorting model.

[0074] Specifically, if the first document score of the third matching document is lower than E1, the original document is returned and a preset re-ranking model (such as the bge-reranker-base model) is used to perform a second ranking on the first matching document and the second matching document.

[0075] Step S6: Obtain the fourth matching document in the second sorting result that meets the fourth sorting threshold.

[0076] Specifically, from the second sorting results, the fourth matching document that satisfies the fourth sorting threshold T4.

[0077] Step S7: Calculate the second document score corresponding to the fourth matching document, and determine whether the second document score is greater than or equal to the second expected threshold E2.

[0078] Specifically, the score of the second document corresponding to the fourth matching document is calculated in the same way.

[0079] Step S8: If the score of the second document is greater than or equal to the second expected threshold, then the data corresponding to the third matching document is used as the data standard.

[0080] Specifically, if the score of the second document is greater than or equal to the second expected threshold E2, then the data corresponding to the d2 most matching documents that are greater than the second expected threshold E2 are used as the data standard. If no matching document is found at this time, the original document is returned for a new query.

[0081] This application employs the aforementioned method to obtain the dataset corresponding to the target project in response to the data standardization operation for the target project. Using the data table names and data item names in the dataset as input, a first data standard retrieval operation is performed. Through a large language model, the dataset is divided into triples corresponding to the data item names based on semantic description relationships, and target keywords are extracted from the dataset based on these triples. Using the target keywords, data table names, and data item names as input, a second data standard retrieval operation is performed on the target project. The data standard corresponding to the target project is obtained through this second data standard retrieval operation. Thus, by combining a dual recall mechanism of full-text search and vector search, semantic parsing and keyword extraction based on a large language model, and a multi-level ranking optimization strategy, the accuracy of the data standardization process is significantly improved.

[0082] Please refer to Figure 5 This illustration shows a schematic diagram of a data standard setting device based on a large artificial intelligence model provided in an embodiment of this application. The device includes an acquisition module 51 and a processing module 52, wherein...

[0083] The acquisition module 51 is used to respond to the operation of defining data standards for the target project and obtain the dataset corresponding to the target project; it performs a first data standard retrieval operation with the data table name and data item name in the dataset as input, and the first data standard retrieval operation corresponds to the first retrieval result; if the data standard corresponding to the target project is not obtained through the first data standard retrieval operation, the dataset is input into a large language model to obtain the semantic description relationship in the dataset; through the large language model, the triples corresponding to the data item names are divided according to the semantic description relationship, and the target keywords in the dataset are extracted according to the triples.

[0084] The processing module 52 is used to perform a second data standard retrieval operation on the target item using the target keyword, data table name and data item name as input. The second data standard retrieval operation corresponds to the second retrieval result. The first retrieval result and the second retrieval result are sorted by a preset sorting operation to obtain the data standard corresponding to the target item.

[0085] In one possible implementation, the acquisition module 51 is used to acquire the dataset corresponding to the target project, specifically including: dividing data into data classes, data subclasses, and data items according to the business links and business processes corresponding to the target project, and constructing a data subset through the data classes, data subclasses, and data items; and using the collection of data subsets as the dataset corresponding to the target project.

[0086] In one possible implementation, the acquisition module 51 is used to input the dataset into a large language model to obtain the semantic description relationship of the data item names in the dataset. Specifically, this includes: using the large language model to parse the hierarchical structure of the dataset in a progressive manner, the hierarchical structure including dataset name, data subset name, data class name, data table name, and data item name; segmenting the data item names into words, and ranking the data item names according to the accuracy of the word segmentation results based on the positional relationship of the data item names in the hierarchical structure; and using the ranking result as the semantic description relationship output by the large language model.

[0087] In one possible implementation, the acquisition module 51 is used to divide the data item names into triples according to semantic description relationships using a large language model, and extract target keywords from the dataset based on the triples. Specifically, this includes: dividing the data item names into triples according to semantic description relationships using a large language model, where each triple includes the subject, object, and relationship; and extracting target keywords from the dataset based on the triples.

[0088] The processing module 52 is used for the second data standard retrieval operation, which includes a first search operation and a second search operation. It obtains the second search result corresponding to the second data standard retrieval operation, specifically by: obtaining the first matching document corresponding to the first search operation and obtaining the second matching document corresponding to the second search operation; and using the first matching document and the second matching document as the second search result.

[0089] In one possible implementation, the processing module 52 is used to obtain the first matching document corresponding to the first search operation, specifically including: using the target keyword as the query condition, performing full-text search on the original document in the target project to obtain multiple first query documents; calculating the relevance score between each first query document and the original document, and sorting the multiple relevance scores; obtaining the first query document that meets the first preset sorting threshold in the sorting results as the first matching document.

[0090] In one possible implementation, the processing module 52 is used to obtain the second matching document corresponding to the second search operation, specifically including: mapping the target keywords to query vectors using a preset text vectorization model; constructing a vector database of the target item based on the HNSW index, the vector database being used to store the correspondence between multiple second query documents and multiple document vectors, with one document vector corresponding to one second query document; calculating multiple similarity values ​​between the query vector and multiple document vectors in the vector database, with one document vector corresponding to one similarity value, and sorting the multiple similarity values; obtaining the target document vector corresponding to the similarity value that meets the second preset sorting threshold in the sorting results, and using the second query document corresponding to the target document vector as the second matching document.

[0091] In one possible implementation, the processing module 52 is used to sort the first search result and the second search result by a preset sorting operation to obtain the data standard corresponding to the target item. Specifically, this includes: performing a first sort on the first search result and the second search result by weighted summation; obtaining a third matching document in the first sorting result that meets a third preset sorting threshold, and performing a second sort on the third matching document by a re-sorting model; obtaining a fourth matching document in the second sorting result that meets a fourth preset sorting threshold; calculating a second document score corresponding to the fourth matching document, and determining whether the second document score is greater than or equal to an expected threshold; if the second document score is greater than or equal to the expected threshold, then using the data corresponding to the fourth matching document as the data standard.

[0092] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0093] This application also provides an electronic device. (See reference...) Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: at least one processor 601, at least one communication bus 602, a user interface 603, at least one network interface 604, and a memory 605.

[0094] The communication bus 602 is used to enable communication between these components.

[0095] The user interface 603 may include a display screen and a camera. Optionally, the user interface 603 may also include a standard wired interface and a wireless interface.

[0096] The network interface 604 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0097] The processor 601 may include one or more processing cores. The processor 601 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 605, and by calling data stored in the memory 605. Optionally, the processor 601 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 601 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 601 and may be implemented as a separate chip.

[0098] The memory 605 may include random access memory (RAM) or read-only memory. Optionally, the memory 605 may include a non-transitory computer-readable storage medium. The memory 605 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 605 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 605 may also be at least one storage device located remotely from the aforementioned processor 601. (Refer to...) Figure 6The memory 605, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for developing data standards based on a large-scale artificial intelligence model.

[0099] exist Figure 6 In the illustrated electronic device, the user interface 603 is primarily used to provide an input interface for the user and acquire user input data; while the processor 601 can be used to call the data standard formulation application stored in the memory 605 based on the large artificial intelligence model. When executed by one or more processors 601, the electronic device performs one or more of the methods described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0100] This application also provides a computer-readable storage medium storing instructions. When executed by one or more processors, these instructions cause an electronic device to perform one or more of the methods described in the above embodiments.

[0101] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0102] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0106] The above description is merely an exemplary embodiment disclosed in this application and should not be construed as limiting the scope of this application. Any equivalent changes and modifications made in accordance with the teachings of this application shall still fall within the scope of this application.

[0107] This application is intended to cover any variations, uses, or adaptations disclosed herein that follow the general principles disclosed herein and include common knowledge or customary technical means in the art that are not described in this application.

Claims

1. A method for formulating data standards based on large-scale artificial intelligence models, characterized in that, The method includes: In response to the operation of defining data standards for the target project, the dataset corresponding to the target project is obtained; Using the data table name and data item name in the dataset as input, a first data standard retrieval operation is performed, and the first data standard retrieval operation corresponds to the first retrieval result. If the data standard corresponding to the target item is not obtained through the first data standard retrieval operation, the dataset is input into a large language model to obtain the semantic description relationship in the dataset. Specifically, this includes: using the large language model to parse the hierarchical structure of the dataset in a progressive manner, the hierarchical structure including the dataset name, data subset name, data class name, data table name, and data item name; segmenting the data item name into words, and ranking the data item names according to the accuracy of the word segmentation results based on the positional relationship of the data item names in the hierarchical structure; and using the ranking result as the semantic description relationship output by the large language model. Using the large language model, the data item names are divided into triples according to the semantic description relationship, and the target keywords in the dataset are extracted based on the triples. Specifically, this includes: using the large language model, dividing the data item names into triples according to the semantic description relationship, where each triple includes the subject, object, and relationship; and extracting the target keywords in the dataset based on the triples. Using the target keyword, the data table name, and the data item name as input, a second data standard retrieval operation is performed on the target item, and the second data standard retrieval operation corresponds to the second retrieval result. The first and second search results are sorted using a preset sorting operation to obtain the data standard corresponding to the target item.

2. The method according to claim 1, characterized in that, The process of obtaining the dataset corresponding to the target project specifically includes: Data categories, data subcategories, and data items are divided according to the business links and business processes corresponding to the target project, and data subsets are constructed through the data categories, the data subcategories, and the data items; The set of the data subsets is taken as the dataset corresponding to the target project.

3. The method according to claim 1, characterized in that, The second data standard retrieval operation includes a first search operation and a second search operation, and obtains the second search result corresponding to the second data standard retrieval operation, specifically including: By obtaining the first matching document corresponding to the first search operation, and obtaining the second matching document corresponding to the second search operation; The first matching document and the second matching document are used as the second search result.

4. The method according to claim 3, characterized in that, The step of obtaining the first matching document corresponding to the first search operation specifically includes: The first data standard retrieval operation is used to perform a full-text search on the original documents in the target project to obtain multiple first query documents; Calculate the relevance score between each of the first query documents and the original document, and sort the multiple relevance scores; The first query document that meets the first preset sorting threshold in the sorting results is selected as the first matching document.

5. The method according to claim 3, characterized in that, The step of obtaining the second matching document corresponding to the second search operation specifically includes: The target keywords are mapped into query vectors using a pre-defined text vectorization model; A vector database for the target project is constructed based on the HNSW index. The vector database is used to store the correspondence between multiple second query documents and multiple document vectors, with one document vector corresponding to one second query document. Calculate multiple similarity values ​​between the query vector and multiple document vectors in the vector database, where each document vector corresponds to one similarity value, and sort the multiple similarity values. Obtain the target document vector corresponding to the similarity value that satisfies the second preset sorting threshold in the sorting results, and use the second query document corresponding to the target document vector as the second matching document.

6. The method according to claim 3, characterized in that, The step of sorting the first search result and the second search result through a preset sorting operation to obtain the data standard corresponding to the target item specifically includes: The first and second search results are sorted by weighted summation according to their respective weights. Retrieve the third matching document from the first sorting results that meets the third preset sorting threshold, and A second sort is performed using the third matching document as described in the resorting model; Retrieve the fourth matching document from the second sorting results that meets the fourth sorting threshold; Calculate the second document score corresponding to the fourth matching document, and determine whether the second document score is greater than or equal to the expected threshold; If the score of the second document is greater than or equal to the expected threshold, then the data corresponding to the fourth matching document will be used as the data standard.

7. A data standard setting device based on a large-scale artificial intelligence model, characterized in that, The device includes an acquisition module and a processing module, wherein, The acquisition module is configured to, in response to the operation of defining data standards for a target project, acquire the dataset corresponding to the target project; perform a first data standard retrieval operation using the data table names and data item names in the dataset as input, the first data standard retrieval operation corresponding to a first retrieval result; if the data standard corresponding to the target project is not acquired through the first data standard retrieval operation, the dataset is input into a large language model to obtain the semantic description relationships in the dataset, specifically including: using the large language model to parse the hierarchical structure of the dataset in a progressive manner, the hierarchical structure including dataset name, data subset name, data class name, data table name, and... The data item names are segmented into words, and the data item names are ranked according to the accuracy of the segmentation results, based on their positional relationship in the hierarchical structure. The ranking results are used as the semantic description relationship output by the large language model. The large language model is used to divide the data item names into triples according to the semantic description relationship, and the target keywords in the dataset are extracted based on the triples. Specifically, this includes: dividing the data item names into triples according to the semantic description relationship using the large language model, where each triple includes the subject, object, and relationship; and extracting the target keywords from the dataset based on the triples. The processing module is used to perform a second data standard retrieval operation on the target item using the target keyword, the data table name, and the data item name as input, the second data standard retrieval operation corresponding to a second retrieval result; and to sort the first retrieval result and the second retrieval result through a preset sorting operation to obtain the data standard corresponding to the target item.

Images