Multi-source heterogeneous data governance method for generating artificial intelligence model

Abstract

The present application relates to the technical field of knowledge collection of artificial intelligence models, and particularly relates to a multi-source heterogeneous data management method for generative artificial intelligence models. The present application provides a multi-source heterogeneous data management method for generative artificial intelligence models. The present application firstly extracts entities in knowledge data and obtains a feature vector; further obtains a focus index according to the number distribution of entities of each aspect in the knowledge data, and then obtains a focus factor of each aspect in combination with the distribution of the focus index in the database; further obtains a supporting index according to the entity similarity index of synonymous entities of all the same aspects of the target data and other knowledge data, in combination with the focus factor; finally, obtains the real credibility of the target data in combination with the credibility weight of the database according to the supporting index of the target data by all other knowledge data, and performs data layering, so as to improve the information screening accuracy and enhance the trust degree of the model to the key knowledge.

Description

Technical Field

[0001] This invention relates to the field of knowledge collection technology for artificial intelligence models, and specifically to a method for governing multi-source heterogeneous data for generative artificial intelligence models. Background Technology

[0002] Generative AI capabilities and knowledge are acquired through training and learning. To improve the generative model's ability to creatively generate content and enhance the usability of the generated results, it is necessary to provide it with a large amount of data during both the training and generation phases. This data provided to large models primarily consists of web pages, academic papers, and text and other non-receptive data from numerous open-source databases obtained through distributed web crawlers. Additionally, it includes extensive data from the internal databases of many collaborating companies. This data is then processed using the MinHash LSH algorithm and numerous other algorithms to perform deduplication and cleaning operations.

[0003] However, there are numerous sources of data, including open-source databases and data obtained from the internet, and it is difficult to distinguish between true and false data. Although existing technologies can filter out abnormal or unreliable data based on the credibility of each database and the Isolation Forest algorithm, for many domain-specific professional data, especially the professional knowledge involved in various disciplines in the education field, there are relatively few professional platforms among the sources. Moreover, the amount of various related data on many open-source network platforms is enormous, making it extremely difficult to verify the authenticity of data by relying on the credibility of the data source platform. Ultimately, this leads to a significant reduction in usable data or a decrease in the usability of large model generation results. Summary of the Invention

[0004] To address the technical problem of inaccurate judgment of the credibility of data sources in existing artificial intelligence models, which affects the credibility of large models, the present invention aims to provide a method for multi-source heterogeneous data governance for generative artificial intelligence models. The specific technical solution adopted is as follows:

[0005] Acquire knowledge data from education-related databases; extract entities from each piece of knowledge data and obtain the corresponding feature vectors;

[0006] Based on the quantity distribution of entities in each aspect of each knowledge data, obtain the emphasis index for each aspect; based on the distribution of the emphasis index in the database to which each knowledge data belongs and the emphasis index, obtain the emphasis factor for each aspect.

[0007] Based on the similarity features between the feature vectors of different entities in the same aspect, entity similarity index is obtained and synonymous entities are filtered out; knowledge data is selected one by one as target data; based on the entity similarity index of the target data and the synonymous entities of other knowledge data in all the same aspects, combined with the emphasis factor, the supporting index corresponding to the target data is obtained.

[0008] Based on all the corroborating indicators of the target data from all the other knowledge data, and combined with the credibility weight of the database to which each piece of knowledge data belongs, the true credibility of the target data is obtained and the data is stratified.

[0009] Furthermore, the method for obtaining the emphasis factor includes:

[0010] For any of the knowledge data, obtain the standard score corresponding to the average value of the emphasis index of each aspect in the database across all aspects; merge the standard score and the emphasis index of each aspect to obtain the emphasis factor of each aspect in the knowledge data.

[0011] Furthermore, the method for obtaining the supporting indicators includes:

[0012] The proportion of the emphasis factor in each aspect of the knowledge data is used as the corroborating weight for each aspect of the knowledge data.

[0013] For any other knowledge data and target data, the sum of the entity similarity indices corresponding to all the synonymous entities in each aspect is fused with the corroboration weight of the knowledge data in each aspect to obtain the corroboration index of the knowledge data for the target data.

[0014] Furthermore, the method for obtaining the authenticity and credibility includes:

[0015] For each other knowledge data, the proportion of the credibility weight to the sum of all credibility weights is used as the corroboration weight; based on the corroboration weight, all other knowledge data are integrated to provide corroboration indicators for the target data to obtain the true credibility of the target data.

[0016] Furthermore, the method for performing data layering includes:

[0017] When the true credibility is greater than or equal to the first preset threshold, the corresponding knowledge data is marked as core data; when the true credibility is less than the first preset threshold but greater than or equal to the second preset threshold, the corresponding knowledge data is marked as medium credibility data; when the true credibility is less than or equal to the second preset threshold, the corresponding knowledge data is marked as low credibility data.

[0018] Furthermore, the method for obtaining the entity similarity index includes:

[0019] The cosine similarity between the feature vectors of the two entities is used as the entity similarity index.

[0020] Furthermore, the method for obtaining the synonymous entity includes:

[0021] Two entities whose entity similarity index is greater than a preset similarity threshold are denoted as synonymous entities.

[0022] Furthermore, the method for obtaining the key indicators includes:

[0023] The proportion of the number of entities in each aspect of the knowledge data to the total number of entities in all aspects is used as the emphasis indicator for each aspect of the knowledge data.

[0024] Furthermore, the method for obtaining the feature vector includes:

[0025] For each piece of knowledge data: the text data and image data in the knowledge data are separated, the continuous text is segmented using a Hidden Markov Model, and the part-of-speech tags corresponding to the segmented words are labeled. The segmented words are mapped into dense vectors as feature vectors for the segmented words. For each piece of image data, mean filtering is used to remove noise. The image is converted into a feature vector using SIFT descriptors and SIFT transform. Finally, through multimodal alignment, all vectors are projected into the same semantic space.

[0026] Furthermore, the method for obtaining the credibility weight includes:

[0027] Based on a preset initial credibility, the dataset of the database is input into the credibility iterative calculation framework, and the credibility weight of each database is output.

[0028] The present invention has the following beneficial effects:

[0029] This invention first extracts entities from each piece of knowledge data and obtains corresponding feature vectors, providing a foundation for subsequent analysis. Next, it obtains emphasis indicators based on the quantity distribution of entities in each aspect of the knowledge data, quantifying the degree of attention paid to different content aspects. Then, it combines this with the distribution of emphasis indicators within the database to obtain emphasis factors. By leveraging the overall emphasis tendency of the database, it corrects the emphasis indicators of individual knowledge data, preparing for subsequent comparisons of different knowledge data. Furthermore, it obtains entity similarity indicators between different entities in the same aspect, quantifying the degree of similarity between entities and filtering out synonymous entities, preparing for subsequent analysis of the similarity between two pieces of knowledge data in their descriptions. Further, based on the entity similarity indicators of synonymous entities of the target data and other knowledge data in all the same aspects, combined with the emphasis factors, it obtains supporting indicators for the target data, characterizing the degree of corroboration of the target data by other data and reflecting the authenticity and credibility of the target data. Finally, based on the supporting indicators of all other knowledge data for the target data, combined with the credibility weight of the database to which each piece of knowledge data belongs, it obtains the authenticity and credibility of the target data, characterizing the credibility of the target data, and performs data stratification. This invention analyzes the content emphasis of knowledge data, compares the similarity of synonymous entities in the same aspect, and combines the database credibility for weighted calculation to obtain the credibility score of knowledge data and realize hierarchical management, thereby improving the accuracy of information filtering and enhancing the model's trust in key knowledge. Attached Figure Description

[0030] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart illustrating a method for managing multi-source heterogeneous data in a generative artificial intelligence model, provided as an embodiment of the present invention;

[0032] Figure 2 This is a flowchart illustrating a method for obtaining supporting indicators according to an embodiment of the present invention. Detailed Implementation

[0033] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a multi-source heterogeneous data governance method for generative artificial intelligence models proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0035] The following description, in conjunction with the accompanying drawings, details a specific scheme for a multi-source heterogeneous data governance method for generative artificial intelligence models provided by the present invention.

[0036] Please see Figure 1 The diagram illustrates a flowchart of a multi-source heterogeneous data governance method for generative artificial intelligence models according to an embodiment of the present invention, specifically including:

[0037] Step S1: Obtain knowledge data from education-related databases; extract entities from each piece of knowledge data and obtain the corresponding feature vectors.

[0038] In one embodiment of the invention, knowledge data from education-related databases is first acquired: Through a distributed system, web crawlers are used to scrape relevant data from numerous open-source data platforms across various disciplines. Furthermore, extensive data in the education field is collected and downloaded from numerous professional databases, such as online learning platforms, academic paper websites, and professional journals.

[0039] Then, we obtained internal educational subject data from partner companies and schools.

[0040] The education-related data includes (but is not limited to) the following: basic student information, such as student ID, name, gender, class, etc.; numerous student learning behavior logs, such as student study notes, electronic textbooks, electronic courseware, as well as knowledge summaries, problem-solving processes for after-class exercises, etc.

[0041] Then, these types of data are organized and categorized. For example, professional terms are aligned and organized using educational ontologies (such as course knowledge graphs) to avoid different abbreviations for the same term in different colleges, such as equation and linear equation in two variables.

[0042] For professional data in the field of education, in order to more clearly indicate its content, each piece of data includes a name, definition, application method, etc. (analogous to data dimensions). It may also contain some other data. This data is relatively objective and will not change with human will. Therefore, similar knowledge data in many databases should be the same in these aspects.

[0043] However, data from different sources have different focuses in these aspects, and the credibility of these data sources also varies. Based on these characteristics, the credibility of knowledge data can be comprehensively judged based on the consistency between each knowledge data and data from other sources in different aspects.

[0044] Each piece of knowledge data contains a large amount of information. It describes a related event through numerous professional terms and organizational processes. For example, when explaining a theorem, it will first state the content of the theorem, then explain its application method, and then derive related knowledge such as its converse theorem and corollaries. This series of descriptions constitutes a piece of data.

[0045] Therefore, it is necessary to extract entities from each piece of knowledge data and obtain the corresponding feature vectors to prepare for subsequent comparison of different knowledge data and analysis of the reference value or corroboration between the data.

[0046] In one embodiment of the present invention, for each piece of knowledge data: the text data and image data in the knowledge data are separated, the continuous text is segmented into words using a hidden Markov model, the continuous text is divided into many shorter words, and the part of speech corresponding to the words is labeled. The words are mapped into dense vectors using a word vector (Word2Vec) model as the feature vectors of the words.

[0047] For each image data, mean filtering is used for denoising. The image is converted into a feature vector through SIFT descriptors and SIFT transform. All vectors are projected into the same semantic space through multimodal alignment to capture semantic information. All data are processed and aligned through the same semantic space.

[0048] At the same time, we invited many experts in the field of education to formulate regular expressions (or syntax patterns), such as noun name: xxx theorem, main content is xxx, its application method is xxx, its converse negation theorem is xxx, and its corollary is xxx.

[0049] Then, by learning the OneRel model through multi-task learning and combining it with preset regular expressions (or syntax patterns), entities in the knowledge data are identified, and numerous entities and relationships in the knowledge data are obtained.

[0050] It should be noted that the algorithms and technologies used are all existing technologies and will not be elaborated further.

[0051] Step S2: Based on the distribution of the number of entities in each aspect of each knowledge data, obtain the emphasis index for each aspect; based on the distribution and emphasis index of the emphasis index within the database to which each knowledge data belongs, obtain the emphasis factor for each aspect.

[0052] In the field of educational knowledge data, different databases have different focuses. Some databases mainly focus on the application of theorems, while others focus on various corollaries or proofs of theorems. This leads to different reference values ​​for different aspects of the data when comparing data from different sources. Therefore, it is necessary to analyze the emphasis characteristics of knowledge data.

[0053] Considering that the description of a certain aspect in the knowledge data is more detailed and richer than that of other aspects, it indicates that the data focuses more on that aspect. Therefore, by obtaining the emphasis index of each aspect based on the number distribution of entities in each aspect of the knowledge data, we can quantify the degree of attention paid to different content aspects of each knowledge data, and provide a basis for obtaining rich comparative indicators from the perspective of the entire database.

[0054] Preferably, in one embodiment of the present invention, considering that the higher the proportion of the number of entities in a certain aspect, the higher the degree of emphasis on that aspect and the larger the emphasis index, the proportion of the number of entities in each aspect of the knowledge data to the total number of entities in all aspects is used as the emphasis index for each aspect in the corresponding knowledge data; wherein the distribution of the number of entities is expressed in the form of quantity proportion.

[0055] For a knowledge database, there is a certain focus when collecting knowledge data. For example, a question bank focuses more on the application of knowledge points. Therefore, when uploading data to the question bank, the focus of the check will inevitably be on the application of knowledge points. So the database will check the content that it focuses on more, and vice versa.

[0056] If a piece of knowledge data aligns with the focus of a database, then the credibility of that focus will be relatively higher than other aspects. Therefore, based on the distribution and focus indicators within the database to which each piece of knowledge data belongs, we can obtain the focus factor for each aspect. By leveraging the overall focus of the database, we can correct the focus indicators of individual knowledge data to obtain more accurate focus factors, thus preparing for subsequent comparisons of different knowledge data.

[0057] Preferably, in one embodiment of the present invention, considering that the standard score (z-score) can measure the deviation of data from the mean, thereby showing the emphasis of each aspect in the database, for any knowledge data, the standard score corresponding to the average of the emphasis index of each aspect in the database is obtained in all aspects.

[0058] The dataset used to calculate the standard score is a dataset composed of the average of the emphasis indicators for each aspect in the database. The average of the emphasis indicators represents the overall emphasis indicator for a particular aspect. Considering that the larger the standard score, the further away it is from the average of the dataset, the larger the overall emphasis indicator for a particular aspect, and the larger the emphasis indicator for knowledge data in this aspect, the higher the degree of emphasis of knowledge data in this aspect, and the larger the emphasis factor.

[0059] Based on this, the standard scores and emphasis indicators of each aspect are integrated to obtain the emphasis factors of each aspect within the corresponding knowledge data.

[0060] As an example, after linearly normalizing the standard scores of each aspect in the database to the corresponding data dimension, the product of the normalization result and the emphasis index of the knowledge data in the corresponding aspect is used as the emphasis factor of the knowledge data in the corresponding aspect, thus obtaining the emphasis factor of each aspect in the knowledge data.

[0061] It should be noted that the analysis process for the key indicators and key factors for each aspect of each knowledge data is the same, and will not be repeated here.

[0062] Step S3: Based on the similarity features between the feature vectors of different entities in the same aspect, obtain entity similarity indicators and filter out synonymous entities; select knowledge data as target data one by one; based on the entity similarity indicators of the target data and other knowledge data in all the same aspects of synonymous entities, combined with the emphasis factor, obtain the supporting indicators corresponding to the target data.

[0063] If the entity descriptions of the same aspect in two knowledge data are similar, it means that the entities represent the same semantics or meaning, which is reflected in the similarity between the feature vectors. Therefore, based on the similarity features between the feature vectors of different entities in the same aspect, we can obtain entity similarity index, quantify the degree of similarity between entities, and screen out synonymous entities to prepare for subsequent analysis of the degree of similarity between the descriptions of the two knowledge data.

[0064] Preferably, in one embodiment of the present invention, considering that the greater the cosine similarity between vectors, the higher the degree of similarity between vectors, the cosine similarity between the feature vectors of two entities is used as an entity similarity index, and the cosine similarity reflects the similarity features between the feature vectors. Two entities with an entity similarity index greater than a preset similarity threshold are denoted as synonymous entities.

[0065] As an example, the preset similarity threshold is 0.7.

[0066] It should be noted that cosine similarity is a well-known technology. In other embodiments of the present invention, the implementer may adjust the preset similarity threshold as needed, which will not be elaborated further.

[0067] For any two pieces of knowledge data, the more identical content or entities are in the descriptions of various aspects (such as application and inference), the more likely the two pieces of knowledge data can be considered to corroborate each other. That is, everyone agrees that the inference and application of the theorem are like this. However, considering that knowledge data from different sources have different focuses, it is necessary to adjust the descriptions based on the corresponding focuses of the two pieces of data. In other words, the focus of the corroborating data is more reliable.

[0068] Therefore, knowledge data is selected one by one as target data. Considering that the entity similarity index quantifies the degree of similarity between synonymous entities, the entity similarity index of the target data and other knowledge data in all the same aspects is used as the basis for reference weight adjustment based on the weight of the target data and the emphasis factor. The corresponding supporting index of the target data is obtained to represent the degree of supporting the target data by other data, reflect the authenticity and credibility of the target data, and provide a basis for finally obtaining the credibility weight.

[0069] Preferably, in one embodiment of the present invention, please refer to Figure 2 The flowchart illustrates a method for obtaining corroborating indicators according to an embodiment of the present invention, specifically including:

[0070] Step S301: Use the proportion of each aspect of the knowledge data as the supporting weight for each aspect of the knowledge data.

[0071] Considering that the larger the proportion of a certain aspect of the emphasis factor in the total of all aspects of the emphasis factor, the higher the degree of emphasis on the corresponding aspect, and the higher the corroboration weight when corroborating the target data, the corroboration weight of each aspect in the knowledge data can be obtained in this way.

[0072] Step S302: For any other knowledge data and target data, merge the sum of entity similarity indices corresponding to all synonymous entities in each aspect with the corroborating weight of the knowledge data in each aspect to obtain the corroborating index of the corresponding knowledge data for the target data.

[0073] Considering other knowledge data and target data, the higher the sum of entity similarity indicators corresponding to synonymous entities in a certain aspect, the more synonymous entities there are and the higher the degree of similarity between synonymous entities. At the same time, the higher the corroborating weight of knowledge data in the corresponding aspect, the higher the emphasis of knowledge data in this aspect, the stronger the reference value, and the higher the corroborating indicators.

[0074] Based on this, the corroborating weight of any other knowledge data in each aspect is used as the weighting weight. The sum of the entity similarity indices corresponding to all synonymous entities in each aspect is weighted and summed. The weighted sum is used as the independent variable. After normalization mapping through the softsign function, the function mapping result is used as the corroborating index of the corresponding knowledge data on the target data.

[0075] It should be noted that the analysis process for the supporting indicators of the target knowledge data is the same for each other knowledge data besides the target knowledge data, and will not be repeated here; the softsign function is a well-known technique.

[0076] Step S4: Based on the corroborating indicators of all other knowledge data for the target data, and combined with the credibility weight of the database to which each piece of knowledge data belongs, obtain the true credibility of the target data and perform data stratification.

[0077] Considering that the corroboration indicators of other knowledge data represent the degree of corroboration of the target data, the credibility of the target data is characterized from the perspective of corroboration by other data; however, data in different databases (data sources) have inherent defects in authenticity, and it is necessary to adjust the corroboration indicators of each data for the target data according to the credibility of each database. Therefore, based on the corroboration indicators of all other knowledge data for the target data, combined with the credibility weight of the database to which each piece of knowledge data belongs, the true credibility of the target data is obtained, the credibility of the target data is characterized, and data is layered in this way to complete the governance of the callable data of the generative artificial intelligence model.

[0078] Preferably, in one embodiment of the present invention, based on a preset initial credibility, the dataset of the database is input into the credibility iterative calculation framework, and the credibility weight of each database is output.

[0079] As an example, with an initial confidence level of 0.5, a traditional confidence level iteration calculation framework (using the well-known TRUTHFINDER algorithm) is used to input the datasets from numerous collected databases into the confidence level iteration calculation framework, and output the confidence level weight of each database.

[0080] Preferably, in one embodiment of the present invention, for each other knowledge data, considering that the larger the proportion of the credibility weight of the knowledge data in the sum of the credibility weights of all other knowledge data, the higher the credibility of the corresponding supporting indicator, the proportion of the credibility weight in the sum of the credibility weights of all knowledge data is used as the supporting weight.

[0081] The authenticity and credibility of the target data are obtained by integrating all other knowledge data with corroborating weights.

[0082] Among them, the corroborating weight is used as the weight for the weighted summation. All other knowledge data corroborating indicators of the target data are integrated through weighted summation. The weighted summation result is linearly normalized in all data dimensions corresponding to the target data, and the result is used as the authenticity and credibility of the target data.

[0083] In one embodiment of the present invention: when the true credibility is greater than or equal to a first preset threshold, the corresponding knowledge data is marked as core data; when the true credibility is less than the first preset threshold but greater than or equal to a second preset threshold, the corresponding knowledge data is marked as medium-credibility data; when the true credibility is less than or equal to the second preset threshold, the corresponding knowledge data is marked as low-credibility data.

[0084] As an example, the first preset threshold is 0.8 and the second preset threshold is 0.6.

[0085] Core data is allocated to the core data warehouse and can be used to train generative AI models or as a knowledge base for generative AI models. Medium-credibility data is considered to be basically credible, but may have some minor issues. It can be placed in a sandbox environment to verify the model's capabilities.

[0086] For low-credibility data, since this part of the data has low credibility, it needs to be isolated and re-verified.

[0087] During secondary verification, the initial credibility of these data and the database in the credibility iterative calculation framework (TRUTHFINDER algorithm, a well-known technique) is the average credibility of these data.

[0088] In summary, addressing the technical problem of inaccurate data source credibility assessment in existing artificial intelligence models, which affects the credibility of large models, this invention provides a multi-source heterogeneous data governance method for generative artificial intelligence models. This invention first extracts entities from each piece of knowledge data and obtains corresponding feature vectors; further, it obtains emphasis indicators based on the quantity distribution of entities in each aspect of the knowledge data, and then, combined with the distribution of emphasis indicators within the database, obtains emphasis factors for each aspect; further, it obtains supporting indicators for the target data based on entity similarity indicators of synonymous entities in all the same aspects between the target data and other knowledge data, combined with the emphasis factors; finally, it obtains the true credibility of the target data and performs data stratification based on the supporting indicators of all other knowledge data for the target data, combined with the credibility weight of the respective database. This invention obtains a credibility score for knowledge data and achieves stratified management by extracting entity features from knowledge data, analyzing content emphasis, comparing the similarity of synonymous entities in the same aspect, and performing weighted calculations based on database credibility, thereby improving the accuracy of information filtering and enhancing the model's trust in key knowledge.

[0089] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0090] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for governing multi-source heterogeneous data in generative artificial intelligence models, characterized in that, The method includes: Acquire knowledge data from education-related databases; extract entities from each piece of knowledge data and obtain the corresponding feature vectors; Based on the quantity distribution of entities in each aspect of each knowledge data, obtain the emphasis index for each aspect; based on the distribution of the emphasis index in the database to which each knowledge data belongs and the emphasis index, obtain the emphasis factor for each aspect. Based on the similarity features between the feature vectors of different entities in the same aspect, entity similarity index is obtained and synonymous entities are filtered out; knowledge data is selected one by one as target data; based on the entity similarity index of the target data and the synonymous entities of other knowledge data in all the same aspects, combined with the emphasis factor, the supporting index corresponding to the target data is obtained. Based on all the corroborating indicators of the target data from all the other knowledge data, and combined with the credibility weight of the database to which each piece of knowledge data belongs, the true credibility of the target data is obtained and the data is stratified. The method for obtaining the supporting indicators includes: taking the proportion of the emphasis factor in each aspect of the knowledge data as the supporting weight for each aspect of the knowledge data; for any other knowledge data and the target data, merging the sum of the entity similarity indicators corresponding to all the synonymous entities in each aspect, and the supporting weight of the knowledge data in each aspect to obtain the supporting indicators of the knowledge data for the target data.

2. The method for multi-source heterogeneous data governance for generative artificial intelligence models according to claim 1, characterized in that, The method for obtaining the emphasis factor includes: For any of the knowledge data, obtain the standard score corresponding to the average value of the emphasis index of each aspect in the database across all aspects; merge the standard score and the emphasis index of each aspect to obtain the emphasis factor of each aspect in the knowledge data.

3. The method for multi-source heterogeneous data governance for generative artificial intelligence models according to claim 1, characterized in that, The methods for obtaining the authenticity and credibility include: For each other knowledge data, the proportion of the credibility weight to the sum of all credibility weights is used as the corroboration weight; based on the corroboration weight, all other knowledge data are integrated to provide corroboration indicators for the target data to obtain the true credibility of the target data.

4. The method for multi-source heterogeneous data governance for generative artificial intelligence models according to claim 1, characterized in that, The method for performing data layering includes: When the true credibility is greater than or equal to the first preset threshold, the corresponding knowledge data is marked as core data; when the true credibility is less than the first preset threshold but greater than or equal to the second preset threshold, the corresponding knowledge data is marked as medium credibility data; when the true credibility is less than or equal to the second preset threshold, the corresponding knowledge data is marked as low credibility data.

5. The method for multi-source heterogeneous data governance for generative artificial intelligence models according to claim 1, characterized in that, The method for obtaining the entity similarity index includes: The cosine similarity between the feature vectors of the two entities is used as the entity similarity index.

6. The method for multi-source heterogeneous data governance for generative artificial intelligence models according to claim 1, characterized in that, The method for obtaining the synonymous entity includes: Two entities whose entity similarity index is greater than a preset similarity threshold are denoted as synonymous entities.

7. The method for multi-source heterogeneous data governance for generative artificial intelligence models according to claim 1, characterized in that, The methods for obtaining the key indicators include: The proportion of the number of entities in each aspect of the knowledge data to the total number of entities in all aspects is used as the emphasis indicator for each aspect of the knowledge data.

8. The method for multi-source heterogeneous data governance for generative artificial intelligence models according to claim 1, characterized in that, The method for obtaining the feature vector includes: For each piece of knowledge data: the text data and image data in the knowledge data are separated, the continuous text is segmented using a Hidden Markov Model, and the part-of-speech tags corresponding to the segmented words are labeled. The segmented words are mapped into dense vectors as feature vectors for the segmented words. For each piece of image data, mean filtering is used to remove noise. The image is converted into a feature vector using SIFT descriptors and SIFT transform. Finally, through multimodal alignment, all vectors are projected into the same semantic space.

9. A method for governing multi-source heterogeneous data in generative artificial intelligence models according to claim 1, characterized in that, The method for obtaining the credibility weight includes: Based on a preset initial credibility, the dataset of the database is input into the credibility iterative calculation framework, and the credibility weight of each database is output.

Images