Multi-source heterogeneous data management method for generative artificial intelligence model

Abstract

The invention relates to the technical field of knowledge collection of artificial intelligence models, in particular to a multi-source heterogeneous data management method for a generative artificial intelligence model. The invention provides a multi-source heterogeneous data management method for a generative artificial intelligence model. The method comprises the following steps: firstly, extracting entities in knowledge data and obtaining feature vectors; further obtaining emphasis indexes according to the quantity distribution of entities in each aspect in the knowledge data, and obtaining emphasis factors in each aspect by combining the distribution of emphasis indexes in a database; further obtaining an evidence index according to entity similarity indexes of synonymous entities of the target data and other knowledge data in all the same aspects in combination with the emphasis factor; and finally, according to the evidence indexes of all other knowledge data for the target data, in combination with the credibility weight of the database to which the knowledge data belongs, obtaining the true credibility of the target data and carrying out data layering, thereby improving the information screening accuracy and enhancing the credibility of the model for key knowledge.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge collection of artificial intelligence models, and in particular to a multi-source heterogeneous data governance method for generative artificial intelligence models. BACKGROUND

[0002] The capability and knowledge of a generative artificial intelligence are obtained through training and learning. In order to improve the capability of a generative model in creative content generation and improve the usability of the generation result, a large amount of data needs to be provided to the model during the training and generation stages. The data provided to the large model is basically text and other non-accepting data in web pages, academic papers and a large number of open source databases obtained through distributed crawlers, in addition to a large amount of data in the internal databases of numerous cooperative enterprises. Then, the data is processed through MinHash LSH algorithm and other algorithms to complete the operations of deduplication and cleaning.

[0003] However, the data sources obtained from open source databases and the Internet are numerous and difficult to distinguish between true and false. Although the existing technology can remove abnormal data or untrustworthy data according to the credibility of each database and the isolation forest algorithm, it is extremely difficult to verify the authenticity of the data through the credibility of the data source platform for numerous field-specific professional data, especially the professional knowledge of various disciplines involved in the education field. There are relatively few professional platforms in the source, and there is a large amount of related data on various open source network platforms, which ultimately leads to a large reduction in usable data or a reduction in the usability of the generation result of the large model. SUMMARY

[0004] In order to solve the technical problem of inaccurate credibility judgment of the data source of the artificial intelligence model, which affects the credibility of the large model, the purpose of the present application is to provide a multi-source heterogeneous data governance method for generative artificial intelligence models, and the technical solution adopted is as follows: Obtain knowledge data of an education-related database; extract entities in each piece of knowledge data and obtain corresponding feature vectors; Obtain a focus indicator of each aspect according to the number distribution of entities of each aspect in each piece of knowledge data; obtain a focus factor of each aspect according to the distribution of the focus indicator in the database to which each piece of knowledge data belongs and the focus indicator; Obtain an entity similarity indicator according to similar features between the feature vectors of different entities of the same aspect, and filter out synonymous entities; select the knowledge data one by one as target data; obtain an evidence indicator corresponding to the target data according to the entity similarity indicators of the synonymous entities of all the same aspects of the target data and other knowledge data, in combination with the focus factor; According to the corroboration index of the target data by all other knowledge data, combined with the credibility weight of the database to which each piece of knowledge data belongs, the real credibility of the target data is obtained and data stratification is performed.

[0005] Further, the method for obtaining the focus factor comprises: For any knowledge data, the average value of the focus index of each aspect in the database is obtained, and the corresponding standard score in all aspects is obtained. The standard score and the focus index of each aspect are fused to obtain the focus factor corresponding to each aspect in the knowledge data.

[0006] Further, the method for obtaining the corroboration index comprises: The proportion value of the focus factor of each aspect in the knowledge data is taken as the corroboration weight of each aspect in the knowledge data. For any other knowledge data and the target data, the sum of the entity similarity indexes corresponding to all the synonymous entities in each aspect is fused with the corroboration weight of the knowledge data in each aspect to obtain the corroboration index of the knowledge data to the target data.

[0007] Further, the method for obtaining the real credibility comprises: For each other knowledge data, the proportion value of the credibility weight in the total sum of all credibility weights is taken as a corroboration weighting weight. The corroboration indexes of all other knowledge data to the target data are fused based on the corroboration weighting weight to obtain the real credibility of the target data.

[0008] Further, the method for performing data stratification comprises: When the real credibility is greater than or equal to a first preset threshold, the corresponding knowledge data is marked as core data; when the real credibility is less than the first preset threshold and greater than or equal to a second preset threshold, the corresponding knowledge data is marked as medium credibility data; when the real credibility is less than or equal to the second preset threshold, the corresponding knowledge data is marked as low credibility data.

[0009] Further, the method for obtaining the entity similarity index comprises: The cosine similarity between the feature vectors of two entities is taken as the entity similarity index.

[0010] Further, the method for obtaining the synonymous entity comprises: Two entities with an entity similarity index greater than a preset similarity threshold are recorded as synonymous entities.

[0011] Further, the obtaining method of the aspect emphasis index comprises: A proportion value of the number of entities of each aspect in the knowledge data in the total number of entities of all aspects is taken as an aspect emphasis index corresponding to each aspect in the knowledge data.

[0012] Further, the obtaining method of the feature vector comprises: For each piece of knowledge data: separate the text data and the image data in the knowledge data, perform word segmentation on continuous text through a hidden Markov model, and label the part-of-speech corresponding to the word segmentation, map the word segmentation into a dense vector as a feature vector of the word segmentation; perform denoising on each piece of image data using mean filtering, convert the image into a feature vector through a SIFT descriptor and a SIFT transform; and project all vectors into the same semantic space through multi-modal alignment.

[0013] Further, the obtaining method of the credibility weight comprises: Based on a preset initial credibility, input a data set of a database into a credibility iterative calculation framework, and output a credibility weight of each database.

[0014] The present application has the following beneficial effects: The present application firstly extracts entities in each piece of knowledge data and obtains corresponding feature vectors, providing an analysis basis for the subsequent; further obtains an aspect emphasis index according to the number distribution of entities of each aspect in the knowledge data, quantifies the attention degree of the knowledge data in different content aspects, and then combines the distribution of the aspect emphasis index in the database to obtain an aspect emphasis factor, corrects the aspect emphasis index of a single piece of knowledge data by means of the overall aspect emphasis tendency of the database, and prepares for subsequent comparison of different knowledge data; further obtains an entity similarity index between different entities of the same aspect, quantifies the similarity degree between the entities, and selects synonymous entities, preparing for subsequent analysis of the similarity degree between two pieces of knowledge data in description; further obtains a corroboration index corresponding to the target data according to the entity similarity index of synonymous entities of all the same aspects between the target data and other knowledge data, in combination with the aspect emphasis factor, representing the corroboration degree of other data to the target data, reflecting the real credibility degree of the target data; further obtains the real credibility of the target data according to the corroboration index of the target data of all other knowledge data, in combination with the credibility weight of the database to which each piece of knowledge data belongs, representing the credibility degree of the target data, and performing data layering. The present application obtains the credibility score of the knowledge data and realizes layer management by analyzing the content emphasis degree of the knowledge data, comparing the similarity of synonymous entities of the same aspect, and combining the database credibility for weighted calculation, improving the information screening accuracy and enhancing the trust degree of the model to the key knowledge. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed in the description of the embodiments or the prior art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0016] Figure 1 A flowchart of a multi-source heterogeneous data governance method for generating a generative artificial intelligence model provided by an embodiment of the present application; Figure 2 A flowchart of a method for obtaining evidence indicators provided by an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined purpose, the specific implementation, structure, features and effects of the multi-source heterogeneous data governance method for generating a generative artificial intelligence model according to the present application are described in detail as follows. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0019] The specific scheme of the multi-source heterogeneous data governance method for generating a generative artificial intelligence model provided by the present application is specifically described below with reference to the drawings.

[0020] Please refer to Figure 1 which shows a flowchart of a multi-source heterogeneous data governance method for generating a generative artificial intelligence model provided by an embodiment of the present application, specifically including: Step S1: Obtain knowledge data of an education-related database; extract entities in each piece of knowledge data and obtain corresponding feature vectors.

[0021] In an embodiment of the present application, first, the knowledge data of the education-related database is obtained: through a distributed system, a network crawler is used to crawl a large amount of related data of a large number of disciplines from a large number of open source data platforms. In addition, a large amount of education field data is collected and downloaded in a large number of professional field databases, such as a large number of online learning platforms and professional databases and professional journals of paper websites.

[0022] Then, the internal education-related discipline data of the cooperative enterprise and the school is obtained.

[0023] Among them, the education-related data includes (but not limited to) the following: student basic information such as student ID, name, gender, class, etc.; learning behavior logs of many students, such as student learning notes, etc., electronic teaching materials, electronic courseware, and knowledge summary, after-school problem solving process, etc.

[0024] Then these types of data are sorted and divided, such as aligning and sorting professional terms in it through education ontology (such as course knowledge graph), avoiding the use of different abbreviations for the same term in different colleges, such as equation and binary first order.

[0025] For professional data in the field of education, in order to more clearly indicate its content, each piece of data contains name, definition, application method, etc. (analogous to data dimensions), and may also contain some other aspects of data. These data are relatively objective and will not change with human will, so in many databases, similar knowledge data in these aspects should be the same; But the data from different sources have different emphases in these aspects, and the credibility of these data sources also varies. Based on the above characteristics, the credibility of knowledge data can be comprehensively judged according to the consistency of each knowledge data with other source data in different aspects.

[0026] For each piece of knowledge data, the information content is large, which describes a related event through many professional terms and organization processes, such as the explanation of a theorem. It will first explain the content of the theorem, then describe its application method, and then extend its inverse theorem, corollary and other related knowledge content. This series of thing description constitutes a data.

[0027] Therefore, it is necessary to extract the entities in each piece of knowledge data and obtain the corresponding feature vector, in order to prepare for subsequent comparison of different knowledge data and analysis of the reference value or supporting degree between data.

[0028] In an embodiment of the present application, for each piece of knowledge data: separate the text data and image data in the knowledge data, perform word segmentation on continuous text through a hidden Markov model, divide the continuous text into many shorter words, and label the part-of-speech corresponding to the words. The word segmentation is mapped to a dense vector as the feature vector of the word segmentation through a word vector (Word2Vec) model; The mean filter is used to denoise each image data, the SIFT descriptor and SIFT transformation are used to convert the image into a feature vector, and all vectors are projected into the same semantic space through multi-modal alignment to capture the semantic information therein; all data are aligned through the same semantic space.

[0029] Meanwhile, a plurality of experts in the field of education are invited to formulate regular expressions (or syntax patterns) such as a theorem name: xxx theorem, main content: xxx, application method: xxx, inverse theorem: xxx, and inference: xxx.

[0030] Then, the OneRel model is trained through multitask learning, and the entities in the knowledge data are recognized by combining preset regular expressions (or syntax patterns), so as to obtain a plurality of entities and relationships in the knowledge data.

[0031] It should be noted that the algorithms and techniques used herein are all prior art and will not be described again.

[0032] Step S2: According to the number distribution of entities of each aspect in each piece of knowledge data, obtain the emphasis index of each aspect; and according to the distribution of the emphasis index in the database to which each piece of knowledge data belongs and the emphasis index, obtain the emphasis factor of each aspect.

[0033] In the knowledge data in the field of education, different databases have different focuses, that is, some databases mainly focus on the application of theorems, some databases focus on various inferences of theorems, or the proof of theorems, which leads to different reference values of different aspects of data when comparing data from different sources. Therefore, it is necessary to analyze the emphasis characteristics of the knowledge data.

[0034] If the content of a certain aspect in the knowledge data is more detailed and rich than the description of other aspects, it means that the data pays more attention to the content of this aspect. Therefore, according to the number distribution of entities of each aspect in each piece of knowledge data, the emphasis index of each aspect can be obtained, which can quantify the attention degree of each piece of knowledge data in different content aspects and provide a basis for obtaining rich comparison indexes from the perspective of the entire database.

[0035] Preferably, in an embodiment of the present application, the higher the proportion of the number of entities of a certain aspect occupies, the higher the emphasis degree of this aspect, and the larger the emphasis index. Therefore, the proportion value of the number of entities of each aspect in the knowledge data to the total number of entities of all aspects is taken as the emphasis index of each aspect in the corresponding knowledge data; wherein the number distribution of entities is represented in the form of proportion.

[0036] For a knowledge database, there is a certain emphasis when collecting knowledge data, for example, the application of knowledge points is paid more attention to in the question bank, so when uploading data to the question bank, the check focus is necessarily on the application of knowledge points; therefore, the content of the emphasis of the database is necessarily relatively high in the degree of checking, and vice versa. If a piece of knowledge data matches the focus of the database, the credibility of the focused aspect of the data will be relatively higher than other aspects, so according to the distribution of the focused index in the database to which each piece of knowledge data belongs and the focused index, the focused factor of each aspect is obtained, the focused index of the single knowledge data is corrected by the overall focused tendency of the database, and a more accurate focused factor is obtained to prepare for subsequent comparison of different knowledge data.

[0037] Preferably, in an embodiment of the present application, the standard score (z-score) can measure the deviation distribution of the data relative to the average value, so as to show the focus of each aspect in the database, so for any knowledge data, the average value of the focused index of each aspect in the database is obtained, and the corresponding standard score in all aspects is obtained. Among them, the data set for calculating the standard score is the data set composed of the average value of the focused index of each aspect in the database, and the overall focused index of an aspect is represented by the average value of the focused index. Considering that the larger the standard score is, the farther the distance from the average value of the data set is, the larger the overall focused index of an aspect is, and the larger the focused index of the knowledge data in this aspect is, the higher the focused degree of the knowledge data in this aspect is, and the larger the focused factor is; Therefore, the standard score and the focused index of each aspect are fused to obtain the focused factor of each aspect in the corresponding knowledge data.

[0038] As an example, after linear normalization of the standard scores of each aspect in the database in the corresponding data dimension, the product of the normalized result and the focused index of the knowledge data in the corresponding aspect is taken as the focused factor of the knowledge data in the corresponding aspect, and the focused factor of each aspect in the knowledge data is obtained.

[0039] It should be noted that the analysis process of the focused index and the focused factor of each aspect of each knowledge data is the same and will not be repeated.

[0040] Step S3: obtaining the entity similarity index and screening out synonymous entities according to the similar features between the feature vectors of different entities of the same aspect; selecting the knowledge data one by one as the target data; obtaining the corresponding evidence index of the target data according to the entity similarity index of the synonymous entities of the same aspect of the target data and other knowledge data in all aspects, and combining the focused factor.

[0041] For the contents of two pieces of knowledge data, if the entity description of the same aspect is similar, it means that the semantics or meaning represented by the entity is the same, which is reflected in the similarity between the feature vectors, so the entity similarity index is obtained according to the similar features between the feature vectors of different entities of the same aspect, the similarity between the entities is quantified, and the synonymous entities are screened out to prepare for subsequent analysis of the similarity between the two knowledge data in the description.

[0042] Preferably, in one embodiment of the present application, the greater the cosine similarity between vectors, the higher the similarity between the vectors, so the cosine similarity between the feature vectors of two entities is taken as the entity similarity index, and the similarity between the feature vectors is represented by the cosine similarity. Two entities with an entity similarity index greater than a preset similarity threshold are recorded as synonymous entities.

[0043] As an example, the preset similarity threshold is 0.7.

[0044] It should be noted that the cosine similarity is a known technology, and in other embodiments of the present application, the implementer can adjust the preset similarity threshold as needed, and will not be described again.

[0045] For any two pieces of knowledge data, the more the same content or entities in the description of each aspect (such as application aspect, inference aspect), the more it can be considered that the two pieces of knowledge data can mutually support each other, that is, everyone believes that the inference and application of the theorem are like this, but considering that knowledge data from different sources has different emphases, so the description needs to be adjusted by the corresponding emphasis of each aspect in the two data, that is, the emphasis of the supporting data is more referable.

[0046] Therefore, the knowledge data is selected one by one as the target data; considering that the entity similarity index quantifies the similarity between synonymous entities, the entity similarity index of the synonymous entities of all the same aspects between the target data and other knowledge data is obtained, and the referability weight is modified in combination with the emphasis factor to obtain the supporting index corresponding to the target data, which represents the supporting degree of other data to the target data, reflects the real and reliable degree of the target data, and provides a basis for finally obtaining the credibility weight.

[0047] Preferably, in one embodiment of the present application, please refer to Figure 2 which shows a flowchart of a method for obtaining a supporting index provided by one embodiment of the present application, specifically comprising: Step S301: Taking the proportion value of the emphasis factor of each aspect in the knowledge data as the supporting weight of each aspect in the corresponding knowledge data.

[0048] Considering that the greater the proportion value of the emphasis factor of a certain aspect of the knowledge data in the total sum of the emphasis factors of all aspects, the higher the emphasis degree of the corresponding aspect, and the higher the supporting weight when supporting the target data, the supporting weight of each aspect in the knowledge data is obtained in this way.

[0049] Step S302: For any other knowledge data and target data, the sum of the entity similarity indicators corresponding to all synonymous entities in each aspect is fused with the evidence weight of the knowledge data in each aspect, and the evidence indicator of the corresponding knowledge data to the target data is obtained.

[0050] Considering other knowledge data and target data, the higher the sum of the entity similarity indicators corresponding to synonymous entities in an aspect, the more synonymous entities exist and the higher the similarity between synonymous entities, and the higher the evidence weight of the knowledge data in the corresponding aspect, the higher the emphasis of the knowledge data in this aspect, the stronger the reference, and the higher the evidence indicator; Based on this, the sum of the entity similarity indicators corresponding to all synonymous entities in each aspect is weighted and summed with the evidence weight of any other knowledge data in each aspect as the weighting weight, and the weighted sum result is taken as the independent variable. After normalization mapping by the softsign function, the function mapping result is taken as the evidence indicator of the corresponding knowledge data to the target data.

[0051] It should be noted that each other knowledge data except the target knowledge data has the same analysis process for the evidence indicator of the target data, which will not be repeated. The softsign function is a known technology.

[0052] Step S4: According to the evidence indicators of all other knowledge data to the target data, the real credibility of the target data is obtained by combining the credibility weight of each knowledge data belonging to the database, and the data is layered.

[0053] Considering that the evidence indicators of other knowledge data to the target data represent the degree of evidence received by the target data, and the credibility of the target data is represented from the perspective of evidence of other data; and the data in different databases (data sources) has a natural real degree defect, which needs to be corrected according to the credibility of each database. Therefore, according to the evidence indicators of all other knowledge data to the target data, the real credibility of the target data is obtained by combining the credibility weight of each knowledge data belonging to the database, the credibility of the target data is represented, and data is layered to complete the management of the callable data of the generative artificial intelligence model.

[0054] Preferably, in an embodiment of the present application, based on the preset initial credibility, the data set of the database is input into the credibility iterative calculation framework, and the credibility weight of each database is output.

[0055] As an example, the preset initial credibility is 0.5, a plurality of collected database data sets are input into a credibility iterative calculation framework (using a known TRUTHFINDER algorithm), and the credibility weight of each database is output.

[0056] Preferably, in an embodiment of the present application, for each other knowledge data, the greater the proportion value of the credibility weight of the knowledge data in the total of the credibility weights of all other knowledge data, the higher the credibility of the corresponding supporting index, so the proportion value of the credibility weight in the total of the credibility weights of all knowledge data is taken as the supporting weighting weight. Based on the supporting weighting weight, the supporting indicators of all other knowledge data to the target data are fused to obtain the real credibility of the target data.

[0057] Among them, the supporting weighting weight is taken as the weight of weighted summation, and the supporting indicators of all other knowledge data to the target data are fused by weighted summation, and the weighted summation is linearly normalized in all data dimensions corresponding to the target data, and the result is taken as the real credibility of the target data.

[0058] In an embodiment of the present application: when the real credibility is greater than or equal to a first preset threshold, the corresponding knowledge data is marked as core data; when the real credibility is less than the first preset threshold and greater than or equal to a second preset threshold, the corresponding knowledge data is marked as medium credibility data; and when the real credibility is less than or equal to the second preset threshold, the corresponding knowledge data is marked as low credibility data.

[0059] As an example, the first preset threshold is 0.8 and the second preset threshold is 0.6.

[0060] For core data, it is divided into a core data warehouse, which can be used to train a generative artificial intelligence model or as a knowledge base of a generative artificial intelligence model; for medium credibility data, it is considered to be basically credible, but there may be some small problems, which can be put into a sandbox environment to verify the model capability.

[0061] For low credibility data, since the credibility of this part of data is low, it needs to be isolated and re-verified.

[0062] In the secondary verification, the initial credibility of these data and databases in the credibility iterative calculation framework (TRUTHFINDER algorithm, known technology) is the average value of the credibility of these data.

[0063] In summary, in view of the inaccurate credibility discrimination of the data source of the artificial intelligence model and the technical problem of affecting the credibility of the large model, the application provides a multi-source heterogeneous data management method for generating a generative artificial intelligence model. The application first extracts entities in each piece of knowledge data and obtains corresponding feature vectors; further obtains an emphasis index according to the number distribution of entities in each aspect of the knowledge data, and then obtains an emphasis factor of each aspect in combination with the distribution of the emphasis index in the database; further obtains a supporting index corresponding to the target data according to the entity similarity index of synonymous entities of the target data and other knowledge data in all the same aspects, in combination with the emphasis factor; finally, obtains the real credibility of the target data according to the supporting index of the target data by all other knowledge data, in combination with the credibility weight of the database, and performs data stratification. The application extracts the entity features in the knowledge data, analyzes the content emphasis degree, compares the similarity of synonymous entities in the same aspect, and performs weighted calculation in combination with the database credibility, to obtain the credibility score of the knowledge data and realize stratified management, improve the information screening accuracy, and enhance the trust degree of the model to the key knowledge.

[0064] It should be noted that the above-mentioned sequence of the embodiments of the application is only for description, and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or may be advantageous.

[0065] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment mainly describes the difference from other embodiments.

Claims

1. A multi-source heterogeneous data governance method for generating a generative artificial intelligence model, characterized in that, The method comprises: acquiring knowledge data of an education-related database; extracting entities in each piece of the knowledge data and acquiring corresponding feature vectors; acquiring a focus index of each aspect according to the number distribution of entities of each aspect in each piece of the knowledge data; acquiring a focus factor of each aspect according to the distribution of the focus index in the database to which each piece of the knowledge data belongs and the focus index; acquiring an entity similarity index according to similar features between the feature vectors of different entities of the same aspect and screening out synonymous entities; selecting the knowledge data one by one as target data; acquiring a supporting index corresponding to the target data according to the entity similarity index of the synonymous entities of all the same aspects of the target data and other knowledge data in combination with the focus factor; acquiring a real credibility of the target data according to the supporting index of the target data by all other knowledge data in combination with a credibility weight of the database to which each piece of the knowledge data belongs and performing data stratification.

2. The multi-source heterogeneous data governance method for generating artificial intelligence model according to claim 1, wherein, The method for acquiring the focus factor comprises: for any piece of the knowledge data, acquiring a standard score corresponding to all aspects from the average value of the focus index of each aspect in the database to which the knowledge data belongs; and acquiring a focus factor of each aspect in the knowledge data by fusing the standard score of each aspect and the focus index.

3. The multi-source heterogeneous data governance method for generating artificial intelligence model according to claim 1, characterized in that, The method for acquiring the supporting index comprises: taking a proportion value occupied by the focus factor of each aspect in the knowledge data as a supporting weight of each aspect in the knowledge data; for any other piece of the knowledge data and the target data, fusing the sum of the entity similarity indexes corresponding to all the synonymous entities of each aspect and the supporting weight of the knowledge data in each aspect to acquire a supporting index of the knowledge data for the target data.

4. The multi-source heterogeneous data governance method for generating artificial intelligence model according to claim 1, characterized in that, The method for acquiring the real credibility comprises: for each other piece of the knowledge data, taking a proportion value occupied by the credibility weight in the sum of all the credibility weights as a supporting weighted weight; and acquiring the real credibility of the target data based on the supporting weighted weight by fusing the supporting indexes of the target data by all other pieces of the knowledge data.

5. The multi-source heterogeneous data governance method for generating artificial intelligence model according to claim 1, characterized in that, The method for performing data stratification comprises: when the real credibility is greater than or equal to a first preset threshold, marking the corresponding knowledge data as core data; when the real credibility is less than the first preset threshold and greater than or equal to a second preset threshold, marking the corresponding knowledge data as moderately credible data; and when the real credibility is less than or equal to the second preset threshold, marking the corresponding knowledge data as low credibility data.

6. The multi-source heterogeneous data governance method for generating artificial intelligence model according to claim 1, characterized in that, The method for acquiring the entity similarity index comprises: taking a cosine similarity between the feature vectors of two entities as an entity similarity index.

7. The multi-source heterogeneous data governance method for generating artificial intelligence model according to claim 1, characterized in that, The method for acquiring the synonymous entities comprises: taking two entities with an entity similarity index greater than a preset similarity threshold as synonymous entities.

8. The multi-source heterogeneous data governance method for generating artificial intelligence model according to claim 1, characterized in that, The method for acquiring the focus index comprises: A proportion value of a number of entities of each aspect in the knowledge data in a total number of entities of all aspects is taken as an emphasis index corresponding to each aspect in the knowledge data.

9. The multi-source heterogeneous data governance method for generating artificial intelligence model according to claim 1, wherein, The feature vector acquisition method comprises: For each piece of knowledge data: separate text data and image data in the knowledge data, perform word segmentation on continuous text through a hidden Markov model, annotate a part of speech corresponding to the word segmentation, map the word segmentation into a dense vector as a feature vector of the word segmentation, use mean filtering to denoise each piece of image data, convert the image into a feature vector through a SIFT descriptor and a SIFT transform, and project all vectors into the same semantic space through multi-modal alignment.

10. The multi-source heterogeneous data governance method for generating artificial intelligence model according to claim 1, wherein, The credibility weight acquisition method comprises: Based on a preset initial credibility, input a data set of a database into a credibility iterative calculation framework, and output a credibility weight of each database.

Images