A training data governance monitoring system and method for artificial intelligence

Abstract

The application discloses a kind of training data governance monitoring systems and methods for artificial intelligence, it is related to artificial intelligence technical field, by corpus preprocessing, sample is divided into semantic unit and establishes text index;Length normalization and position coding are executed to the text to be trained, and semantic feature vector is generated by tensor splicing;Cluster analysis is carried out to vector set, and high-dimensional hypersphere is constructed with cluster center as the center of sphere;Vector density of each cluster is calculated based on hypersurface volume and the number of vectors, and then the retention graduation value positively correlated with density is determined;Probability filtering is carried out on feature vector by combining uniform distribution random number and retention graduation value, to realize the inhibition of redundant corpus and the retention of rare corpus;Finally, the effective text after screening is restored and output by using text index, by quantifying semantic space distribution, the de-duplication and monitoring of training sample are realized, while inhibiting overfitting, the training efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a training data governance and monitoring system and method for artificial intelligence. Background Technology

[0002] With the rapid development of artificial intelligence technologies such as large language models, the quality and governance efficiency of large-scale training datasets have become key factors determining the upper limit of model performance. Traditional data deduplication monitoring methods mostly rely on literal matching or hash collision detection at the character level, failing to effectively identify the deep connotations of the corpus at the semantic level. Furthermore, existing methods lack fine-grained quantification of semantic features when processing multi-source heterogeneous corpora. This leads to a significant consumption of computational resources during model training on learning repetitive general knowledge, reducing training efficiency and wasting resources. In addition, conventional governance schemes, lacking scientific probabilistic filtering mechanisms, are prone to overfitting on specific high-frequency corpora, thus inhibiting the system's ability to capture rare and crucial high-quality knowledge. Therefore, a training data governance and monitoring system and method for artificial intelligence is desired. Summary of the Invention

[0003] The purpose of this invention is to provide a training data governance and monitoring system and method for artificial intelligence, so as to solve the problems raised in the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a method for training data governance and monitoring in artificial intelligence, the method comprising: Step 1: Collect corpus samples from historical training logs, divide the corpus samples into multiple semantic units using a byte-pair encoding algorithm, establish mapping rules between semantic units and word vectors, and establish text indexes between semantic units and the original text information in the corpus samples. Step 2: Perform length normalization on each sentence in the training text to make the length of all sentences reach a preset uniform threshold. Perform position encoding based on the absolute position of each semantic unit in the normalized sentence. Concatenate the word vector and the position encoding vector into a tensor to obtain the semantic feature vector of the semantic unit. Then, gather all semantic feature vectors to form a feature vector set. Step 3: Divide the feature vector set into several clusters using a clustering algorithm. Calculate the arithmetic mean of all feature vectors in each cluster and define it as the cluster center. For each cluster, retrieve the feature vector that is farthest from the cluster center in Euclidean distance and define it as the boundary vector. Construct a hypersphere with the cluster center as the center and the distance from the cluster center to the boundary vector as the radius, with the hypersphere having the same dimension as the semantic feature vector. Step 4: Obtain the hypervolume of each hypersphere structure and the total number of semantic feature vectors contained in each cluster, calculate the vector density of each cluster, and calculate the corresponding retention resolution value based on the vector density, where the retention resolution value is positively correlated with the vector density. Step 5: Assign a corresponding quantization interval to each cluster and map the corresponding retention score value. When the semantic feature vector to be processed is obtained, locate the specific cluster to which the semantic feature vector belongs, generate a random number within the quantization interval. If the random number is greater than or equal to the retention score value, perform the operation of retaining the semantic feature vector. If the random number is less than the retention score value, perform the operation of discarding the semantic feature vector. Step 6 extracts the semantic units corresponding to all retained semantic feature vectors, retrieves the corresponding original text information through the text index, and marks the retrieved original text information as the filtered valid text for subsequent artificial intelligence model training.

[0005] Furthermore, in step 1, the text content in the training samples is collected, and then the text is iteratively merged using a byte-pair encoding algorithm to construct a semantic unit set. This semantic unit set serves as a reference template for segmenting semantic units. The corpus is segmented, transforming long texts into sequences of sub-words composed of semantic units, laying the foundation for subsequent vectorized representation. In this way, complex original text is transformed into statistically significant sequences of sub-words or lexical units, thereby capturing the semantic features of the text at a fine-grained level. Mapping rules associate each unique semantic unit with an initial word vector of a predetermined dimension, representing the semantic position of that unit in the initial vector space.

[0006] Furthermore, in step 2, the positional encoding of semantic units in the normalized sentence is calculated using sine and cosine functions. For example, the Transformer positional encoding formula can be used to obtain the absolute position, dimension index, and total dimension of the semantic units in the normalized sentence. The directly obtained sequence of semantic units is mapped to the reference sentence using a scaling method. The absolute position index represents the sequence number of the semantic unit in the current sentence, and the total dimension is set to a preset fixed value. By concatenating the word vectors and the positional encoding vectors into a tensor, the generated semantic feature vector contains both semantic and positional information, and its final dimension is the predetermined dimension.

[0007] Furthermore, in step 3, the Euclidean distance between any feature vector in the j-th cluster and the cluster center is first calculated. Then, the Euclidean distances from all semantic feature vectors in the cluster to the cluster center are traversed. The maximum value of the Euclidean distance is defined as the radius of the hypersphere corresponding to the j-th cluster.

[0008] Furthermore, in step 4, by traversing the semantic feature vectors in the clusters that are farthest from the cluster center, the distance between the semantic feature vector and the cluster center is used as the radius. Combined with the dimension of the cluster, the hypersphere corresponding to each cluster is calculated using the hypervolume calculation formula. By calculating the high-dimensional hypervolume, the spatial occupancy of semantic units within a specific semantic region can be accurately characterized.

[0009] Furthermore, the formula for calculating vector density is: the total number of semantic feature vectors contained within a cluster divided by the hypervolume of the hypersphere corresponding to that cluster. Vector density reflects the degree of information redundancy within a specific semantic category; the higher the density, the more duplicate samples there are within that semantic region.

[0010] Furthermore, the formula for calculating the retention resolution is: the preset unit value minus the ratio of the minimum vector density of all clusters to the vector density of the current cluster. This retention resolution value falls within a preset numerical range. It can be adjusted to match the preset unit value by multiplying the ratio of the minimum vector density of all clusters to the vector density of the current cluster by a scaling factor. When the density of a cluster equals the minimum density, the retention resolution value is the starting value of the interval, meaning that the corpus in that region is extremely scarce and needs to be fully retained.

[0011] Furthermore, in step 5, the quantization interval is set to a preset closed interval, and the generated random numbers follow a uniform distribution. By setting a higher retention gradation value for high vector density clusters, their retention probability is reduced, and the retention probability is determined based on the retention gradation value. This mechanism ensures that highly redundant corpora are filtered out with a higher probability, while low-redundancy rare corpora are retained with a higher probability. This results in the semantic feature vectors extracted from each cluster having a more consistent spatial distribution density, achieving global balance of the dataset.

[0012] Furthermore, an implementation example is provided to illustrate the above calculation logic. When the vector density of the first cluster is greater than the vector density of the second cluster, and both are greater than the minimum vector density among all clusters: For the first cluster, a first retention gradation value is calculated based on its density; a uniformly distributed random number is generated within a preset interval. If the generated random number is greater than the first retention gradation value, the semantic unit corresponding to that vector is retained. For the second cluster, the calculated second retention gradation value is higher than the first retention gradation value; if the generated random number is less than the second retention gradation value, the semantic unit corresponding to that vector is discarded. Therefore, regarding the vector density of the corresponding semantic feature vector, a higher vector density leads to a higher retention gradation value, and thus a higher retention threshold.

[0013] Furthermore, in step 6, the text reconstruction process relies on a pre-established text index. Each semantic feature vector carries a label pointing to its original semantic unit, which is associated with the specific original text record number through the text index. After probabilistic filtering, reverse retrieval is performed only on the semantic feature vectors marked as retained, extracting the corresponding text fragments from the original corpus. These fragments are then reassembled or stored as independent samples. The filtered effective text maintains the diversity of the original semantic distribution while significantly reducing redundancy.

[0014] A training data governance and monitoring system for artificial intelligence, comprising: a data preprocessing module, a feature extraction module, a cluster management module, a governance and monitoring module, and a data output module; The data preprocessing module is used to collect corpus samples and divide them into semantic units, establish word vector mapping rules and text indexes; The feature extraction module is used to normalize and positionally encode the text to be trained, and generates a set of semantic feature vectors for semantic units through tensor concatenation; The clustering management module is used to divide the feature vector set into several clusters, calculate the cluster centers, and construct the hypersphere corresponding to each cluster. The governance and monitoring module is used to calculate the vector density and retention fraction of each cluster, and to perform discard or retention operations on the semantic feature vectors based on the quantitative comparison results of the generated random numbers and the retention fraction. The data output module is used to restore the original text information corresponding to the preserved vector based on the text index and output a set of valid text.

[0015] The clustering management module also includes: a vector density calculation unit and a density coefficient calculation unit; The vector density calculation unit is used to calculate the vector density of a cluster based on the volume of the hypersphere and the number of vectors in the cluster. The density coefficient calculation unit is used to calculate the density coefficient based on the vector density of the cluster and the minimum vector density among all clusters.

[0016] The governance and monitoring module also includes: a unit for retaining the division value calculation, a random number generation unit, and a vector judgment unit; The retained scale value calculation unit is used to invert the density coefficient to obtain the retained scale value; The random number generation unit is used to generate random numbers that follow a uniform distribution within the quantization interval; The vector judgment unit is used to determine whether the semantic feature vectors meet the retention conditions based on the retention scale value of the cluster and the random number, and to retain the semantic feature vectors that meet the retention conditions.

[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. Compared to traditional surface-level deduplication methods based on string matching, this invention, by constructing semantic feature vectors and a multi-dimensional hypersphere structure, can accurately quantify the distribution density of text in the semantic space. This method can not only identify repetitions with completely identical characters, but also identify semantic repetitions with different expressions but highly redundant core knowledge points.

[0018] 2. By introducing a mechanism that preserves the resolution value and uses random probability filtering, a positive correlation is established between the preserved resolution value and the vector density. This automatically suppresses high-frequency repetitive data while maximizing the preservation of rare and high-value edge knowledge. This mechanism makes the distribution of training data in the vector space more uniform, reducing the risk of local overfitting in the AI ​​model, improving the model's learning effect on distributed knowledge, and increasing the training efficiency of the AI ​​model.

[0019] 3. By constructing a hypersphere model and calculating the hypervolume and vector density, rigorous quantitative metrics are provided for training data governance. The governance monitoring module can provide real-time feedback on data quality status and achieve precise tracing from the feature space to the original text through text indexing. This closed-loop monitoring method and system improves the transparency of data screening and ensures the controllability and efficiency of the artificial intelligence model training process. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the structure of a training data governance and monitoring system for artificial intelligence according to the present invention; Figure 2 This is a flowchart illustrating a training data governance and monitoring method for artificial intelligence according to the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example: Figures 1-2 As shown, the present invention provides a technical solution: a method for training data governance and monitoring in artificial intelligence. Step 1: Collect corpus samples from historical training logs, divide the corpus samples into multiple semantic units using a byte-pair encoding algorithm, establish mapping rules between semantic units and word vectors, and establish text indexes between semantic units and the original text information in the corpus samples. For the acquired corpus samples, natural language sentences and paragraphs are segmented. For example, segmentation can be performed using byte-based encoding algorithms or by directly identifying semantic units and morphemes in paragraphs using a bag-of-words model, thus dividing natural language into semantic units. By segmenting characters or words, long texts can be transformed into discrete, statistically significant sequences of subwords.

[0023] The initial word vectors corresponding to semantic units are obtained through a natural language recognition model, and a mapping table between semantic units and initial word vectors is constructed.

[0024] The mapping rules are stored in memory in the form of a lookup table. Furthermore, a text index is established between semantic units and the original text information in the corpus samples. The text index can employ an inverted index structure, recording the physical offset, document unique identifier, and specific position within the document for each semantic unit.

[0025] Step 2: Perform length normalization on each sentence in the training text to make the length of all sentences reach a preset uniform threshold. Perform position encoding based on the absolute position of each semantic unit in the normalized sentence. Concatenate the word vector and the position encoding vector into a tensor to obtain the semantic feature vector of the semantic unit. Then, gather all semantic feature vectors to form a feature vector set. Specifically, a uniform threshold is first set based on the length of the reference statement. For example, a fixed number of semantic units is set as the normalized standard statement. For each semantic unit in the statement to be processed, the position of the semantic unit is linearly mapped to the standard statement. The ratio of linear mapping is the ratio of the number of semantic units in the statement to be processed to the number of semantic units in the standard statement.

[0026] After the mapping operation is completed, the semantic units of the statement to be processed are positionally encoded at their absolute positions in the normalized statement. The positional encoding vector PE is calculated as follows: PE(pos, 2i) = sin(pos / 10000) 2i / d ), PE (pos, 2i+1) = cos (pos / 10000 2i / d ), where pos represents the absolute position subscript of the semantic unit in the normalized statement, i is the dimension index, and d is the total dimension of the position encoding vector.

[0027] Subsequently, a tensor concatenation operation is performed, concatenating the initial word vectors obtained in step 1 with the positional encoding vectors calculated in step 2 along different dimensions to obtain a semantic feature vector. This semantic feature vector simultaneously contains the semantic connotations of the words themselves, such as word meaning and part of speech, as well as their positional topological information within the sentence. All semantic feature vectors generated from all sentences in the text to be processed are then aggregated to form a feature vector set.

[0028] Step 3: Divide the feature vector set into several clusters using a clustering algorithm. Calculate the arithmetic mean of all feature vectors in each cluster and define it as the cluster center. For each cluster, retrieve the feature vector that is farthest from the cluster center in Euclidean distance and define it as the boundary vector. Construct a hypersphere with the cluster center as the center and the distance from the cluster center to the boundary vector as the radius, with the hypersphere having the same dimension as the semantic feature vector. For each cluster that has been partitioned, the system needs to construct its boundary contour. The system traverses all feature vectors within the cluster and identifies the vector with the largest Euclidean distance to the cluster center, defining it as the boundary vector. A hypersphere is constructed in high-dimensional space with the cluster center as the center and this maximum distance as the radius. This hypersphere defines the semantic space covered by the cluster, and vectors falling within the hypersphere have high semantic homogeneity.

[0029] Step 4: Obtain the hypervolume of each hypersphere structure and the total number of semantic feature vectors contained in each cluster, calculate the vector density of each cluster, and calculate the corresponding retention resolution value based on the vector density, where the retention resolution value is positively correlated with the vector density. Specifically, the hypervolume calculation formula for the hypersphere requires the dimension of the hypersphere as a parameter, and the dimension of the hypersphere is equal to the dimension of the semantic feature vector.

[0030] After obtaining the hypersphere volume, the total number of semantic feature vectors contained in the cluster is divided by its hypersphere volume to obtain the vector density of the cluster. Further, the retention score is calculated based on the vector density, where the retention score of the j-th cluster is denoted as Sj, the minimum value of the vector density among all clusters is denoted as ρmin, and the vector density of the j-th cluster is ρj, Sj=1-(ρmin / ρj). The retention score determines the execution intensity of the subsequent filtering stage.

[0031] Step 5: Assign a corresponding quantization interval to each cluster and map the corresponding retention score value. When the semantic feature vector to be processed is obtained, locate the specific cluster to which the semantic feature vector belongs, generate a random number within the quantization interval. If the random number is greater than or equal to the retention score value, perform the operation of retaining the semantic feature vector. If the random number is less than the retention score value, perform the operation of discarding the semantic feature vector. In this embodiment, each cluster is assigned a quantization interval between (0,1). When a new semantic feature vector is input into the governance and monitoring process, its cluster is first quickly determined using an index structure or cosine similarity, and the retained score value Sj corresponding to that cluster is extracted.

[0032] A random number generator produces a uniformly distributed random number R. The system then performs a comparison: if R < Sj, the feature vector is marked as "discarded"; if R ≥ Sj, the feature vector is marked as "retained". Under this negative feedback adjustment mechanism, high-density clusters have a larger retention gradation value, resulting in a lower probability that the random number falls into the retention interval, thus achieving forced reduction of redundant information. Conversely, vectors from low-density clusters are more likely to be retained. This approach ensures that the distribution density of corpora with different semantic densities tends to be uniform in the feature space after processing.

[0033] Step 6: Extract the semantic units corresponding to all retained semantic feature vectors, retrieve the corresponding original text information through the text index, and mark the retrieved original text information as the filtered valid text for subsequent artificial intelligence model training. Specifically, after the probability filtering step, feature vector identifiers of all items in the "retained" state are collected. Using the text index established in step 1, efficient reverse retrieval is performed. The identifiers are used to locate specific records in the original corpus, and the corresponding original text fragments are extracted. These fragments retain the original language structure and logical relationships. The filtered valid texts are reassembled to generate a training dataset, which is then fed into the artificial intelligence model for training.

[0034] To further illustrate the application of the above technical logic in actual engineering scenarios, this embodiment provides a complete process implementation example.

[0035] In a data governance task for a large language model, the extracted semantic feature vector has a dimension of k=4.

[0036] In step 3, calculate a cluster CA, whose cluster center cA=(0.5,0.5,0.5,0.5). By traversing the 500 feature vectors contained in the cluster, find the boundary vector vb=(0.6,0.6,0.6,0.6) that is farthest from the center.

[0037] The distance between the two vectors is calculated to obtain the cluster radius rA = 0.2.

[0038] In step 4, the hypervolume of the 4-dimensional hypersphere is calculated. According to the hypervolume formula, when k=4: the hypervolume VA≈0.007896.

[0039] Calculate the vector density ρA: ρA = 500 / 0.007896 ≈ 63323.2.

[0040] The minimum vector density ρmin is obtained among all clusters in the global domain at this time, which is 12664.6. The retention fraction value SA of this cluster is calculated as follows: SA = 1 - (12664.6 / 63323.2) = 1 - 0.2 = 0.8.

[0041] In the probability filtering stage of step 5, if the cluster matches a new semantic feature vector, the random number generator generates a uniformly distributed random number R=0.85. Since 0.85≥0.8, the vector is retained. If the generated R=0.45, since 0.45<0.8, the vector is discarded.

[0042] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for training data governance and monitoring in artificial intelligence, characterized in that: The methods include: Collect corpus samples from historical training logs, divide the corpus samples into semantic units, establish mapping rules between semantic units and word vectors, and at the same time establish text indexes between semantic units and the original text information in the corpus samples; For each sentence in the training text, length normalization is performed. Position encoding is performed based on the position of each semantic unit in the normalized sentence. The word vector and the position encoding vector are concatenated into a tensor to obtain the semantic feature vector of the semantic unit. All semantic feature vectors are then collected to form a feature vector set. The feature vector set is divided into several clusters. The arithmetic mean of all feature vectors in each cluster is calculated, and the arithmetic mean is used as the cluster center. For each cluster, a hypersphere with the same dimension as the semantic feature vector is constructed with the cluster center as the center and the distance from the cluster center to the boundary as the radius. Obtain the hypervolume of each hypersphere and the total number of semantic feature vectors contained in each cluster, calculate the vector density of each cluster, and calculate the corresponding retention resolution value based on the vector density. The retention resolution value is positively correlated with the vector density. The methods for calculating the retained scale value include: The ratio of the number of semantic feature vectors in each cluster to the volume of the corresponding hypersphere of the cluster is used as the vector density of the cluster, and the minimum value of the vector density among all clusters is obtained. The ratio of the minimum value to the vector density of any cluster is used as the density coefficient of the arbitrary cluster. The density coefficient is then inverted to obtain the retained fraction value of the arbitrary cluster. A quantization interval is assigned to each cluster and the corresponding retention score is mapped. When the semantic feature vector to be processed is obtained, the cluster to which the semantic feature vector belongs is matched, and a random number is generated within the quantization interval. If the random number is greater than or equal to the retention score, the semantic feature vector is retained. If the random number is less than the retention score, the semantic feature vector is discarded. Extract the semantic units corresponding to all retained semantic feature vectors, retrieve the corresponding original text information through the text index, and mark the retrieved original text information as the filtered valid text.

2. The method for training data governance and monitoring for artificial intelligence according to claim 1, characterized in that: Methods for creating a text index include: The process of text indexing between semantic units and original text information involves constructing a bidirectional mapping table that records the correspondence between the unique identifier of each semantic unit and its position in the original corpus; and storing the bidirectional mapping table in a database.

3. The method for training data governance and monitoring for artificial intelligence according to claim 2, characterized in that: Methods for generating semantic feature vectors include: Set a reference sentence of fixed length, or use the longest sentence in the corpus sample as the reference sentence, and linearly map the position of the semantic unit in each sentence in the text to be trained to the reference sentence to obtain the normalized sentence of each sentence in the corpus sample. The semantic units are positionally encoded based on their position information in the normalized statement to obtain the position encoding vector of each semantic unit. The word vector and encoding vector corresponding to the semantic unit are concatenated to obtain the semantic feature vector. At the same time, the index information of the semantic feature vector and the semantic unit is established.

4. The method for training data governance and monitoring for artificial intelligence according to claim 1, characterized in that: Methods for constructing a hypersphere include: Using the cluster center of the cluster as the center of the hypersphere, we iterate through the distances between all semantic feature vectors in the cluster and the cluster center, take the semantic feature vector farthest from the cluster center as the boundary vector, and take the distance between the cluster center and the boundary vector as the hypersphere radius to construct the hypersphere corresponding to each cluster.

5. The method for training data governance and monitoring for artificial intelligence according to claim 1, characterized in that: The steps for processing the semantic feature vectors also include: Obtain the cluster corresponding to the semantic feature vector to be processed, and obtain the retention gradation value corresponding to the cluster. If the generated random number is less than the retention gradation value, the semantic feature vector to be processed is determined to be redundant information and is triggered to be abandoned. If the random number is greater than or equal to the retention gradation value, the semantic feature vector to be processed is determined to be retained. The random number follows a uniform distribution within the quantization interval.

6. The method for training data governance and monitoring for artificial intelligence according to claim 1, characterized in that: The method for retrieving corresponding original text information using the text index includes: Obtain the semantic feature vector and the index information of the semantic unit. For the retained semantic feature vector, use the index information of the semantic feature vector and the semantic unit, and then perform a reverse retrieval of the original text information of the corpus sample based on the text index between the semantic unit and the original text information in the corpus sample. Mark the original text information retrieved in the reverse retrieval as the filtered valid text.

7. A training data governance and monitoring system for artificial intelligence, used to execute the training data governance and monitoring method for artificial intelligence as described in any one of claims 1-6, characterized in that: The system includes: The system includes a data preprocessing module, a feature extraction module, a clustering management module, a governance and monitoring module, and a data output module. The data preprocessing module is used to collect corpus samples and divide them into semantic units, establish word vector mapping rules and text indexes; The feature extraction module is used to normalize and positionally encode the text to be trained, and generates a set of semantic feature vectors for semantic units through tensor concatenation; The clustering management module is used to divide the feature vector set into several clusters, calculate the cluster centers, and construct the hypersphere corresponding to each cluster. The governance and monitoring module is used to calculate the vector density and retention fraction of each cluster, and to perform discard or retention operations on the semantic feature vectors based on the quantitative comparison results of the generated random numbers and the retention fraction. The data output module is used to restore the original text information corresponding to the preserved vector based on the text index and output a set of valid text.

8. A training data governance and monitoring system for artificial intelligence according to claim 7, characterized in that: The clustering management module also includes: a vector density calculation unit and a density coefficient calculation unit; The vector density calculation unit is used to calculate the vector density of a cluster based on the volume of the hypersphere and the number of vectors in the cluster; the density coefficient calculation unit is used to calculate the density coefficient based on the vector density of the cluster and the minimum value of the vector density among all clusters.

9. A training data governance and monitoring system for artificial intelligence according to claim 7, characterized in that: The governance and monitoring module also includes: a unit for retaining the division value calculation, a random number generation unit, and a vector judgment unit; The retention scale value calculation unit is used to invert the density coefficient to obtain the retention scale value; the random number generation unit is used to generate random numbers that follow a uniform distribution within the quantization interval; the vector judgment unit is used to determine whether the semantic feature vector meets the retention condition based on the retention scale value of the cluster and the random number, and retains the semantic feature vector that meets the retention condition.

Images