An abnormal data determination method, device and medium based on a large language model
By employing multi-dimensional clustering and local outlier detection algorithms, the problem of unsatisfactory dimensionality reduction in training vocabulary sets for large language models is solved, the accuracy of outlier removal is improved, and the accuracy of clustering results is enhanced.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN NO 6 ENG CO LTD
- Filing Date
- 2026-05-09
- Publication Date
- 2026-07-31
AI Technical Summary
Existing technologies for large language models do not achieve ideal dimensionality reduction results on training vocabulary sets, which leads to the inability to effectively remove outlier data in clustering results, thus affecting training performance.
By performing multi-dimensional clustering on the semantic feature vectors of the vocabulary set trained by the large language model, the clustering effect of each dimension is evaluated, the preferred dimension is selected for clustering, and outlier data points are identified by combining the local outlier detection algorithm.
This improves the accuracy of outlier removal in each category after classifying the vocabulary set trained by the large language model, and enhances the accuracy of clustering results.
Smart Images
Figure CN122490360A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multidimensional data processing technology, and in particular to a method, device and medium for identifying abnormal data based on a large language model. Background Technology
[0002] In the process of machine learning for large language models, the processing of natural language such as words and phrases is the foundation for the generation of large language models. Since it is very difficult to construct the quantitative relationship of natural language words, existing technologies usually use the embedding vector embedding method to convert natural language text data into high-dimensional vectors that computers can understand, that is, semantic feature vectors. Then, the embedding is classified according to the semantic distance of different dimensions in the embedding, thereby realizing the classification learning of natural language.
[0003] However, since embeddings typically have high-dimensional features when describing complex words, the computational cost of clustering increases exponentially with increasing dimensionality, resulting in time-consuming, resource-intensive, and often poor clustering performance. Therefore, dimensionality reduction is usually performed. However, common dimensionality reduction methods, such as principal component analysis (PCA), can lead to inaccurate clustering results because PCA cannot express non-linear semantic relationships. Inaccurate clustering results can result in the presence of words from other categories within the same word class that are difficult to distinguish. These misclassified semantic words, as anomalous data, can significantly interfere with the accuracy of training results for large language models.
[0004] Therefore, current large language models suffer from a technical problem: due to the unsatisfactory dimensionality reduction effect on the training vocabulary set, outlier data in the clustering results of the training vocabulary set cannot be effectively removed, ultimately affecting the training effect of the large language model. Summary of the Invention
[0005] In view of this, the present invention provides a method, device and medium for identifying outlier data based on a large language model, in order to solve the technical problem that outlier data in the clusters obtained after clustering cannot be effectively removed due to the poor dimensionality reduction effect of the training vocabulary set used by the large language model.
[0006] Firstly, a method for identifying outlier data based on a large language model is provided, including: Each word in the training vocabulary set used for training the large language model is embedded to obtain a semantic feature vector of a predetermined number of dimensions for each word. Cluster all words using any dimension of the semantic feature vector, and determine the clustering effectiveness of any dimension based on the number of clusters and the distribution characteristics of the clusters in the clustering results corresponding to any dimension. The degree of advantage of multidimensional clustering in any dimension is determined based on the difference in the number of clusters between the clustering results corresponding to any dimension and the clustering results corresponding to other dimensions, as well as the differences within the clusters. Based on the multidimensional clustering advantage and clustering effectiveness of any dimension, the final clustering effect evaluation value of any dimension is determined. Among all dimensions, the preferred dimension is determined based on the final clustering effect evaluation value. All words are clustered using the preferred dimension, and outlier data points of any resulting cluster are taken as abnormal data points.
[0007] Furthermore, determining the clustering effectiveness of any one of the dimensions includes: Based on the number of clusters in the clustering results corresponding to any dimension and the Euclidean distance between the cluster centers of different clusters, the inter-cluster representation value of the clustering effect of any dimension is determined. The inter-cluster representation value of the clustering effect of any dimension is directly proportional to the number of clusters in the clustering results corresponding to any dimension and inversely proportional to the Euclidean distance between the cluster centers of different clusters. Take any data point in any cluster of the clustering results corresponding to any dimension as the target data point. In the cluster to which the target data point belongs, determine the data point with the closest Euclidean distance to the target data point as the nearest point of the target data point. Calculate the reciprocal of the absolute value of the difference between the feature values of the target data point and its nearest point, and record it as the minimum feature value difference corresponding to the target data point. Record the sum of the minimum feature value differences corresponding to all data points in the cluster to which the target data point belongs as the overall feature value difference of the cluster to which the target data point belongs. Determine the clustering effect within cluster representation value of any dimension based on the mean of the overall feature value differences of each cluster in the clustering results corresponding to any dimension. The clustering effect within cluster representation value of any dimension is proportional to the mean of the overall feature value differences of each cluster in the clustering results corresponding to any dimension. The clustering effectiveness of any dimension is determined based on the inter-cluster representation value and the intra-cluster representation value of the clustering effect in any dimension. The clustering effectiveness of any dimension is directly proportional to both the inter-cluster representation value and the intra-cluster representation value of the clustering effect in any dimension.
[0008] Furthermore, determining the favorability of multidimensional clustering in any of the aforementioned dimensions includes: The dimension other than the aforementioned dimension is denoted as the comparison dimension. The absolute value of the difference between the number of clusters in the clustering result corresponding to the aforementioned dimension and the number of clusters in the clustering result corresponding to the aforementioned comparison dimension is denoted as the difference in the number of clusters between the aforementioned dimension and the aforementioned comparison dimension. The normalized value of the sum of the differences in the number of clusters between the aforementioned dimension and all comparison dimensions is denoted as the first multidimensional clustering favorable characterization term of the aforementioned dimension. Take any cluster in the clustering results corresponding to any comparison dimension as the cluster to be compared, and take the cluster in the clustering results corresponding to any dimension that is closest to the cluster to be compared as a control cluster. Calculate the absolute value of the difference in the number of data points between the cluster to be compared and the control cluster, and use it as the cluster data point difference of any cluster in the clustering results corresponding to any comparison dimension. Calculate the mean of the cluster data point differences of all clusters in the clustering results corresponding to any comparison dimension. The normalized value of the sum of the mean values of the cluster data point differences of all clusters in the clustering results corresponding to all comparison dimensions is denoted as the second multidimensional clustering favorable characterization term of any dimension. The cluster whose average semantic feature value is closest to that of the cluster to be compared in the clustering results corresponding to any dimension is taken as the second-class control cluster. The absolute value of the difference between the average semantic feature value of the cluster to be compared and the second-class control cluster is calculated as the cluster semantic feature value difference of any cluster in the clustering results corresponding to any comparison dimension. The mean of the cluster semantic feature value differences of all clusters in the clustering results corresponding to any comparison dimension is calculated. The normalized value of the sum of the mean of the cluster semantic feature value differences of all clusters in the clustering results corresponding to all comparison dimensions is denoted as the third multidimensional clustering favorable characterization term of any dimension. The degree of multidimensional clustering advantage in any dimension is determined based on the first multidimensional clustering advantage feature, the second multidimensional clustering advantage feature, and the third multidimensional clustering advantage feature. The degree of multidimensional clustering advantage in any dimension is proportional to the first multidimensional clustering advantage feature, the second multidimensional clustering advantage feature, and the third multidimensional clustering advantage feature.
[0009] Furthermore, determining the final clustering performance evaluation value for any of the aforementioned dimensions includes: The final clustering effect evaluation value of any dimension is directly proportional to the multidimensional clustering advantage and the clustering effectiveness of any dimension.
[0010] Furthermore, outlier data points in any of the resulting clusters are considered as anomalous data points, including: After clustering all words according to the preferred dimension and obtaining the corresponding clusters, the local outlier detection algorithm is used to determine the relative outliers in each obtained cluster and record them as outlier data points of each obtained cluster. The outlier data points of each obtained cluster are used as the abnormal data points.
[0011] In a second aspect, a computer device is provided, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the abnormal data determination method based on a large language model as described in the first aspect.
[0012] Thirdly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the abnormal data determination method based on a large language model as described in the first aspect.
[0013] The advantages of this invention compared to the prior art are: This invention analyzes the distribution characteristics of feature values of different language words in the training vocabulary set of a large language model under different dimensions, constructs a distribution feature model to evaluate the clustering effect of each dimension, and further combines the matching correlation characteristics between the clustering results under the current dimension and the clustering results under other dimensions to determine the multidimensional clustering advantage of the current dimension, corrects the clustering effect of the current dimension to obtain a more accurate clustering effect evaluation result, and finally determines the preferred dimension with good clustering effect from all dimensions based on the clustering effect evaluation result. Thus, the accuracy of clustering the large language vocabulary dimension can be improved by selecting the preferred dimension. The abnormal data in the resulting clusters can be more easily identified and removed, thereby improving the accuracy of removing abnormal data in each category after the training vocabulary set used by the large language model is classified. Attached Figure Description
[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0015] Figure 1 This is a flowchart illustrating a method for determining abnormal data based on a large language model, provided in Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of the structure of a computer device provided in Embodiment 2 of the present invention. Detailed Implementation
[0016] To further illustrate the technical solution of the present invention, specific embodiments are described below.
[0017] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a particular feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. Furthermore, a particular feature, structure, or characteristic in one or more embodiments may be combined in any suitable form, and the terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized.
[0018] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0019] Method Implementation Examples: See Figure 1 This is a flowchart illustrating a method for determining abnormal data based on a large language model, as provided in Embodiment 1 of the present invention. Figure 1 As shown, the method may include the following steps: S101 embeds each word in the training vocabulary set used for training the large language model into a semantic feature vector of a predetermined number of dimensions for each word.
[0020] A training vocabulary set is collected for training a large language model, and multi-dimensional embeddings of these words are constructed. Specifically, these words are vectorized and embedded to construct the semantic feature vector corresponding to each word. For ease of understanding, the following example is provided.
[0021] Assuming each word is represented by a 3-dimensional vector, meaning the semantic feature vector corresponding to each word is a 3-dimensional vector, where each dimension represents a semantic feature, and the specific dimensions are [whether it is an animal, whether it can fly, and its size], then the embedding for each word in the training vocabulary set can be constructed as follows: Embeddings={ "Cat": [0.9, 0.1, 0.3], #animal, cannot fly, medium size; "Dog": [0.9, 0.1, 0.5], #animal, cannot fly, medium to large; "Bird": [0.8, 0.9, 0.1], #animal, can fly, small; "Car": [0.1,0.0,0.7], #Non-animal, cannot fly, large; "Airplane": [0.0, 1.0, 0.9], #Non-animal, can fly, very large; "Ship": [0.0,0.0,1.0], #Non-animal, cannot fly, very large; }
[0022] This embedding construction method allows for the construction of embeddings for multiple complex words within the training set, resulting in multi-dimensional embeddings for the training set words. Essentially, this process embeds each word in the training vocabulary set used for training the large language model, yielding a semantic feature vector with a predetermined number of dimensions. These dimensions can be adjusted based on actual needs, such as the number of words in the training vocabulary set, the differences between words, and the accuracy requirements for training the large language model.
[0023] S102, cluster all words using any dimension of the semantic feature vector, and determine the clustering effectiveness of any dimension based on the number of clusters and the distribution characteristics of the clusters in the clustering results corresponding to any dimension.
[0024] The quality of clustering is directly determined by the classification effect of each dimension on the samples, that is, each word in the training vocabulary set. The quality of classification effect of each word under a single dimension can be directly measured by the clustering result obtained after clustering all words according to that dimension.
[0025] In a semantic feature vector, each dimension corresponds to a different semantic feature value; that is, different dimensions correspond to feature values with their own distinct values. After clustering all words along any dimension, we can observe the values of all words under that dimension. Eigenvalues The distribution of features along a dimension is used to judge its effectiveness in classifying words. When the feature values of words along a dimension tend to be extremely similar in extreme localities (e.g., a large range of 0s and 1s), it indicates that the dimension is less effective at distinguishing most words, and subsequent clusters are more likely to contain anomalous data from different categories. In other words, the fewer the number of clusters obtained after clustering all words along any dimension, and the greater the distance between clusters, the more similar the feature values of different words along that dimension tend to be in extreme localities, leading to poorer subsequent clustering performance.
[0026] On the other hand, while aiming for all words to be distributed across as many clusters as possible, it is also necessary to concentrate the data as much as possible within each cluster. This results in data points within clusters that are more likely to contain words of the same type, and makes it easier to identify relatively outlier data points as anomalies. Therefore, we further evaluate the clustering effect. The more similar the feature values of data points within a cluster, the better the cluster's ability to classify words of the same type. By considering the overall effect of all clusters, we can determine the clustering effect for this dimension.
[0027] Based on the above analysis, it can be set to determine the clustering effectiveness of any dimension according to the number of clusters and the distribution characteristics of the clusters in the clustering results corresponding to any dimension, including: Based on the number of clusters in the clustering results corresponding to any dimension and the Euclidean distance between the cluster centers of different clusters, the inter-cluster representation value of the clustering effect of any dimension is determined. The inter-cluster representation value of the clustering effect of any dimension is directly proportional to the number of clusters in the clustering results corresponding to any dimension and inversely proportional to the Euclidean distance between the cluster centers of different clusters. Take any data point in any cluster of the clustering results corresponding to any dimension as the target data point. In the cluster to which the target data point belongs, determine the data point with the closest Euclidean distance to the target data point as the nearest point of the target data point. Calculate the reciprocal of the absolute value of the difference between the feature values of the target data point and its nearest point, and record it as the minimum feature value difference corresponding to the target data point. Record the sum of the minimum feature value differences corresponding to all data points in the cluster to which the target data point belongs as the overall feature value difference of the cluster to which the target data point belongs. Determine the clustering effect within cluster representation value of any dimension based on the mean of the overall feature value differences of each cluster in the clustering results corresponding to any dimension. The clustering effect within cluster representation value of any dimension is proportional to the mean of the overall feature value differences of each cluster in the clustering results corresponding to any dimension. The clustering effectiveness of any dimension is determined based on the inter-cluster representation value and the intra-cluster representation value of the clustering effect in any dimension. The clustering effectiveness of any dimension is directly proportional to both the inter-cluster representation value and the intra-cluster representation value of the clustering effect in any dimension.
[0028] Furthermore, as a preferred embodiment, the effectiveness of clustering in any dimension is as follows:
[0029]
[0030]
[0031] in, This indicates the effectiveness of clustering along the q-th dimension. This represents the inter-cluster representation value of the clustering effect in the q-th dimension. This represents the cluster representation value within the cluster for the q-th dimension.
[0032] middle, This represents the number of clusters obtained after clustering along the q-th dimension. The smaller this number, the more the current dimension is used, and the more the feature values of different words tend to be extremely localized and similar. The stronger the degree of extreme locality of data clusters under the q-th dimension, the lower the effectiveness of clustering in the current q-th dimension. This represents the i-th cluster in the clustering results obtained after clustering along the q-th dimension. This represents the number of clusters other than the i-th cluster. This represents the z-th cluster among the other clusters. Let be the Euclidean distance between the i-th cluster and the cluster centers of the z-th clusters. The greater the Euclidean distance, the more widely the two clusters are spaced apart, and the more it conforms to the characteristics of an extreme local distribution. This represents the distance between the i-th cluster and all other clusters. This represents the combined distance of all clusters corresponding to this interval. The greater the combined distance, the more extreme the distribution of words in the current dimension is in different clusters. The more the current dimension is, the more the feature values of different words tend to be extreme and local. The stronger the degree of extreme local distribution of data clusters under the q-th dimension, the worse the clustering effect of the q-th dimension is in terms of inter-cluster distribution characteristics. The smaller the inter-cluster representation value of the clustering effect of the q-th dimension, the more it is used. Therefore, an inverse proportional form is adopted.
[0033] In this context, words with different but highly similar feature values are also well-suited for clustering into the same group. For example, different breeds of cats are suitable for clustering together, as are cars with different uses. Therefore, this study obtains the clustering effect within a cluster based on the difference in feature values within the clusters. This represents the clustering result obtained after clustering along the q-th dimension, or in other words, the clustering result. The total number of data points within the i-th cluster in the given clusters. Let j be the feature value of the j-th data point. Let be the feature value of the data point in the cluster to which the j-th data point is closest in Euclidean distance. The difference between the feature values of the two data points is considered. The smaller the difference, the higher the similarity between the j-th data point and its nearest other data points, indicating a better clustering effect in the q-th dimension. Therefore, an inverse proportional form is used. , This represents the sum of the differences in feature values between all data points and their nearest data points, serving as the overall similarity among data points within that cluster. A higher overall similarity within a cluster indicates a stronger lexical concentration of data within that cluster. This represents the mean of the concentration of vocabulary in all clusters of the current dimension. The larger the mean, the stronger the concentration of vocabulary in each cluster of the data in the current dimension, the better the clustering effect of the current q-th dimension, and the larger the corresponding cluster representation value within the cluster. This represents the normalization function, used to unify... and The order of magnitude is [number]. Therefore, the effectiveness of clustering across all dimensions can be obtained.
[0034] S103, determine the multidimensional clustering advantage of any dimension based on the difference in the number of clusters between the clustering results corresponding to any dimension and the clustering results corresponding to other dimensions, as well as the differences within the clusters.
[0035] Considering that the final embedding classification is based on multi-dimensional clustering, analyzing only the clustering effect of a single dimension may lead to the clustering results getting stuck in local optima, resulting in poor classification performance of multi-dimensional clusters. Therefore, this paper adjusts the clustering effectiveness of each dimension by comprehensively considering the interaction effect between different dimensions of the embedding.
[0036] To avoid clustering results getting stuck in local optima, the normal expectation is to make the clustering results as diverse as possible. Therefore, we can analyze and examine this by comparing the distribution characteristics of feature values in the clustering results of the current dimension with those of other dimensions. If the current dimension has similar feature value distributions with many other dimensions, it indicates that each category under the current dimension has broader coverage or greater generalization. This means that the current dimension is less specific to a particular word, or that the current dimension is not the only applicable classification standard for a particular word. Therefore, when the current dimension is combined with other dimensions, it is more difficult to generate new and more refined embedding classifications. Thus, if this dimension is used for multidimensional clustering, the obtained clustering results will contain more words with similar feature values but significant differences in reality. For example, regarding the "size" dimension mentioned in the previous steps, most plants and animals, and large industrial products like airplanes and ships, would be grouped into two clusters. However, the plants and animals, as well as airplanes and ships, within these two clusters would be indistinguishable. This is the so-called situation where "the obtained clustering results contain more words with similar feature values but significant differences in reality." Using this dimension for word classification makes it easier for many words that actually belong to different categories to be difficult to distinguish. These words that actually belong to different categories are considered outliers in the current classification results, ultimately interfering with the training effect of the large language model. Therefore, when the current dimension has more similar clusters, similar cluster sizes, and similar feature value distributions within clusters compared to other dimensions, the clustering effect of the current dimension in multidimensional clustering can be considered worse.
[0037] Therefore, it is possible to determine the multidimensional clustering advantage of any dimension based on the difference in the number of clusters and the differences within the clusters between the clustering results corresponding to any dimension and the clustering results corresponding to other dimensions, including: The dimension other than the aforementioned dimension is denoted as the comparison dimension. The absolute value of the difference between the number of clusters in the clustering result corresponding to the aforementioned dimension and the number of clusters in the clustering result corresponding to the aforementioned comparison dimension is denoted as the difference in the number of clusters between the aforementioned dimension and the aforementioned comparison dimension. The normalized value of the sum of the differences in the number of clusters between the aforementioned dimension and all comparison dimensions is denoted as the first multidimensional clustering favorable characterization term of the aforementioned dimension. Take any cluster in the clustering results corresponding to any comparison dimension as the cluster to be compared, and take the cluster in the clustering results corresponding to any dimension that is closest to the cluster to be compared as a control cluster. Calculate the absolute value of the difference in the number of data points between the cluster to be compared and the control cluster, and use it as the cluster data point difference of any cluster in the clustering results corresponding to any comparison dimension. Calculate the mean of the cluster data point differences of all clusters in the clustering results corresponding to any comparison dimension. The normalized value of the sum of the mean values of the cluster data point differences of all clusters in the clustering results corresponding to all comparison dimensions is denoted as the second multidimensional clustering favorable characterization term of any dimension. The cluster whose average semantic feature value is closest to that of the cluster to be compared in the clustering results corresponding to any dimension is taken as the second-class control cluster. The absolute value of the difference between the average semantic feature value of the cluster to be compared and the second-class control cluster is calculated as the cluster semantic feature value difference of any cluster in the clustering results corresponding to any comparison dimension. The mean of the cluster semantic feature value differences of all clusters in the clustering results corresponding to any comparison dimension is calculated. The normalized value of the sum of the mean of the cluster semantic feature value differences of all clusters in the clustering results corresponding to all comparison dimensions is denoted as the third multidimensional clustering favorable characterization term of any dimension. The degree of multidimensional clustering advantage in any dimension is determined based on the first multidimensional clustering advantage feature, the second multidimensional clustering advantage feature, and the third multidimensional clustering advantage feature. The degree of multidimensional clustering advantage in any dimension is proportional to the first multidimensional clustering advantage feature, the second multidimensional clustering advantage feature, and the third multidimensional clustering advantage feature.
[0038] Furthermore, as a preferred option, the advantage of multidimensional clustering in any dimension is as follows:
[0039] Where represents the degree of advantage of multidimensional clustering in the q-th dimension. This represents the number of dimensions other than the current q-th dimension. for The number of clusters in the i-th dimension. This represents the number of clusters in the current q-th dimension. This represents the difference between the number of clusters in the current dimension and the number of clusters in all other dimensions. The smaller this difference, the more similar the number of clusters in the current dimension is to the number of clusters in more dimensions. The worse the clustering effect of the current dimension in multidimensional clustering is, that is, the smaller the favorable characterization term of the first multidimensional clustering is.
[0040] Let be the number of data points in the j-th cluster under the i-th dimension. The number of data points in the q-th dimension and The number of data points in the closest cluster. The smaller the difference in the number of data points between two clusters, the more similar the size of a certain cluster in the current q-th dimension is to the j-th cluster in the ith dimension. The average similarity between the size of all clusters in the i-th dimension and the size of the clusters in the q-th dimension is considered. The smaller the average similarity across all dimensions except the current q-th dimension, the stronger the similarity between the clusters in the current q-th dimension and the clusters in other dimensions. This indicates a poorer clustering effect of the current dimension in multidimensional clustering, and in other words, a smaller favorable characterization term for the second multidimensional clustering.
[0041] Let be the mean or average semantic feature value within the j-th cluster in the i-th dimension, representing the overall semantic feature value level of that cluster. For the q-th dimension and The average semantic feature value within the closest clusters. The smaller this difference, the more similar the feature value distribution is between a certain cluster in the q-th dimension and the j-th cluster in the ith dimension. The average similarity between the feature values of all clusters in the i-th dimension and the feature values of clusters in the current dimension is calculated. The smaller this calculated value is across all dimensions, the stronger the similarity of the average semantic feature values between the current cluster in the q-th dimension and the clusters in more other dimensions. This indicates that the current dimension performs worse in multidimensional clustering, and that the third multidimensional clustering favorable representation term is smaller. The same applies to maximum and minimum value normalization, used to unify the magnitudes of different parts of the formula.
[0042] S104. Based on the multidimensional clustering advantage and clustering effectiveness of any dimension, determine the final clustering effect evaluation value of any dimension. Among all dimensions, determine the preferred dimension based on the final clustering effect evaluation value. Use the preferred dimension to cluster all words and treat the outlier data points of any resulting cluster as abnormal data points.
[0043] After obtaining the multidimensional clustering advantage and clustering effectiveness of any one dimension, the final clustering effect evaluation value of any one dimension can be obtained by combining the two. It is easy to understand that the two are positive representations of the clustering effect of any one dimension from a multidimensional perspective and a single-dimensional perspective, respectively. Therefore, determining the final clustering effect evaluation value of any one dimension includes: The final clustering effect evaluation value of any dimension is directly proportional to the multidimensional clustering advantage and the clustering effectiveness of any dimension.
[0044] Furthermore, the preferred evaluation value for the final clustering effect of any of the aforementioned dimensions is:
[0045] in, This represents the final clustering effect evaluation value of the q-th dimension. It is obtained by combining the multidimensional clustering advantage of this dimension with the original clustering effectiveness, thus obtaining a more accurate evaluation result of the dimensional clustering effectiveness.
[0046] The final clustering performance evaluation values for all dimensions of the embedding are obtained, and the dimensions with final clustering performance evaluation values greater than a preset evaluation threshold are selected as preferred dimensions. The preferred dimensions obtained in this way have a better effect on the clustering of homonymous words, and at the same time, they have a stronger degree of separation of outlier data of different classes in the clusters after classification.
[0047] A multidimensional space is constructed using the obtained preferred dimensions, and k-means clustering is performed within this multidimensional space to classify the embeddings. After classification, for each cluster, the LOF outlier detection algorithm is used to obtain the relatively outlier data points that may exist within that cluster. These outlier data points are then excluded to obtain a cleaner large language model training dataset, resulting in more accurate word classification results for the large language model training dataset.
[0048] Compared to existing technologies that directly apply conventional dimensionality reduction algorithms to the embeddings of different words in a large language model, which can lead to inaccurate clustering results and make it difficult to identify and remove outlier data, this invention analyzes the distribution characteristics of feature values of different language words across different dimensions to construct a distribution feature model to evaluate the clustering effect of each dimension. This yields the clustering effectiveness of each dimension. Furthermore, it combines the matching and correlation features between the clustering results of the current dimension and those of other dimensions to refine the clustering effectiveness of the current dimension, obtaining a more accurate evaluation result. Finally, based on the evaluation results, it determines the optimal dimensions with good clustering performance from all dimensions. This improves the accuracy of clustering large language vocabulary dimensions using the selected optimal dimensions. Consequently, outlier data can be more easily identified and removed from the clusters obtained through clustering using the optimal dimensions, improving the accuracy of outlier removal in each category after classification of the training vocabulary set used by the large language model.
[0049] Figure 2This is a schematic diagram of the structure of a computer device provided in Embodiment 2 of the present invention. Figure 2 As shown, the computer device of this embodiment includes: at least one processor ( Figure 2 Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor, which, when executed by the processor, implements the steps in any of the above method embodiments.
[0050] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 2 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown, or combinations of certain components, or different components.
[0051] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0052] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of the computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.
[0053] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.
[0054] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0055] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0056] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0057] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0058] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A method for identifying outlier data based on a large language model, characterized in that, The method includes: Each word in the training vocabulary set used for training the large language model is embedded to obtain a semantic feature vector of a predetermined number of dimensions for each word. Cluster all words using any dimension of the semantic feature vector, and determine the clustering effectiveness of any dimension based on the number of clusters and the distribution characteristics of the clusters in the clustering results corresponding to any dimension. The degree of advantage of multidimensional clustering in any dimension is determined based on the difference in the number of clusters between the clustering results corresponding to any dimension and the clustering results corresponding to other dimensions, as well as the differences within the clusters. Based on the multidimensional clustering advantage and clustering effectiveness of any dimension, the final clustering effect evaluation value of any dimension is determined. Among all dimensions, the preferred dimension is determined based on the final clustering effect evaluation value. All words are clustered using the preferred dimension, and outlier data points of any resulting cluster are taken as abnormal data points.
2. The method for determining abnormal data based on a large language model according to claim 1, characterized in that, Determining the clustering effectiveness of any one of the aforementioned dimensions includes: Based on the number of clusters in the clustering results corresponding to any dimension and the Euclidean distance between the cluster centers of different clusters, the inter-cluster representation value of the clustering effect of any dimension is determined. The inter-cluster representation value of the clustering effect of any dimension is directly proportional to the number of clusters in the clustering results corresponding to any dimension and inversely proportional to the Euclidean distance between the cluster centers of different clusters. Take any data point in any cluster of the clustering results corresponding to any dimension as the target data point. In the cluster to which the target data point belongs, determine the data point with the closest Euclidean distance to the target data point as the nearest point of the target data point. Calculate the reciprocal of the absolute value of the difference between the feature values of the target data point and its nearest point, and record it as the minimum feature value difference corresponding to the target data point. Record the sum of the minimum feature value differences corresponding to all data points in the cluster to which the target data point belongs as the overall feature value difference of the cluster to which the target data point belongs. Determine the clustering effect within cluster representation value of any dimension based on the mean of the overall feature value differences of each cluster in the clustering results corresponding to any dimension. The clustering effect within cluster representation value of any dimension is proportional to the mean of the overall feature value differences of each cluster in the clustering results corresponding to any dimension. The clustering effectiveness of any dimension is determined based on the inter-cluster representation value and the intra-cluster representation value of the clustering effect in any dimension. The clustering effectiveness of any dimension is directly proportional to both the inter-cluster representation value and the intra-cluster representation value of the clustering effect in any dimension.
3. The method for determining abnormal data based on a large language model according to claim 1, characterized in that, Determining the advantage of multidimensional clustering in any of the aforementioned dimensions includes: The dimension other than the aforementioned dimension is denoted as the comparison dimension. The absolute value of the difference between the number of clusters in the clustering result corresponding to the aforementioned dimension and the number of clusters in the clustering result corresponding to the aforementioned comparison dimension is denoted as the difference in the number of clusters between the aforementioned dimension and the aforementioned comparison dimension. The normalized value of the sum of the differences in the number of clusters between the aforementioned dimension and all comparison dimensions is denoted as the first multidimensional clustering favorable characterization term of the aforementioned dimension. Take any cluster in the clustering results corresponding to any comparison dimension as the cluster to be compared, and take the cluster in the clustering results corresponding to any dimension that is closest to the cluster to be compared as a control cluster. Calculate the absolute value of the difference in the number of data points between the cluster to be compared and the control cluster, and use it as the cluster data point difference of any cluster in the clustering results corresponding to any comparison dimension. Calculate the mean of the cluster data point differences of all clusters in the clustering results corresponding to any comparison dimension. The normalized value of the sum of the mean values of the cluster data point differences of all clusters in the clustering results corresponding to all comparison dimensions is denoted as the second multidimensional clustering favorable characterization term of any dimension. The cluster whose average semantic feature value is closest to that of the cluster to be compared in the clustering results corresponding to any dimension is taken as the second-class control cluster. The absolute value of the difference between the average semantic feature value of the cluster to be compared and the second-class control cluster is calculated as the cluster semantic feature value difference of any cluster in the clustering results corresponding to any comparison dimension. The mean of the cluster semantic feature value differences of all clusters in the clustering results corresponding to any comparison dimension is calculated. The normalized value of the sum of the mean of the cluster semantic feature value differences of all clusters in the clustering results corresponding to all comparison dimensions is denoted as the third multidimensional clustering favorable characterization term of any dimension. The degree of multidimensional clustering advantage in any dimension is determined based on the first multidimensional clustering advantage feature, the second multidimensional clustering advantage feature, and the third multidimensional clustering advantage feature. The degree of multidimensional clustering advantage in any dimension is proportional to the first multidimensional clustering advantage feature, the second multidimensional clustering advantage feature, and the third multidimensional clustering advantage feature.
4. The method for determining abnormal data based on a large language model according to claim 1, characterized in that, Determining the final clustering performance evaluation value for any of the aforementioned dimensions includes: The final clustering effect evaluation value of any dimension is directly proportional to the multidimensional clustering advantage and the clustering effectiveness of any dimension.
5. The method for determining abnormal data based on a large language model according to claim 1, characterized in that, Outlier data points in any of the resulting clusters are considered as abnormal data points, including: After clustering all words according to the preferred dimension and obtaining the corresponding clusters, the local outlier detection algorithm is used to determine the relative outliers in each obtained cluster and record them as outlier data points of each obtained cluster. The outlier data points of each obtained cluster are used as the abnormal data points.
6. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the abnormal data determination method based on a large language model as described in any one of claims 1 to 5.
7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the abnormal data determination method based on a large language model as described in any one of claims 1 to 5.