Data clustering method, related method, equipment and storage medium

Abstract

The invention discloses a data clustering method, a related method, equipment and a storage medium, and the method comprises the steps: carrying out the analysis based on the data features of a plurality of business data, and obtaining the distribution density and target distance of each business data; wherein the target distance represents the minimum value of the feature distance between the business data and each piece of business data with higher distribution density; determining a cluster center based on the distribution density of the business data and the target distance; clustering the service data based on the cluster center to obtain a data cluster; and based on the business data serving as the current cluster center in each latest data cluster, returning to execute the step of determining the cluster center based on the distribution density and the target distance of the business data until a cycle ending condition is met, and selecting each latest data cluster as a target cluster. According to the scheme, the data clustering precision can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data clustering method and related methods, devices and storage media. Background Art

[0002] Data clustering is a statistical data analysis technique used in many fields. For example, data clustering is widely used in many industries such as finance, education, and healthcare.

[0003] At present, the existing data clustering has the problem of poor clustering accuracy, which has an adverse impact on subsequent data applications such as model training. In view of this, how to improve the accuracy of data clustering has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a data clustering method and related methods, devices and storage media, which can improve the accuracy of data clustering.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a data clustering method, including: analyzing based on data features of several business data to obtain the distribution density and target distance of each business data; wherein the target distance represents the minimum value of the characteristic distance between the business data and each business data with a higher distribution density; determining the cluster center based on the distribution density and target distance of the business data; clustering the business data based on the cluster center to obtain data clusters; based on the business data in each latest data cluster that serves as the current cluster center, returning to execute the step of determining the cluster center based on the distribution density and target distance of the business data, until the loop end condition is met, and selecting each latest data cluster as the target cluster.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a data screening method, including: selecting a cluster to be screened from a target cluster obtained by clustering a number of business data; wherein the target cluster is obtained based on the data clustering method in the above first aspect; detecting the business data in the cluster to be screened based on several redundancy detection methods to obtain a redundancy score of the business data; based on the redundancy score of the business data, screening the cluster to be screened to obtain an initial set; extracting feature representations of the business data in the initial set regarding a number of target features; making predictions based on the feature representations of the business data regarding a number of target features to obtain a quality score of the business data; based on the quality score of the business data, screening the initial set to obtain a target set.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an intelligent dialogue method, including: obtaining a first sentence to be replied to; processing the first sentence based on an intelligent dialogue model to obtain a second sentence for replying to the first sentence; wherein the intelligent dialogue model is trained based on a sample set, and the sample set is a target set obtained by the data screening method of the second aspect mentioned above.

[0008] In order to solve the above-mentioned technical problems, the fourth aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, the memory storing at least program instructions, and the processor being used to execute the program instructions to implement the data clustering method in the above-mentioned first aspect, or the data screening method in the second aspect, or the intelligent dialogue method in the third aspect.

[0009] In order to solve the above-mentioned technical problems, the fifth aspect of the present application provides a computer-readable storage medium, storing program instructions that can be executed by a processor, and the program instructions are used to implement the data clustering method of the first aspect, or the data screening method of the second aspect, or the intelligent dialogue method of the third aspect.

[0010] The above scheme analyzes the data characteristics of several business data to obtain the distribution density and target distance of each business data, and the target distance represents the minimum value of the characteristic distance between the business data and each business data with a higher distribution density, and then determines the cluster center based on the distribution density and target distance of the business data, thereby clustering the business data based on the cluster center to obtain a data cluster, and then returns to the step of determining the cluster center based on the distribution density and target distance of the business data based on the business data as the current cluster center, until the loop end condition is met, that is, each latest data cluster is selected as the target cluster. On the one hand, the initial determination of the cluster center by combining the distribution density and the target distance helps to improve the accuracy of determining the cluster center, and on the other hand, the continuous optimization and adjustment of the cluster center during the data clustering process helps to improve the accuracy of the data cluster as much as possible. Therefore, the accuracy of data clustering can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of an embodiment of a data clustering method of the present application; Figure 2 It is a flow chart of an embodiment of the data screening method of the present application; Figure 3 This is a flow chart of an embodiment of the intelligent dialogue method of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of a data clustering device of the present application; Figure 5 It is a schematic diagram of the framework of an embodiment of the data screening device of the present application; Figure 6 It is a schematic diagram of the framework of an embodiment of the intelligent dialogue device of the present application; Figure 7 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 8 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0012] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0013] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0014] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0015] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the data clustering method of the present application. Specifically, it may include the following steps: Step S11: Analyze the data characteristics of a number of business data to obtain the distribution density and target distance of each business data.

[0016] In an implementation scenario, the business to which the business data belongs can be set according to the actual application needs. For example, in an education application scenario, the business data can include but is not limited to: student performance data, student test data, student lecture data, etc.; or in a financial application scenario, the business data can include but is not limited to: market data, transaction data, financial report data, etc. Other application scenarios can be deduced in the same way, and no examples are given here one by one.

[0017] In an implementation scenario, in order to obtain business data, data requirements may be determined first, then data sources may be identified, and finally business data may be obtained.

[0018] In a specific implementation scenario, you can first clarify the type of data that needs to be collected to determine the data requirements. Taking the financial application scenario as an example, in order to train a related network model that can support the identification of financial risks, you need to collect certain specific types of financial data, such as transaction records, duration data, customer information, etc. At this time, you can clearly know the type of data that needs to be collected as described above. In other application scenarios, this can be deduced by analogy, and no examples are given here one by one.

[0019] In a specific implementation scenario, data sources that support access and are available may be determined. Still taking the financial application scenario as an example, data sources may include, but are not limited to: internal databases (e.g., existing databases within the enterprise), external data suppliers (e.g., purchased or licensed third-party data), open source data (e.g., publicly available financial data, illustratively, reports and databases published by relevant institutions, etc.), etc. In other application scenarios, the same can be applied, and examples are not given here one by one.

[0020] In a specific implementation scenario, data can be obtained from a selected data source through relevant technical means, which can be used as business data. For example, API calls can be used to collect data from Internet resources or third-party service providers; or, database queries can be used to extract data from internally stored databases; or, batch processing can be used to extract data from historical records, such as CSV files, Excel tables, etc.; or, public content (such as market news, related reports, etc.) can be extracted from the Internet. Of course, the above examples are only a few possible ways to obtain business data, and other possible ways are not limited here.

[0021] In an implementation scenario, after obtaining the business data, the business data may be preprocessed, including but not limited to: missing value processing, outlier processing, data format unification, etc. For example, for business data with missing values, appropriate filling methods such as linear interpolation, mean filling, deletion of missing samples, etc. may be used to process missing values; or, for business data with abnormal values, statistical methods may be used to determine the reasonable range of abnormal values, and the abnormal values ​​may be deleted or adjusted. In addition, for business data from different data sources, since there may usually be problems with inconsistent formats, in order to improve the convenience of subsequent processing, the business data may be converted into a unified format and unit.

[0022] In an implementation scenario, still taking the financial application scenario as an example, for business data, key features can be selected based on the knowledge in the financial field, and then new features can be extracted based on the data type, and the features can be converted into numerical form or other appropriate feature representations. Then, with the help of dimensionality reduction techniques, such as PCA (principal component analysis) and deep learning models, features can be extracted as data features of business data.

[0023] In an implementation scenario, after extracting the data features of the business data, the business data can be mapped to the feature space based on the data features of the business data. For any business data, if there are more and closer other business data close to it, the distribution density of the business data is higher, and vice versa, the distribution density of the business data is lower. As a possible example, the Gaussian kernel function can be used to calculate the distribution density of the business data. Of course, other functions can also be used to calculate the distribution density of the business data: …… (1) In the above formula (1), represents the distribution density of the i-th business data, Represents the feature distance between the data features of the i-th business data and the j-th business data, It can be seen that for any business data, the more business data there are with smaller distances to its features, the higher the distribution density of the business data.

[0024] In addition, in the embodiment of the present disclosure, the target distance represents the minimum value of the characteristic distance between the business data and each business data with a higher distribution density. That is to say, for any business data, other business data with a higher distribution density can be found, and the characteristic distances between them and each found business data can be calculated, and then the minimum value can be determined as the target distance: = )……(2) In the above formula (2), represents the target distance of the i-th business data, Indicates that the j-th business data meets the distribution density higher than the i-th business data. Indicates the feature distance between the data features of the i-th business data and the j-th business data, and min indicates the minimum value. In addition, a retrieval tree, such as a KD tree, a Ball tree, etc., can be pre-built based on the data features of each business data to accelerate density calculation and nearest neighbor search.

[0025] Step S12: Determine the cluster center based on the distribution density of the business data and the target distance.

[0026] Specifically, the cluster center can be selected from the business data that meets the data distribution conditions, and the data distribution conditions can specifically include: the distribution density is higher than the density threshold, and the target distance is higher than the distance threshold. That is to say, business data with large distribution density and target distance can be selected as the cluster center, so that when the initial clustering is performed, business data that is densely clustered in the feature space and relatively far away from other densely clustered areas can be selected as the initial cluster center as much as possible, which helps to improve the efficiency of data clustering. Therefore, based on the distribution density of each business data, the first data distribution of each distribution density can be statistically obtained, and based on the target distance of each business data, the second data distribution of each target distance can be statistically obtained. For the convenience of description, taking the case that the distribution density of each business data is within the numerical range of [a, b] as an example, the proportion of business data with a distribution density lower than a in all business data can be statistically calculated, and so on, for any value of the distribution density between a and b, the proportion of business data with a distribution density lower than the value in all business data can be statistically obtained, and the first data distribution of the distribution density can be obtained. Similarly, taking the target distance of each business data in the numerical range of [c, d] as an example, the proportion of business data with a target distance lower than c in all business data can be counted. Similarly, for any value of the target distance between c and d, the proportion of business data with a target distance lower than this value in all business data can be counted, and the second data distribution of the target distance can be obtained. On this basis, the distribution density can be selected as the density threshold based on the first data distribution, and the target distance can be selected as the distance threshold based on the second data distribution. In the above method, by counting the first data distribution of the distribution density and the second data distribution of the target distance, the density threshold and the distance threshold are determined based on the two data distributions respectively, and the relevant thresholds for initially determining the cluster center can be dynamically determined according to the actual situation of the business data. Compared with setting a fixed threshold, the accuracy of threshold determination can be improved.

[0027] In an implementation scenario, as described above for the first data distribution, the first ratio of each distribution density can be obtained based on the first data distribution, and the first ratio of the distribution density represents the first proportion of business data with a lower distribution density in a number of business data. On this basis, the distribution density can be selected as the density threshold based on the first ratio of each distribution density. Exemplarily, the distribution density can be sorted in order from low to high, and the first ratio of the distribution density can be detected in turn to see whether it is higher than a certain ratio threshold (e.g., 90%, 95%), and when it is first detected that the first ratio of a certain distribution density is higher than the ratio threshold, the distribution density can be selected as the density threshold.

[0028] In an implementation scenario, as described above for the second data, the second ratio of each target distance can be obtained based on the second data distribution, and the second ratio of the target distance represents the second proportion of the business data with a lower target distance in the plurality of business data. On this basis, the target distance can be selected as the distance threshold based on the second ratio of each target distance. Exemplarily, the target distances can be sorted in order from small to large, and the second ratio of the target distance can be detected in turn to see whether it is higher than a certain ratio threshold (e.g., 90%, 95%), and when the second ratio of a certain target distance is detected for the first time to be higher than the ratio threshold, the target distance can be selected as the distance threshold.

[0029] Step S13: Clustering the business data based on cluster centers to obtain data clusters.

[0030] In one implementation scenario, as a possible example, the business data may be clustered into the data cluster where the cluster center is located based on the feature distances between the data features of the business data and the data features of each cluster center. For example, for any business data, the feature distances between the data features of the business data and the data features of each cluster center may be measured, and the cluster center with the smallest corresponding feature distance may be selected to cluster the business data into the data cluster where the selected cluster center is located.

[0031] In another implementation scenario, different from the aforementioned method, as another possible example, after clustering the business data to the data cluster where the cluster center is located based on the feature distance between the data features of the business data and the data features of each cluster center, some cluster centers can be selected as new cluster centers based on the target distance of the cluster centers in each data cluster. For each business data in the data cluster to which the unselected cluster center belongs, the business data can be clustered to the data cluster where the new cluster center is located based on the distribution density of the business data.

[0032] In a specific implementation scenario, after a number of data clusters are obtained through preliminary clustering, the cluster centers in each data cluster can be sorted in order of the target distance from large to small, and the cluster centers whose target distances are in the front preset order after sorting are selected as new cluster centers. For example, taking 100 data clusters obtained through preliminary clustering as an example, the cluster centers of these 100 data clusters can be sorted in order of the target distance from large to small, and the cluster centers whose target distances are in the front 50 are selected as new cluster centers. At this time, cluster centers with relatively larger target distances can be further selected from the 100 cluster centers as new cluster centers for subsequent further data clustering, so that the data clusters can be further reduced from the 100 preliminary clusters to 50. Of course, the preset order can be set according to the actual situation, such as 50%, 40%, etc., which is not limited here.

[0033] In a specific implementation scenario, after selecting a new cluster center, for each business data in the data cluster to which the unselected cluster center belongs, based on the distribution density of the business data, a data cluster with a higher distribution density and located at the new cluster center can be selected as a candidate cluster, and then based on the characteristic distance between the data features of the business data and the data features of the business data in each candidate cluster, the business data is clustered into the candidate cluster. Exemplarily, still taking the example that the data clusters can be further reduced from 100 preliminary clusters to 50, for any business data in the data cluster to which the unselected cluster center belongs, a data cluster where the business data with a higher distribution density than the business data is located can be screened out from the 50 data clusters as a candidate cluster. If there are 20 data clusters with a distribution density of business data higher than the distribution density of the business data, the characteristic distance between the data features of the business data and the data features of the business data in the 20 candidate clusters is calculated, and the data cluster with the minimum characteristic distance is selected, such as the 10th data cluster among the 20 data clusters, and the business data can be clustered into the selected data cluster, such as the 10th data cluster. For other business data, the same can be applied, that is, each business data in the data cluster to which the unselected cluster center belongs can be clustered into the data cluster to which each new cluster center belongs.

[0034] Step S14: Based on the business data in each latest data cluster as the current cluster center, return to the step of determining the cluster center based on the distribution density of the business data and the target distance until the loop end condition is met, and select each latest data cluster as the target cluster.

[0035] In one implementation scenario, for the business data currently serving as the cluster center in each latest data cluster (for example, the aforementioned 50 data clusters), the aforementioned step S12 can be returned, and the cycle can be repeated. That is, for these business data currently serving as the cluster center, further refinement and selection can be performed, that is, the cluster center can be determined based on the distribution density and target distance of these business data, such as further reducing the 50 cluster centers to determine 25 cluster centers. Based on this, the business data is clustered based on the cluster center to obtain data clusters, such as re-clustering 25 data clusters, and then the aforementioned step S12 can be returned based on the business data currently serving as the cluster center in the 25 latest data clusters to further refine and reduce, such as reducing to 15 data clusters.

[0036] In an implementation scenario, each time the step of determining the cluster center based on the distribution density and target distance of the business data is returned, the latest data cluster can also be clustered and evaluated based on the cluster evaluation index to obtain a cluster score, and the loop end condition includes that the latest cluster score is higher than the score threshold and no longer increases. Exemplarily, the cluster evaluation index may include but is not limited to: Silhouette coefficient, Dunn coefficient, etc., and the specific type of the cluster evaluation index is not limited here. Exemplarily, when the cluster score of the cluster evaluation index is the same as the cluster score of the cluster evaluation index in the previous iteration loop in a certain iteration loop, and is higher than a certain score threshold, it is considered that the current latest data cluster has met the required clustering effect, and even if it is further refined and reduced, it will not improve the clustering effect, so the loop iteration can be ended. Of course, the above loop end condition is only a possible example in the actual application process, and other possible loop end conditions are not limited thereto. For example, the loop end condition can also be set to include: the current number of iterations is not less than the number threshold (such as 5 times, 10 times, etc.).

[0037] The above scheme analyzes the data characteristics of several business data to obtain the distribution density and target distance of each business data, and the target distance represents the minimum value of the characteristic distance between the business data and each business data with a higher distribution density, and then determines the cluster center based on the distribution density and target distance of the business data, thereby clustering the business data based on the cluster center to obtain a data cluster, and then returns to the step of determining the cluster center based on the distribution density and target distance of the business data based on the business data as the current cluster center, until the loop end condition is met, that is, each latest data cluster is selected as the target cluster. On the one hand, the initial determination of the cluster center by combining the distribution density and the target distance helps to improve the accuracy of determining the cluster center, and on the other hand, the continuous optimization and adjustment of the cluster center during the data clustering process helps to improve the accuracy of the data cluster as much as possible. Therefore, the accuracy of data clustering can be improved.

[0038] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of the data screening method of the present application. Specifically, it may include the following steps: Step S21: Select a cluster to be screened from among target clusters obtained by clustering a number of business data.

[0039] In the disclosed embodiment, the target cluster is obtained based on the process steps in the above-mentioned data clustering method embodiment. For details, please refer to the above-mentioned data clustering method embodiment, which will not be repeated here. Of course, in addition to this, the target cluster can also be obtained by clustering a number of business data based on other clustering methods. Exemplarily, other clustering methods including but not limited to: K-Means, hierarchical clustering, model clustering, etc. can be used, which are not limited here. For the specific clustering process, please refer to the technical details of other clustering methods such as K-Means, hierarchical clustering, model clustering, etc., which will not be repeated here.

[0040] In an implementation scenario, any one of several target clusters may be selected as a cluster to be screened.

[0041] In another implementation scenario, different from the aforementioned method, as another possible method, the data subject of each target cluster can also be obtained, and then based on the data subject of each target cluster, the target cluster is selected as the cluster to be screened. Exemplarily, the subject features of different data subjects can be pre-constructed, and then the data subject of each target cluster is determined based on the similarity between the data features of the business data in each target cluster and the subject features of different data subjects. After obtaining the data subject of each target cluster, the target cluster can be selected as the cluster to be screened according to the actual application needs. For example, taking the expectation of training a neural network model that can support financial risk identification as an example, the target cluster related to "financial risk" can be selected according to the data subject of each target cluster as the cluster to be screened. Of course, the above example is only a possible example in the actual application process, and it does not limit which target cluster is specifically selected as the cluster to be screened.

[0042] Step S22: Based on several redundancy detection methods, the business data in the to-be-screened clusters are respectively detected to obtain redundancy scores of the business data.

[0043] In one implementation scenario, several redundancy detection methods can be used to respectively score the sub-redundancy of the business data, and then based on the sub-redundancy scores detected by the several redundancy detection methods, a comprehensive feature of the business data can be constructed, so that a prediction can be made based on the comprehensive feature of the business data to obtain a redundancy score. The above method, by combining various redundancy detection methods to respectively score the sub-redundancy of the business data, obtains the comprehensive feature of the business data, and makes a prediction based on the sub-redundancy to obtain the redundancy score of the business data, and can realize redundancy detection of the business data by combining various redundancy detection methods.

[0044] In a specific implementation scenario, several redundancy detection methods may include but are not limited to: a first redundancy detection based on a neural network, a second redundancy detection based on business data content, a third redundancy detection based on time series analysis, etc. The specific type of the redundancy detection method is not limited here.

[0045] In a specific implementation scenario, taking N redundant detection methods as an example, each redundant detection method can detect a sub-redundancy score for the business data, and the sub-redundancy scores of the N redundant detection methods can be integrated to construct a comprehensive feature of the business data, that is, an N-dimensional feature. On this basis, the comprehensive feature can be input into a redundant detection model (such as a multi-layer neural network) so that the redundant detection model outputs a redundant score. It should be noted that the higher the redundancy score, the higher the possibility that the business data is redundant data, and vice versa, the lower the redundancy score, the lower the possibility that the business data is redundant data.

[0046] In one implementation scenario, when several redundancy detection methods include a first redundancy detection based on a neural network, the first correlation between different features of the business data can be obtained, and the importance of each of the different features of the business data can be obtained. On this basis, the feature representation of the business data with respect to at least one feature can be predicted based on the neural network model, and at least the sub-redundancy score of the business data with respect to the first redundancy detection can be obtained, so that the redundancy score of the business data can be obtained at least based on the sub-redundancy score of the business data with respect to the first redundancy detection. The above method, by first measuring the first correlation between different features and the importance of each of the different features, determines which feature representation of the business data is extracted, and performs redundancy detection accordingly, which helps to improve the accuracy of redundancy detection.

[0047] In a specific implementation scenario, as a possible example, the first correlation between different features can be measured based on the Pearson correlation coefficient: …… (3) In the above formula (3), Indicates the first correlation between feature X and feature Y, represents the i-th feature element in feature X, represents the i-th feature element in feature Y, Represents the average value of each feature element in feature X, It represents the average value of each feature element in feature Y. Of course, the above example is only a possible implementation method of using the Pearson correlation coefficient to measure the first correlation. When other correlation coefficients are used to measure the first correlation, the same can be applied and no further examples are given here.

[0048] In a specific implementation scenario, as a possible example, the importance of different features can be determined based on Lasso regression to facilitate subsequent feature selection: …… (4) In the above formula (4), represents the predictor variable, represents the response variable, represents the L1 norm of the coefficient vector, is a regularization parameter that controls the intensity of the penalty for the coefficients. For details, please refer to the technical details of Lasso regression, which will not be repeated here. Of course, the above example is only one possible implementation method of using Lasso regression to measure importance. When other methods are used to measure importance, the same method can be used. No more examples are given here.

[0049] In a specific scenario, after obtaining the first correlation between different features and the importance of each of the different features, features whose first correlation with other features is as low as possible and whose own importance is as high as possible can be selected, and feature representations of the business data regarding these features can be extracted. The feature representations of the business data regarding these features can then be directly input into the neural network model to at least obtain a sub-redundancy score of the business data regarding the first redundancy detection.

[0050] In a specific implementation scenario, the feature representation of the business data with respect to at least one of the aforementioned features is predicted based on the neural network model, and the missing detection result, anomaly detection result, and security identification result of the business data can also be obtained. For example, taking the missing detection result as an example, the position of the missing value in the business data can be marked, such as by using a Boolean variable representation, and then using KNN, mean filling or multiple interpolation to process the missing value: = ( )……(5) In the above formula (5), represents the missing values ​​after filling. Indicates the i-th related value in the business data. It should be noted that the above is a formula expression using mean filling. When other methods are used for filling, the same can be applied. No examples are given here. Alternatively, taking the anomaly detection result as an example, a statistical method can be used to calculate the mean and standard deviation of each feature, and the 3-sigma principle can be used to identify abnormal points. The formula is as follows: or . In addition, the isolation forest algorithm can also be used to detect anomalies in multidimensional features. Or, taking the security identification results as an example, the self-supervised learning method can be used to build an autoencoder model to detect potential anomalies or signs of attacks by reproducing inputs. In addition, it is also possible to determine that high-loss reconstruction points are security risks, and record the locations and eigenvalues ​​of these points. Of course, in order to improve the detection accuracy of the neural network model, the neural network model can be tuned and verified in advance. For example, grid search or Bayesian optimization can be used to efficiently select the best parameters of the model, and then K-fold cross validation can be used to evaluate the model performance to ensure stability and generalization ability. Finally, the accuracy, recall rate, F1 score and other indicators are used to comprehensively evaluate the model's ability to identify redundancy, missingness, anomalies and security risks. After the neural network model training converges, the neural network model can be deployed to detect business data in real time through the neural network model. Of course, the detection effect of the neural network model can also be monitored during the use of the neural network model, so as to adjust the model parameters and structure according to the detection effect to correspond to the newly changed data features and attack methods.

[0051] In one implementation scenario, when several redundancy detection methods include a second redundancy detection based on the content of business data, analysis can be performed based on the semantic features of the business data to obtain the association detection results between different business data. Exemplarily, the association detection results between different business data may include the existence of an association relationship, or the absence of an association relationship, etc. Alternatively, the association detection results between different business data may include the degree of association between different business data. On this basis, graph structure data can be constructed based on the semantic features of the business data and the association detection results between different business data, so that data distinction results representing whether different business data are similar but different can be generated based on the graph structure data, and then the sub-redundancy score of the business data on the second redundancy detection can be obtained based on the similarity between the semantic features of different business data and the data distinction results, so as to obtain the redundancy score of the business data on the second redundancy detection, at least the sub-redundancy score of the business data on the second redundancy detection. In the above method, by constructing graph structure data to generate data distinction results representing whether different business data are similar but different, the sub-redundancy score is determined by combining the data distinction results and the similarity between the semantic features of different business data, and the redundancy score can be improved.

[0052] In a specific implementation scenario, as a possible example, text embedding models such as BERT and table embedding models such as TabNet can be used to process business data to obtain embedded representations of business data and ensure that different types of business data such as text and tables are represented in the same vector space. In addition, pre-trained models such as BERT and RoBERTa can be fine-tuned on specific corpora to improve performance in text processing in specific fields. On this basis, architectures such as Transformer can be used to process embedded representations to help establish long-distance dependency information. LSTM, GRU, etc. can also be combined to capture sequence features, and then these features are combined through feature extraction of different granularities (such as characters, words, sentences, paragraphs, etc.) combined with Attention mechanism weighting, and finally, semantic role labeling, entity recognition and other technologies are combined to extract richer semantic information to obtain semantic features.

[0053] In a specific implementation scenario, as a possible example, the LDA topic model and the deep learning model (such as the variational autoencoder VAE) can be used to jointly mine potential associations to improve the recognition ability of multi-level semantic relationships. After obtaining the association detection results, GNNs can be used to construct graph structure data to identify complex association patterns, such as reference relationships between documents, common topics, etc. Exemplarily, the semantic features of each business data can be used as nodes in the graph structure data, and if the association detection results between any two business data represent the association relationship between the business data, the nodes corresponding to the two business data in the graph structure data can be connected to form graph structure data. In addition, if the association detection results further include the degree of association, the degree of association can also be identified on the connection line. Of course, the above examples are only several possible representations of graph structure data, and the specific form of graph structure data is not limited thereto. For details, please refer to the technical details of GNNs, and no examples will be given one by one here.

[0054] In a specific implementation scenario, as a possible example, GAN can be used to generate data distinction results of "similar but different". Specifically, the model can be pre-trained to distinguish highly correlated but non-redundant data pairs, and contrastive learning methods such as SimCLR can be used to enhance the model's ability to distinguish semantically similar but non-redundant data.

[0055] In a specific implementation scenario, as a possible example, a twin network structure can be used to calculate the similarity between different business data by training two neural networks with shared weights. In addition, multi-task learning can be introduced on the similarity calculation model, with tasks such as topic classification and sentiment analysis to improve the generalization ability of feature extraction and similarity calculation.

[0056] In a specific implementation scenario, as a possible example, after obtaining the similarity and data differentiation results, the threshold can be dynamically set through the analysis results of the data distribution (such as Gaussian mixture model GMM) to improve the redundancy detection accuracy of different types of data, and then the similarity threshold and weight can be automatically adjusted for different fields (such as insurance, banking, securities, etc.), and the transfer learning method can be used to improve cross-domain adaptability. Based on this, if the similarity between different business data is higher than the threshold, it can be considered that there is a redundancy risk for different business data, but if the data differentiation result is "similar but different", it can be considered that the different business data is not redundant data, and the similarity between the two is determined to be sub-redundancy.

[0057] In a specific implementation scenario, Hadoop / Spark can be used to process large-scale data sets, and distributed computing can speed up big data analysis. In addition, the algorithm can be expanded to support multiple languages, and multi-language redundancy detection can be achieved through cross-language pre-training models (such as mBERT), structured data, unstructured data, and semi-structured data, so that the system has a wider range of application scenarios, including web content, social media information, etc.

[0058] In an implementation scenario, when there are several redundancy detection methods including the third redundancy detection based on time series analysis, a multivariate time series model (such as VAR, SLTM, etc.) can be constructed to model historical data, and tested and verified to select the best model and parameters, and then further tested using a deep learning model. This allows a set of online reflux data collection algorithms to be deployed, and data processing and testing can be performed. Based on the manual labeling results, the training and updating model can be implemented, and time series analysis can be used for longitudinal testing to obtain the sub-redundancy score of the third redundancy detection, so as to avoid data redundancy caused by time overlap as much as possible.

[0059] Step S23: Based on the redundancy score of the business data, the clusters to be screened are screened to obtain an initial set.

[0060] In one implementation scenario, as a possible example, it can be determined whether the redundancy score of the business data is higher than a score threshold. If so, the business data can be regarded as redundant data. Otherwise, the business data can be regarded as non-redundant data and included in the initial set.

[0061] In another implementation scenario, different from the aforementioned implementation, as another possible example, a dynamic threshold mechanism can also be introduced to automatically adjust the threshold range according to data distribution and usage, and divide the priority level according to the threshold. For example, high priority: immediate removal, medium priority: pending further verification, low priority: optional retention, manual review. On this basis, secondary analysis of medium priority data can be performed, such as using the self-supervised learning model trained by financial data characteristics and target task data, and secondary analysis of medium priority scoring data can be performed to identify potentially valuable data. For some high-value data or data with scores around the critical value, it is recommended to develop intuitive intelligent review tools to improve the efficiency of manual review and provide automated auxiliary suggestions for manual review to ensure that key data is not deleted by mistake. Then, low priority data can be sampled and retained to ensure the diversity and integrity of data samples. Finally, the redundant data detected can be filtered in the time dimension in combination with the results of time series analysis. If some data are repeated at specific time points, the sliding window technology can be used to filter the data in the time dimension to retain the data at the necessary time points. Through the above series of operations, the initial set can be obtained.

[0062] Step S24: extracting feature representations of the business data in the initial set with respect to several target features.

[0063] In an implementation scenario, the target features can be determined based on feature engineering, such as combining time series features and interactive features to generate more meaningful features. For example, for time series features, they can include but are not limited to: rolling window features, seasonal features, and periodic features. For example, in the financial field, they can include but are not limited to moving averages (MA), exponential moving averages (EMA), volatility, etc.; or, for interactive features, the interactions between different features can be generated through a polynomial feature generator, such as considering the nonlinear relationship of features. On this basis, the feature representation of the above target features can be extracted from the business data in the initial set for further processing.

[0064] In another implementation scenario, different from the aforementioned method, as another possible example, several preset features can be determined first, such as the features obtained by feature engineering, and then based on the feature representation of the sample data on several preset features, the preset features can be preliminarily screened as initial features, and then based on the second correlation between the sample data on different initial features, the initial features can be selected as candidate features, and then the sub-feature scores of the candidate features can be fused based on several feature evaluation methods to obtain the feature scores of the candidate features. Finally, based on the feature scores of each candidate feature, the candidate feature can be selected as the target feature, and the feature representation of the target feature can be extracted from the business data in the initial set for further processing. The above method can improve the quality of the target feature through coarse-to-fine screening work such as preliminary selection based on relevance and reselection based on feature evaluation, which helps to improve the accuracy of subsequent business data quality evaluation.

[0065] In a specific implementation scenario, for various preset features, the variance of the corresponding preset features can be obtained based on the feature representation of each sample data on the preset features, and the mutual information between the corresponding preset features and the target variable can be obtained based on the variable information of each sample data on the target variable and the feature representation of the preset features. It should be noted that the calculation process of the above variance and mutual information can refer to the technical details of the two, which will not be repeated here. In addition, if the variance of a certain preset feature is small, it means that the contribution of this preset feature is small in most cases, and if the mutual information corresponding to a certain preset feature is small, it means that the correlation between this preset feature and the target variable is small, that is, the contribution is extremely limited in most cases. On this basis, the preset features can be selected as the initial features based on the variance and mutual information of each preset feature. For example, the preset features with both large variance and large mutual information can be selected as the initial features. In the above method, by calculating the variance and mutual information of each preset feature, and then selecting the initial features based on this, more effective features can be screened out as much as possible, which helps to improve the accuracy of subsequent quality evaluation.

[0066] In a specific implementation scenario, after selecting the initial features, it can be detected whether there are multiple initial features whose second correlations are not less than the correlation threshold. If so, only one initial feature is retained as a candidate feature, and the step of detecting whether there are multiple initial features whose second correlations are not less than the correlation threshold is returned until the second correlation between any two initial features is less than the correlation threshold. It should be noted that the second correlation can be obtained by measuring correlation coefficients including but not limited to the Pearson correlation coefficient, the Spearman rank correlation coefficient, the Kendall correlation coefficient, etc. The specific process can refer to the technical details of the above correlation coefficients, which will not be repeated here. As a possible example, taking the initial feature A, initial feature B and initial feature C as an example, if the second correlation between initial feature A and initial feature B is not lower than the correlation threshold, the second correlation between initial feature A and initial feature C is not lower than the correlation threshold, and the second correlation between initial feature B and initial feature C is lower than the correlation threshold, then only initial feature B and initial feature C can be retained as candidate features, that is, at this time, the second correlation between any two initial features can satisfy that they are both lower than the correlation threshold; or, if the second correlation between initial feature A and initial feature B is not lower than the correlation threshold, and the second correlation between initial feature B and initial feature C is not lower than the correlation threshold, then only any one of initial feature A, initial feature B and initial feature C can be retained as a candidate feature. Of course, the above examples are only several possible examples in actual application, and do not limit other situations, so they are not given one by one here. The above method, by cyclically detecting whether there are multiple initial features whose second correlations are not lower than the correlation threshold, and selecting the initial features in the process until the second correlation between any two initial features is lower than the correlation threshold, can retain high-quality features with the lowest possible correlation with each other as much as possible, which helps to reduce the computational load of subsequent quality scores.

[0067] In a specific implementation scenario, after obtaining the candidate features, several feature evaluation methods can be used to evaluate the candidate features respectively to obtain the sub-feature scores of the candidate features. Exemplarily, several feature evaluation methods may include but are not limited to: feature evaluation based on tree models (such as random forests, etc.), feature evaluation based on filters (such as chi-square tests, etc.), etc., and the feature evaluation methods are not limited here. On this basis, the sub-feature scores of the candidate features can be fused by each feature evaluation method to obtain the feature scores of the candidate features. Exemplarily, weighting, summation, etc. can be used for fusion. In particular, harmonic mean can also be used for fusion. For the specific process of harmonic mean, please refer to the technical details of harmonic mean, which will not be repeated here. In addition, the embodiment of the present disclosure does not limit the fusion method of sub-feature scores.

[0068] In a specific implementation scenario, after obtaining the feature scores of each candidate feature, the candidate features that are located before a preset rank (e.g., top 5, top 6, etc., or top 10%, top 50%, etc.) can be selected as target features in order of feature scores from high to low.

[0069] In a specific implementation scenario, in order to further ensure the accuracy of the target features selected in the above manner as much as possible, the selected target features can also be used to train a lightweight model, such as combining Lasso regression and tree models to measure feature importance. Specifically, Lasso regression can be used first to obtain feature coefficients as the importance index of the target feature, and cross-validation can be used to tune to find the optimal value, and then the tree model can be used to automatically process feature selection and provide the importance of each target feature. The feature importance obtained from Lasso regression and tree models is then comprehensively analyzed to further eliminate unimportant target features.

[0070] In a specific implementation scenario, in order to reduce the subsequent processing load as much as possible, it is also possible to ensure that key information is retained while reducing feature dimensions and improving model performance. Applying dimensionality reduction methods such as t-SNE and embedded models on important features after feature selection (i.e., the target features that are ultimately retained) can greatly optimize dimensionality reduction performance.

[0071] Step S25: Predictions are made based on the feature representations of the business data with respect to several target features to obtain a quality score for the business data.

[0072] Specifically, after determining the target feature and extracting the feature representation of the business data with respect to the target feature, a prediction can be made based on the feature representation of the business data with respect to the target feature to obtain a quality score of the business data. As a possible example, a neural network can be deployed, and the data quality of the business data can be evaluated in real time through the neural network model. Of course, in this process, the relevant strategies of the neural network model can also be updated, such as regularly updating the clustering algorithm, the network parameters or network structure of the neural network model, the natural language processing model, and the feature selection algorithm to improve the accuracy of redundant detection and removal. In addition, the filtering threshold of the weight in the comprehensive score can be dynamically adjusted according to the actual application situation and data characteristics.

[0073] Step S26: Based on the quality score of the business data, the initial set is filtered to obtain the target set.

[0074] Specifically, after obtaining the quality score of the business data, the business data with a quality score lower than a scoring threshold can be filtered, that is, only the business data with a quality score not lower than the above scoring threshold is retained and included in the target set. In addition, after the above filtering, non-redundant, high-quality business data can be retained. Taking into account that there may be some inconsistent or incomplete data or some data does not belong to a specific field (such as the financial field), these business data that are finally retained can be integrated. Specifically, the integration work can ensure that all data are coordinated and consistent by checking and repairing inconsistencies in the data. It is also possible to manually check the data integrity and whether the data meets the scenario requirements to ensure that there is no erroneous information in the target set and related information that may affect security.

[0075] In the above scheme, a cluster to be screened is selected from the target clusters obtained by clustering a number of business data, and the target cluster is obtained based on the process steps in the above data clustering method embodiment, and then the business data in the cluster to be screened are respectively detected based on several redundancy detection methods to obtain the redundancy score of the business data, and based on the redundancy score of the business data, the cluster to be screened is screened to obtain an initial set, and then the feature representation of the business data in the initial set about several target features is extracted, so as to make a prediction based on the feature representation of the business data about several target features to obtain the quality score of the business data, and then based on the quality score of the business data, the initial set is screened to obtain the target set. Therefore, on the basis of the target cluster obtained by the process steps in the above data clustering method embodiment, a series of operations such as redundancy detection and quality evaluation can be further performed, so as to improve the data quality of the business data in the target set as much as possible.

[0076] See also Figure 3 , Figure 3 This is a flow chart of an embodiment of the intelligent dialogue method of the present application. Specifically, it may include the following steps: Step S31: Obtain the first sentence to be replied.

[0077] In one implementation scenario, the first statement to be replied can be set according to actual application needs. For example, in a financial application scenario, the first statement to be replied can include but is not limited to the following content: "Please evaluate the financial risks of buying XX stock at this time", etc. The specific content of the first statement is not limited here. Of course, in other application scenarios, the first statement can be deduced by analogy, and no examples are given here one by one.

[0078] In one implementation scenario, the first sentence may be input in the form of text, voice, etc., and the input method of the first sentence is not limited herein.

[0079] Step S32: Process the first sentence based on the intelligent dialogue model to obtain a second sentence for replying to the first sentence.

[0080] In the disclosed embodiment, the intelligent dialogue model can be trained based on a sample set, and the sample set is a target set screened by the process steps in the above data screening method embodiment. For details, please refer to the above data screening method embodiment, which will not be repeated here. Of course, the sample set can also be obtained by other methods. For example, the sample set can be obtained by manual collection, etc., and the method for obtaining the sample set is not limited here.

[0081] In an implementation scenario, after obtaining the sample set, you can first divide the sample set into training set and validation set, then select the model according to the task requirements, and then design the model architecture, followed by model verification, model tuning, model evaluation and other operations. Finally, the interface goes online to respond to the first statement in real time, and continuously monitors and maintains it.

[0082] In a specific implementation scenario, as a possible example, the sample set can be divided into a training set, a validation set, and a test set, for example, in a ratio of 8:1:1.

[0083] In a specific implementation scenario, a natural language processing model can be used as an intelligent dialogue model according to task requirements. For example, BERT can be used as an intelligent dialogue model to apply to tasks such as text classification, named entity recognition, and sentiment analysis; or GPT can be used as an intelligent dialogue model to apply to tasks such as generating financial reports and automatically answering financial-related questions; or FinBERT can be used as an intelligent dialogue model to apply to tasks such as financial report analysis and financial news classification.

[0084] In a specific implementation scenario, according to the task requirements, a time series prediction model can be used as an intelligent dialogue model. For example, LSTM can be used as an intelligent dialogue model to apply to the prediction of financial time series data, such as stock prices, trading volumes, etc.; or, GRU can be used as an intelligent dialogue model to apply to the same financial time series prediction task; or, a Transformer-based time series prediction model such as TFT (Temporal Fusion Transformer) can be used as an intelligent dialogue model to apply to complex time series prediction tasks, including multi-dimensional and multi-time scale data.

[0085] In a specific implementation scenario, depending on the task requirements, reinforcement learning models such as DQN (Deep Q-Network) and PPO (Proximal Policy Optimization) can be used as intelligent dialogue models for algorithmic trading and portfolio optimization.

[0086] In a specific implementation scenario, according to task requirements, a graph neural network can be used as an intelligent dialogue model to be suitable for tasks such as analyzing financial networks, social trading networks, credit scoring, and for processing financial problems with graph-structured data.

[0087] In a specific implementation scenario, according to the task requirements, integrated learning models such as XGBoost, LightGBM, CatBoost, etc. can be used as intelligent dialogue models to apply to tasks such as credit scoring, risk assessment, customer classification, etc. They combine multiple simple models to improve prediction performance and robustness.

[0088] In a specific implementation scenario, according to the task requirements, a hybrid model can be used as an intelligent dialogue model to solve complex financial problems. For example, the NLP model is combined with the time series model for the joint task of financial report sentiment analysis and stock price prediction.

[0089] In a specific implementation scenario, according to task requirements, the intelligent dialogue model may also include but is not limited to large language models such as LLAMA and Bloom. In addition, when the intelligent dialogue model adopts a large language model, in addition to the aforementioned open source large models such as LLAMA and Bloom, it can also be a large model obtained after fine-tuning the parameters of the open source large model based on specific corpus, or it can also be a custom large model, which is not limited here.

[0090] In a specific implementation scenario, after determining the model type, you can choose the number of layers and the number of neurons in each layer of the model. Specifically, you can consider the specific problem, data set, resource constraints and other factors, increase the number of layers (depth) or neurons in each layer (width) to increase the expressiveness of the model, and consider that the above operations may increase the demand for more memory and computing power. As a possible example, for image processing, you can consider convolutional neural networks, where the number of layers usually ranges from shallower networks (such as VGG) to deeper networks (such as ResNet), and the number of neurons can be the number of convolution kernels in the convolutional layer and the number of neurons in the fully connected layer; or, for natural language processing, you can consider recurrent neural networks or Transformers, where the number of layers can be set from 1-2 recurrent layers (simple tasks, such as RNN or LSTM) to multi-layer Transformers (complex tasks, such as BERT, GPT), and the number of neurons can be the size of the hidden state and the number of neurons in the fully connected layer; or, for tabular data, you can consider multi-layer perception, where the number of layers can be set between 2-5 layers, and the number of neurons usually ranges from hundreds to thousands, depending on the data size and feature complexity; or, for large data sets, you can consider using deeper and more complex networks, and for small data sets, you can use shallower networks.

[0091] In a specific implementation scenario, during the model training process, appropriate hyperparameters can be set, the model can be iteratively trained, and an early stopping mechanism can be introduced to effectively train a deep learning model and prevent overfitting. In practical applications, hyperparameters can also be adjusted according to specific problems, and the model structure and training strategy can be further improved.

[0092] In a specific implementation scenario, during the model verification process, the validation set can be used to automatically evaluate the model training effect based on test verification indicators such as accuracy, recall, F1 score, AUC, etc.

[0093] In a specific implementation scenario, during the model tuning process, tuning can be performed through methods such as hyperparameter tuning, feature selection, and cross-validation.

[0094] In a specific implementation scenario, during the model evaluation process, the previously retained test set can be used to finally evaluate the model's effect, annotate the analysis and output a report to confirm whether the model can meet our needs and expectations.

[0095] In a specific implementation scenario, after the above series of operations, the intelligent dialogue model can be launched. Based on the actual model effect and performance after the launch, it can be judged whether it can meet customer needs. Based on the problems reported online, the model can be optimized and retrained, and the algorithm can be updated from multiple aspects such as data, redundant detection and filtering technology updates and iterations, model architecture, model parameters and acceptance test sets. In addition, the data and model are updated regularly to ensure that the model adapts to the latest market environment and data changes.

[0096] The above scheme obtains a first sentence to be replied, and processes the first sentence based on the intelligent dialogue model to obtain a second sentence for replying to the first sentence, and the intelligent dialogue model is trained based on a sample set, and the sample set is a target set obtained by screening through the process steps in the above data screening method embodiment. Therefore, the intelligent dialogue model can be trained based on a target set with the highest possible quality, which helps to improve the model performance of the intelligent dialogue model. Therefore, when replying to the first sentence based on the intelligent dialogue model, the accuracy of the intelligent dialogue can be improved, which helps to improve the dialogue satisfaction.

[0097] See also Figure 4 , Figure 4 It is a schematic diagram of the framework of an embodiment of the data clustering device of the present application. The data clustering device 40 includes: a data analysis module 41, which is used to analyze based on the data characteristics of a number of business data to obtain the distribution density and target distance of each business data; wherein the target distance represents the minimum value of the characteristic distance between the business data and each business data with a higher distribution density; a center determination module 42, which is used to determine the cluster center based on the distribution density and target distance of the business data; a data clustering module 43, which is used to cluster the business data based on the cluster center to obtain data clusters; a loop iteration module 44, which is used to return to the step of determining the cluster center based on the distribution density and target distance of the business data based on the business data as the current cluster center in each latest data cluster, until the loop end condition is met, and each latest data cluster is selected as the target cluster.

[0098] In the above scheme, the data clustering device 40 analyzes the data features of several business data to obtain the distribution density and target distance of each business data, and the target distance represents the minimum value of the characteristic distance between the business data and each business data with a higher distribution density, and then determines the cluster center based on the distribution density and target distance of the business data, thereby clustering the business data based on the cluster center to obtain a data cluster, and then returns to the step of determining the cluster center based on the distribution density and target distance of the business data based on the business data as the current cluster center, until the loop end condition is met, that is, each latest data cluster is selected as the target cluster. On the one hand, the cluster center is initially determined by combining the distribution density and the target distance, which helps to improve the accuracy of determining the cluster center. On the other hand, the cluster center is continuously optimized and adjusted during the data clustering process, which helps to improve the accuracy of the data cluster as much as possible. Therefore, the accuracy of data clustering can be improved.

[0099] In some disclosed embodiments, the cluster center is selected from business data that meets data distribution conditions, and the data distribution conditions include: the distribution density is higher than a density threshold, and the target distance is higher than a distance threshold.

[0100] In some disclosed embodiments, the center determination module 42 includes a distribution statistics submodule for obtaining a first data distribution of each distribution density based on the distribution density of each business data, and obtaining a second data distribution of each target distance based on the target distance of each business data; the center determination module 42 includes a first determination submodule for selecting the distribution density as the density threshold based on the first data distribution, and the center determination module 42 includes a second determination submodule for selecting the target distance as the distance threshold based on the second data distribution.

[0101] In some disclosed embodiments, the first determination submodule includes a first ratio acquisition unit, which is used to obtain a first ratio of each distribution density based on the first data distribution; wherein the first ratio of the distribution density represents a first proportion of business data with a lower distribution density in a number of business data; the first determination submodule includes a first threshold determination unit, which is used to select a distribution density as a density threshold based on the first ratio of each distribution density.

[0102] In some disclosed embodiments, the second determination submodule includes a second ratio acquisition unit, which is used to obtain a second ratio of each target distance based on a second data distribution; wherein the second ratio of the target distance represents a second proportion of business data with a lower target distance in a number of business data; the second determination submodule includes a second threshold determination unit, which is used to select the target distance as the distance threshold based on the second ratio of each target distance.

[0103] In some disclosed embodiments, the data clustering module 43 includes an initial clustering submodule, which is used to cluster the business data into the data clusters where the cluster centers are located based on the feature distances between the data features of the business data and the data features of each cluster center; the data clustering module 43 includes a center adjustment submodule, which is used to select some cluster centers as new cluster centers based on the target distances of the cluster centers in each data cluster; the data clustering module 43 includes a re-clustering submodule, which is used to cluster the business data into the data clusters where the new cluster centers are located based on the distribution density of the business data for each business data in the data clusters to which the unselected cluster centers belong.

[0104] In some disclosed embodiments, the center adjustment submodule includes a center sorting unit, which is used to sort the cluster centers in each data cluster in descending order of the target distance of the cluster centers in each data cluster; the center adjustment submodule includes a center selection unit, which is used to select the cluster center whose target distance after sorting is located in the front preset sequence position as the new cluster center.

[0105] In some disclosed embodiments, the re-clustering submodule includes a candidate selection unit for selecting a data cluster with a higher distribution density and located at a new cluster center as a candidate cluster based on the distribution density of the business data; the re-clustering submodule includes a candidate clustering unit for clustering the business data into candidate clusters based on the feature distances between the data features of the business data and the data features of the business data in each candidate cluster.

[0106] In some disclosed embodiments, the data clustering device 40 also includes a cluster evaluation module, which is used to perform cluster evaluation on the latest data cluster based on the cluster evaluation index each time returning to execute the step of determining the cluster center based on the distribution density and target distance of the business data, and obtain a cluster score; wherein the loop end condition includes that the latest cluster score is higher than the score threshold and no longer increases.

[0107] See also Figure 5 , Figure 5It is a schematic diagram of the framework of an embodiment of the data screening device of the present application. The data screening device 50 includes: a target selection module 51, which is used to select a cluster to be screened from the target clusters obtained by clustering a number of business data; wherein the target cluster is obtained based on the above-mentioned data clustering device; a redundancy detection module 52, which is used to detect the business data in the cluster to be screened based on several redundancy detection methods, and obtain the redundancy score of the business data; a preliminary screening module 53, which is used to screen the cluster to be screened based on the redundancy score of the business data to obtain an initial set; a feature extraction module 54, which is used to extract the feature representation of the business data in the initial set about a number of target features; a quality prediction module 55, which is used to predict based on the feature representation of the business data about a number of target features, and obtain the quality score of the business data; a secondary screening module 56, which is used to screen the initial set based on the quality score of the business data to obtain the target set.

[0108] In the above scheme, the data screening device 50 selects a cluster to be screened from the target clusters obtained by clustering a number of business data, and the target cluster is obtained based on the above data clustering device embodiment, and then detects the business data in the cluster to be screened based on several redundancy detection methods to obtain the redundancy score of the business data, and screens the clusters to be screened based on the redundancy score of the business data to obtain an initial set, and then extracts the feature representation of the business data in the initial set with respect to a number of target features, thereby making predictions based on the feature representation of the business data with respect to a number of target features to obtain the quality score of the business data, and then screens the initial set to obtain the target set based on the quality score of the business data. Therefore, on the basis of the target cluster obtained in the above data clustering device embodiment, a series of operations such as redundancy detection and quality evaluation can be further performed, so as to improve the data quality of the business data in the target set as much as possible.

[0109] In some disclosed embodiments, the redundancy detection module 52 includes a score acquisition submodule for acquiring sub-redundancy scores of business data respectively obtained by several redundancy detection methods; the redundancy detection module 52 includes a feature construction submodule for constructing comprehensive features of the business data based on sub-redundancy scores detected by several redundancy detection methods respectively for the business data; the redundancy detection module 52 includes a score prediction submodule for making predictions based on the comprehensive features of the business data to obtain redundancy scores.

[0110] In some disclosed embodiments, in the case where several redundancy detection methods include at least a first redundancy detection based on a neural network model, the redundancy detection module 52 includes a data measurement submodule for obtaining a first correlation between different features of the business data and obtaining the importance of each of the different features of the business data; the redundancy detection module 52 includes a feature extraction submodule for extracting a feature representation of at least one feature for the business data based on the first correlation and the importance; the redundancy detection module 52 includes a first prediction submodule for predicting the feature representation of the business data regarding at least one feature based on the neural network model, and at least obtaining a sub-redundancy score of the business data regarding the first redundancy detection; the redundancy detection module 52 includes a first scoring submodule for obtaining a redundancy score of the business data based at least on the sub-redundancy score of the business data regarding the first redundancy detection.

[0111] In some disclosed embodiments, the redundancy detection module 52 includes a result acquisition submodule, which is used to predict the feature representation of the business data regarding at least one feature based on a neural network model to obtain at least one of a missing detection result, an anomaly detection result, and a security identification result of the business data.

[0112] In some disclosed embodiments, when several redundancy detection methods include at least a second redundancy detection based on business data content, the redundancy detection module 52 includes an association analysis submodule for performing analysis based on the semantic features of the business data to obtain association detection results between different business data; the redundancy detection module 52 includes a graph construction submodule for constructing graph structure data based on the semantic features of the business data and the association detection results between different business data; the redundancy detection module 52 includes a data distinction submodule for generating data distinction results representing whether different business data are similar but different based on the graph structure data; the redundancy detection module 52 includes a second prediction submodule for obtaining a sub-redundancy score of the business data regarding the second redundancy detection based on the similarity between the semantic features of different business data and the data distinction results; the redundancy detection module 52 includes a second scoring submodule for converting at least the sub-redundancy score of the business data regarding the second redundancy detection to a redundancy score of the business data.

[0113] In some disclosed embodiments, the data screening device 50 includes a feature screening module for preliminarily screening preset features as initial features based on feature representations of sample data regarding several preset features; the data screening device 50 includes a feature preliminary selection module for selecting initial features as candidate features based on a second correlation between different initial features in the sample data; the data screening device 50 includes a feature scoring module for fusing sub-feature scores of candidate features based on several feature evaluation methods to obtain feature scores of candidate features; the data screening device 50 includes a feature selection module for selecting candidate features as target features based on feature scores.

[0114] In some disclosed embodiments, the feature screening module includes a feature statistics submodule for obtaining, for various preset features, statistically the variance of the corresponding preset features based on the feature representation of each sample data with respect to the preset features, and for obtaining, based on the variable information of each sample data with respect to the target variable and the feature representation of the preset features, statistically the mutual information between the corresponding preset features and the target variable; the feature screening module includes an initial selection submodule for selecting a preset feature as the initial feature based on the variance and mutual information of each preset feature.

[0115] In some disclosed embodiments, the feature preliminary selection module includes a correlation detection submodule for detecting whether there are multiple initial features whose second correlations are not lower than the correlation threshold; the feature preliminary selection module includes a feature retention submodule for retaining an initial feature as a candidate feature when the second correlations are not lower than the correlation threshold, and returning to the step of detecting whether there are multiple initial features whose second correlations are not lower than the correlation threshold until the second correlation between any two initial features is lower than the correlation threshold.

[0116] See also Figure 6 , Figure 6 It is a schematic diagram of the framework of an embodiment of the intelligent dialogue device of the present application. The intelligent dialogue device 60 includes: a sentence acquisition module 61, which is used to acquire a first sentence to be replied; a sentence reply module 62, which is used to process the first sentence based on the intelligent dialogue model to obtain a second sentence for replying to the first sentence; wherein the intelligent dialogue model is trained based on a sample set, and the sample set is a target set screened by the above-mentioned data screening device.

[0117] In the above scheme, the intelligent dialogue device 60 obtains a first sentence to be replied, and processes the first sentence based on the intelligent dialogue model to obtain a second sentence for replying to the first sentence, and the intelligent dialogue model is trained based on a sample set, and the sample set is a target set screened by the above data screening device embodiment. Therefore, the intelligent dialogue model can be trained based on a target set with the highest possible quality, which helps to improve the model performance of the intelligent dialogue model. Therefore, when replying to the first sentence based on the intelligent dialogue model, the accuracy of the intelligent dialogue can be improved, which helps to improve the dialogue satisfaction.

[0118] See also Figure 7 , Figure 7It is a schematic diagram of the framework of an embodiment of an electronic device 70 of the present application. The electronic device 70 includes a memory 71 and a processor 72 coupled to each other, the memory 71 at least storing program instructions, and the processor 72 is used to execute the program instructions to implement the steps in any of the above-mentioned data clustering method embodiments, or the steps in any of the above-mentioned data screening method embodiments, or the steps in any of the above-mentioned intelligent dialogue method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 70 may include but is not limited to: smart phones, tablet computers, learning machines, office books, servers and other devices, and the specific type of the electronic device 70 is not limited here.

[0119] Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps in any of the above data clustering method embodiments, or the steps in any of the above data screening method embodiments, or the steps in any of the above intelligent dialogue method embodiments. The processor 72 can also be called a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip with signal processing capabilities. The processor 72 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 72 can be implemented by an integrated circuit chip.

[0120] In the above scheme, the electronic device 70 implements the steps in any of the above data clustering method embodiments, or the steps in any of the above data screening method embodiments, or the steps in any of the above intelligent dialogue method embodiments. On the one hand, the cluster center is initially determined by combining the distribution density and the target distance, which helps to improve the accuracy of determining the cluster center. On the other hand, the cluster center is continuously optimized and adjusted during the data clustering process, which helps to improve the accuracy of the data cluster as much as possible. Therefore, the accuracy of data clustering can be improved. In addition, when executing any of the above data screening method embodiments, a series of operations such as redundancy detection and quality evaluation can be further performed on the basis of the target cluster obtained by the aforementioned data clustering method embodiment, which can improve the data quality of the business data in the target set as much as possible. When executing any of the above intelligent dialogue method embodiments, the intelligent dialogue model can be trained based on the target set with the highest quality possible, which helps to improve the model performance of the intelligent dialogue model. Therefore, when replying to the first sentence based on the intelligent dialogue model, the accuracy of the intelligent dialogue can be improved, which helps to improve the dialogue satisfaction.

[0121] See also Figure 8 , Figure 8 Schematic diagram of a framework of an embodiment of a computer-readable storage medium 80 of the present application. The computer-readable storage medium 80 stores program instructions 81 that can be executed by a processor, and the program instructions 81 are used to implement the steps in any of the above data clustering method embodiments.

[0122] In the above scheme, the computer-readable storage medium 80 implements the steps in any of the above data clustering method embodiments, or the steps in any of the above data screening method embodiments, or the steps in any of the above intelligent dialogue method embodiments. On the one hand, the cluster center is initially determined by combining the distribution density and the target distance, which helps to improve the accuracy of determining the cluster center. On the other hand, the cluster center is continuously optimized and adjusted during the data clustering process, which helps to improve the accuracy of the data cluster as much as possible. Therefore, the accuracy of data clustering can be improved. In addition, when executing any of the above data screening method embodiments, a series of operations such as redundancy detection and quality evaluation can be further performed on the basis of the target cluster obtained by the aforementioned data clustering method embodiment, which can improve the data quality of the business data in the target set as much as possible. When executing any of the above intelligent dialogue method embodiments, the intelligent dialogue model can be trained based on the target set with the highest quality possible, which helps to improve the model performance of the intelligent dialogue model. Therefore, when replying to the first sentence based on the intelligent dialogue model, the accuracy of the intelligent dialogue can be improved, which helps to improve the dialogue satisfaction.

[0123] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0124] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0125] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0126] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0127] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0129] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A data clustering method, characterized in that: include: Analyze the data characteristics of several business data to obtain the distribution density and target distance of each of the business data; wherein the target distance represents the minimum value of the characteristic distance between the business data and each of the business data with a higher distribution density; Determining a cluster center based on the distribution density and target distance of the business data; Clustering the business data based on the cluster centers to obtain data clusters; Based on the business data in each of the latest data clusters as the current cluster center, return to execute the step of determining the cluster center based on the distribution density and target distance of the business data until the loop end condition is met, and select each of the latest data clusters as the target cluster.

2. The method according to claim 1, characterized in that The cluster center is selected from business data that meets a data distribution condition, and the data distribution condition includes: the distribution density is higher than a density threshold, and the target distance is higher than a distance threshold.

3. The method according to claim 2, characterized in that The determining of the cluster center based on the distribution density and the target distance of the service data includes: Based on the distribution density of each of the business data, a first data distribution of each of the distribution densities is obtained by statistics, and based on the target distance of each of the business data, a second data distribution of each of the target distances is obtained by statistics; Based on the first data distribution, the distribution density is selected as the density threshold, and based on the second data distribution, the target distance is selected as the distance threshold.

4. The method according to claim 3, characterized in that The selecting the distribution density as the density threshold based on the first data distribution includes: Based on the first data distribution, obtaining a first ratio of each of the distribution densities; wherein the first ratio of the distribution density represents a first proportion of the business data having a lower distribution density in the plurality of business data; Based on a first ratio of each of the distribution densities, the distribution density is selected as the density threshold.

5. The method according to claim 3, characterized in that: The selecting the target distance as the distance threshold based on the second data distribution includes: Based on the second data distribution, obtaining a second ratio of each of the target distances; wherein the second ratio of the target distance represents a second proportion of the business data having a lower target distance in the plurality of business data; Based on a second ratio of each of the target distances, the target distance is selected as the distance threshold.

6. The method according to claim 1, characterized in that The clustering of the business data based on the cluster center to obtain a data cluster includes: Clustering the business data into the data cluster where the cluster center is located based on the feature distances between the data features of the business data and the data features of each cluster center; Based on the target distances of the cluster centers in each of the data clusters, selecting some of the cluster centers as new cluster centers; For each of the business data in the data cluster to which the unselected cluster center belongs, the business data is clustered to the data cluster where the new cluster center is located based on the distribution density of the business data.

7. The method according to claim 6, characterized in that The selecting part of the cluster centers as new cluster centers based on the target distances of the cluster centers in each of the data clusters includes: Sorting the cluster centers in each of the data clusters according to the target distances of the cluster centers in each of the data clusters from large to small; The cluster center whose target distance is located at a front preset rank after sorting is selected as the new cluster center.

8. The method according to claim 6, characterized in that The clustering of the business data to the data cluster where the new cluster center is located based on the distribution density of the business data includes: Based on the distribution density of the business data, selecting a data cluster having a higher distribution density and located at the center of the new cluster as a candidate cluster; The business data is clustered into the candidate clusters based on feature distances between the data features of the business data and the data features of the business data in each of the candidate clusters.

9. A data screening method, characterized in that: include: Selecting a cluster to be screened from a target cluster obtained by clustering a number of business data; wherein the target cluster is obtained based on the data clustering method according to any one of claims 1 to 8; Based on several redundancy detection methods, the business data in the to-be-screened clusters are respectively detected to obtain a redundancy score of the business data; Based on the redundancy score of the business data, the to-be-screened clusters are screened to obtain an initial set; Extracting feature representations of the business data in the initial set with respect to a plurality of target features; Predicting based on the feature representation of the business data with respect to the plurality of target features to obtain a quality score of the business data; Based on the quality score of the business data, the initial set is filtered to obtain a target set.

10. The method according to claim 9, characterized in that In the case where the plurality of redundancy detection methods include at least a first redundancy detection based on a neural network model, the plurality of redundancy detection methods are used to detect the service data in the to-be-screened clusters respectively to obtain a redundancy score for the service data, including: Obtaining a first correlation between different features of the business data, and obtaining the importance of each of the different features of the business data; Extracting a feature representation of at least one feature for the business data based on the first relevance and the importance; Predicting a feature representation of the business data with respect to the at least one feature based on the neural network model, and obtaining at least a sub-redundancy score of the business data with respect to the first redundancy detection; A redundancy score of the service data is obtained based at least on the sub-redundancy score of the service data with respect to the first redundancy detection.

11. The method according to claim 9, characterized in that In the case where the plurality of redundancy detection methods include at least a second redundancy detection based on the content of the service data, the plurality of redundancy detection methods are used to detect the service data in the to-be-screened clusters respectively to obtain a redundancy score for the service data, including: Analyze the semantic features of the business data to obtain association detection results between different business data; Constructing graph structure data based on the semantic features of the business data and the association detection results between different business data; Based on the graph structure data, generating a data distinction result indicating whether different business data are similar but different; Obtaining a sub-redundancy score of the business data with respect to the second redundancy detection based on the similarity between the semantic features of different business data and the data distinction result; At least a sub-redundancy score of the service data with respect to the second redundancy detection is added to the redundancy score of the service data.

12. The method according to claim 9, characterized in that Before extracting feature representations of the business data in the initial set with respect to a plurality of target features, the method further includes: Based on the feature representation of the sample data with respect to a plurality of preset features, preliminarily selecting the preset features as initial features; Based on the second correlation between the sample data and different initial features, selecting the initial features as candidate features; Based on several feature evaluation methods, the sub-feature scores of the candidate feature are respectively integrated to obtain a feature score of the candidate feature; Based on the feature score, the candidate feature is selected as the target feature.

13. An intelligent dialogue method, characterized in that: include: Get the first sentence to be replied; The first sentence is processed based on an intelligent dialogue model to obtain a second sentence for replying to the first sentence; wherein the intelligent dialogue model is trained based on a sample set, and the sample set is a target set screened by the data screening method described in any one of claims 9 to 12.

14. An electronic device, characterized in that: It includes a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor is used to execute the program instructions to implement the data clustering method described in any one of claims 1 to 8, or the data screening method described in any one of claims 9 to 12, or the intelligent dialogue method described in claim 13.

15. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the data clustering method described in any one of claims 1 to 8, or the data screening method described in any one of claims 9 to 12, or the intelligent dialogue method described in claim 13.

Images