Data processing method and device

Abstract

The present application provides a data processing method and device, the method comprising: obtaining various clusters corresponding to the original medical knowledge data set; performing outlier detection processing on the various clusters, if the total number of data points in the cluster is less than a set threshold or the cluster does not contain medical data labeled as medical textbooks, then placing all medical data in the cluster into an outlier candidate set; pruning the outlier candidate set according to the labels of the data points in the outlier candidate set; and performing outlier processing on the data points in the pruned outlier candidate set. The data processing method and device provided by the embodiment of the present invention screens outlier data in the original medical knowledge data through an improved clustering and outlier algorithm specifically suitable for the application scenario, and also uses a pruning algorithm specifically suitable for the data characteristics of the application scenario to ensure the computational efficiency of outlier processing and maximize the retention of available medical knowledge data.

Description

Technical Field

[0001] The present invention relates to the field of information processing, and in particular to a data processing method and device. Background Art

[0002] With the development of big data technologies, the importance of data has become a common consensus in emerging fields such as artificial intelligence. For example, data related to medical knowledge, especially data describing disease symptoms, is often used as corpus data for training artificial intelligence models in the field of intelligent medicine or for building medical knowledge graphs.

[0003] However, the effective application of medical knowledge data must be based on data consistency and standardization. However, existing technologies have sought to enrich the sources of medical knowledge data and build a more robust medical knowledge corpus, using medical textbooks, online articles, electronic medical records, and speech-to-text conversions from medical videos as avenues for capturing medical knowledge data. However, the unstable credibility of internet data, the diverse language expressions in electronic medical records, and the unreliability of speech-to-text conversions from medical videos are inherent flaws in these data sources. In contrast, the medical knowledge data available solely from formal sources such as medical textbooks and reference books is extremely limited and cannot meet the corpus data requirements of robust artificial intelligence models. Therefore, improving the quality of raw medical knowledge data from multiple sources is an urgent problem. Furthermore, in the context of big data, improving the quality of raw medical knowledge data from multiple sources often involves large-scale computational costs. The inherent characteristics of medical knowledge data make general, efficient data processing methods less applicable. Therefore, improving the processing efficiency of raw medical knowledge data is a significant challenge. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a data processing method and device.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0006] Performing initial clustering on the medical data in the acquired original medical knowledge dataset to obtain various clusters corresponding to the original medical knowledge dataset;

[0007] Performing outlier detection processing on each cluster, the outlier detection processing comprising: if the total number of data points in the cluster is less than a set threshold or the cluster does not contain medical data labeled as medical textbooks, then placing all medical data in the cluster into an outlier candidate set; otherwise, sequentially calculating the distance from each data point in the cluster to the cluster center, and placing medical data with a distance greater than the radius and a label other than medical textbooks into the outlier candidate set; wherein the cluster center is the vectorized mean of the medical data in the cluster labeled as medical textbooks;

[0008] Based on a clustering algorithm, the outlier candidate set is pruned according to the labels of the data points in the outlier candidate set to obtain a pruned outlier candidate set;

[0009] Perform outlier processing on the data points in the pruned outlier candidate set to determine whether the data point is an outlier.

[0010] Optionally, the pruning process of the outlier candidate set based on the clustering algorithm according to the labels of the data points in the outlier candidate set specifically includes:

[0011] For data points labeled as online articles, when the distance from the cluster center is the farthest in any round of clustering, the data point is pruned;

[0012] For data points labeled as electronic medical records, when the distance from the cluster center is the farthest in any round of clustering, the data point is vector-recalculated using the medical dictionary. If the distance from the cluster center is reduced, the data point after vector recalculation is retained; otherwise, the data point is pruned.

[0013] For a data point labeled as speech text, when its distance from its cluster center is the farthest in any round of clustering, the ratio of the average distance of other data points of the same family from the cluster center to the distance of the data point to the cluster center is calculated. If the ratio is less than the accuracy of the speech recognition model used to convert the speech text, the data point is pruned.

[0014] Optionally, performing initial clustering on the medical data in the acquired original medical knowledge dataset to obtain various clusters corresponding to the original medical knowledge dataset specifically includes:

[0015] (1) Randomly select k data points from the original medical knowledge dataset as the initial centroids, each centroid corresponds to a cluster;

[0016] (2) For the remaining data points in the original medical knowledge dataset, the distance from each data point to each centroid is calculated one by one, and the cluster corresponding to the centroid closest to the data point is taken as the cluster to which the data point belongs;

[0017] (3) Calculate the mean of the data points in the cluster and use the mean as the new centroid;

[0018] (4) Repeat steps (2) and (3) until the calculated centroid no longer changes, and obtain k clusters corresponding to the data set.

[0019] Optionally, calculating the distance from each data point in the cluster to the cluster center specifically includes:

[0020] Calculate the Euclidean distance, Mahalanobis distance, or Hamming distance from each data point in the cluster to the cluster center.

[0021] Optionally, the method further includes:

[0022] Before performing outlier detection on each cluster, the cluster quality is evaluated and clusters that do not meet the quality requirements are discarded.

[0023] Optionally, performing outlier processing on the data points in the pruned outlier candidate set specifically includes:

[0024] The density-based LOF algorithm performs outlier processing on the data points in the pruned outlier candidate set.

[0025] In a second aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0026] A clustering module, configured to perform initial clustering on the medical data in the acquired original medical knowledge dataset to obtain various clusters corresponding to the original medical knowledge dataset;

[0027] An outlier detection module is configured to perform outlier detection processing on each cluster, the outlier detection processing comprising: if the total number of data points in the cluster is less than a set threshold or the cluster does not contain medical data labeled as a medical textbook, then all medical data in the cluster are placed in an outlier candidate set; otherwise, the distance from each data point in the cluster to the cluster center is calculated in sequence, and medical data with a distance greater than the radius and a label other than a medical textbook are placed in the outlier candidate set; wherein the cluster center is the vectorized mean of the medical data in the cluster labeled as a medical textbook;

[0028] A pruning processing module is used to prune the outlier candidate set based on the labels of the data points in the outlier candidate set based on a clustering algorithm to obtain a pruned outlier candidate set;

[0029] The outlier processing module is used to perform outlier processing on the data points in the pruned outlier candidate set to determine whether the data point is an outlier.

[0030] Optionally, the pruning processing module specifically includes:

[0031] The web article processing module is used to prune the data point labeled as a web article when it is farthest from its cluster center in any round of clustering.

[0032] The electronic medical record processing module is used to recalculate the vector of a data point labeled as an electronic medical record when it is farthest from its cluster center in any round of clustering. If the distance from the cluster center is reduced, the data point after vector recalculation is retained; otherwise, the data point is pruned.

[0033] The speech-to-text processing module is used to calculate the ratio of the average distance of other data points of the same family from the cluster center to the distance of the data point from the cluster center when the data point is farthest from its cluster center in any round of clustering. If the ratio is less than the accuracy of the speech recognition model used to convert the speech text, the data point is pruned.

[0034] Optionally, the clustering module includes:

[0035] The centroid determination module is used to randomly select k data points in the data set as the initial centroids, and each centroid corresponds to a cluster;

[0036] The calculation module is used to calculate the distance from each data point to each centroid for the remaining data points in the data set, and take the cluster corresponding to the nearest centroid as the cluster to which the data point belongs;

[0037] Mean calculation module, used to calculate the mean of data points in the cluster;

[0038] A second centroid determination module is used to use the mean as a new centroid;

[0039] A second calculation module is used to recalculate according to the new centroid until the calculated centroid no longer changes;

[0040] The cluster determination module is used to obtain k clusters corresponding to the data set.

[0041] Optionally, calculating the distance from each data point in the cluster to the cluster center specifically includes:

[0042] Calculate the Euclidean distance, Mahalanobis distance, or Hamming distance from each data point in the cluster to the cluster center.

[0043] The data processing method and device provided by the embodiments of the present invention, based on the respective characteristics of medical knowledge data from multiple sources, screen and process the outlier data in the original medical knowledge data through improved clustering and outlier algorithms specifically suitable for this application scenario, thereby improving the overall quality of the medical knowledge data and facilitating the subsequent use of the data as corpus for training artificial intelligence models in the medical field. In addition, through a pruning algorithm specifically suitable for the data characteristics of this application scenario, the computational efficiency of outlier processing is guaranteed, and the available medical knowledge data is retained to the maximum extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments of the present application.

[0045] Figure 1 A schematic diagram of a data processing method according to an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a pruning method according to an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of a clustering method according to an embodiment of the present invention;

[0048] Figure 4 A schematic structural diagram of a data processing device provided in an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of the structure of a clustering module provided in an embodiment of the present invention;

[0050] Figure 6 A schematic diagram of the structure of a pruning processing module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the solutions of the embodiments of the present invention, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings and implementation methods.

[0052] These and other features and characteristics of the present invention, its method of operation, the functions of the related elements of the structure, the combination of parts, and the economy of manufacture may be better understood with reference to the following description and accompanying drawings, which form a part of this specification. However, it is to be understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present invention. It is to be understood that the drawings are not drawn to scale. Various structural diagrams are used in the present invention to illustrate various variations according to embodiments of the present invention.

[0053] like Figure 1 FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention. The method comprises the following steps:

[0054] Step S101 : performing initial clustering on the medical data in the acquired original medical knowledge dataset to obtain various clusters corresponding to the original medical knowledge dataset.

[0055] In the embodiments of the present invention, medical knowledge data primarily refers to the relevant text data used to build a medical data corpus, such as text describing different symptoms, diagnosing different diseases, or providing instructions for medications for corresponding diseases, etc., which are not specifically limited in the embodiments of the present invention. This data is particularly important in the application scenarios where artificial intelligence and healthcare are combined. It can serve as corpus data for training artificial intelligence models in the field of intelligent healthcare or be used to build medical knowledge graphs.

[0056] In this step, the original medical knowledge dataset is primarily obtained from medical textbooks, online articles, electronic medical records, and speech-to-text conversions from medical videos. While medical textbooks typically present medical knowledge in a standardized manner, the single-minded nature of the presentation results in a relatively small data volume. Data such as online articles, electronic medical records, and speech-to-text conversions from medical videos, however, suffer from large amounts of data but uneven quality due to factors such as the unstable credibility of internet data, the diverse language expressions in electronic medical records, and the unreliability of speech-to-text conversions from medical videos. Any piece of medical knowledge data obtained from any source will be treated as a data point in the dataset composed of the original medical knowledge data. These data points can be vectorized before forming the dataset.

[0057] After acquiring a large dataset of raw medical knowledge data from multiple sources, this step first involves clustering the dataset. The principle behind clustering is that, for example, for symptom description data, data points belonging to the same cluster likely describe the same type of symptoms.

[0058] Step S102, performing outlier detection processing on the various clusters, the outlier detection processing includes: if the total number of data points in the cluster is less than a set threshold or the cluster does not contain medical data labeled as medical textbooks, then all medical data in the cluster are placed in an outlier candidate set; otherwise, the distance from each data point in the cluster to the cluster center is calculated in turn, and medical data with a distance greater than the radius and a label other than medical textbooks are placed in the outlier candidate set; wherein the cluster center is the vectorized mean of the medical data labeled as medical textbooks in the cluster.

[0059] The data processing method provided in this embodiment of the present invention aims to normalize medical knowledge data and address outliers within the data. Outliers are the primary cause of decreased quality in medical knowledge corpora. Prior to outlier treatment, outlier detection must be performed on the clustered medical knowledge data to identify candidate outlier points.

[0060] For medical knowledge data from multiple sources, each piece of knowledge data will be labeled according to its source. The labels can be defined as the following four types: medical textbook labels, which represent data obtained from credible sources such as medical textbooks and reference books through text analysis algorithms; online article labels, which represent data obtained from Internet materials such as medical forums and public accounts through text analysis algorithms; electronic medical record labels, which represent data obtained from electronic medical records created in medical databases and filled out by doctors; and voice text labels, which represent data extracted from medical texts generated by voice recognition models from audio streams of medical podcasts and medical videos. Among them, medical knowledge data labeled as medical textbooks are defaulted as standard data by the embodiment of the present invention, and they serve as a reference in the data processing process of each step of the embodiment of the present invention.

[0061] When performing outlier detection processing in this step, it is first necessary to detect whether each cluster as a whole is a potential outlier. Specifically, if all the data in a cluster does not contain medical data labeled as medical textbooks, then the reliability of the cluster will be greatly reduced. Therefore, it is necessary to place all the data points represented by the medical data in the cluster into the outlier candidate set for further outlier processing. In addition, if the total number of data points in a cluster is small, then the cluster is identified as a weak cluster in the embodiment of the present invention, and the risk of its effect on the corpus data is high. Therefore, it is also necessary to place all the data points represented by the medical data in the cluster into the outlier candidate set for further outlier processing.

[0062] If the cluster as a whole does not meet the above conditions, the cluster will be identified as a valid cluster in the embodiment of the present invention. For valid clusters, it is necessary to further identify potential outliers in the cluster. This is different from the general method. Since the medical knowledge data labeled as medical textbooks are defaulted as standard data by the embodiment of the present invention, it is necessary to recalculate the cluster center based on the medical textbook label and calculate the distance from each data point in the cluster to the cluster center in turn. Specifically, the cluster center is the vectorized mean of the medical data labeled as medical textbooks in the cluster. At this time, the potential outliers need to meet two conditions: first, the distance is greater than the radius of the cluster, indicating that the data point is far away from the medical meaning of the cluster; second, a second limiting condition needs to be added, that is, the label cannot be medical data from a medical textbook. This is to reflect that when describing the same medical knowledge, for example, when describing the same symptom, even standard data from medical textbooks will have two expressions that are quite different in text. Therefore, the purpose of adding the second condition is to prevent high-quality standard data from being accidentally eliminated.

[0063] In the clustering process, for each data point in the data set, the distance from the center of the cluster to which it belongs can be, but is not limited to, any of the following: Euclidean distance, Mahalanobis distance, or Hamming distance.

[0064] According to the above outlier detection process, the weak cluster as a whole or specific data points in the valid cluster can be filtered into the outlier candidate set.

[0065] Step S103: pruning the outlier candidate set to obtain a pruned outlier candidate set.

[0066] Before outlier processing is performed on the generated outlier candidate set, the data volume of the outlier candidate set may be extremely large, especially for the massive amount of raw medical knowledge data collected in the context of big data. In fact, the ratio of valid and invalid data in the outlier candidate set may be very high. Therefore, to ensure the computational efficiency of outlier processing while maximizing the retention of usable data based on the characteristics of medical knowledge data, it is necessary to prune the outlier candidate set before outlier processing.

[0067] In existing pruning methods, the K-means clustering algorithm can be used to prune the outlier candidate set. Embodiments of the present invention can also be based on the K-means clustering algorithm. However, the pruning conditions of existing K-means clustering algorithms are designed based on the data values themselves, without considering the specificity of medical knowledge data. Therefore, using such pruning conditions will result in poor data quality after pruning.

[0068] In this step, since the outlier candidate set only contains data points labeled as online articles, electronic medical records, and voice texts, it is necessary to determine whether the data point should be pruned based on the characteristics of these three types of data.

[0069] The K-means clustering algorithm involves multiple rounds of clustering. Existing algorithms perform pruning identification and processing after clustering is complete. For massive amounts of data, the number of clustering rounds and the amount of computation required for each round can become very large. Therefore, this step performs pruning during the clustering process.

[0070] like Figure 2 As shown in FIG, the specific steps of pruning are as follows.

[0071] Step S201: for a data point labeled as a web article, when its distance from its cluster center is the farthest in any round of clustering, prune the data point;

[0072] Step S202: For a data point labeled as an electronic medical record, when its distance from the cluster center is the farthest during any round of clustering, the data point is vector-recalculated using the medical dictionary. If the distance from the cluster center is reduced, the data point after vector recalculation is retained; otherwise, the data point is pruned.

[0073] Step S203: For a data point labeled as speech text, when its distance from its cluster center is the farthest in any round of clustering, calculate the ratio of the average distance of other data points of the same family from the cluster center to the distance of the data point from the cluster center. If the ratio is less than the accuracy of the speech recognition model used to convert the speech text, prune the data point.

[0074] Specifically, for data points labeled as web articles, if they are farthest from their cluster center during any round of clustering, they are pruned. In this embodiment of the present invention, web articles have the lowest data credibility. Therefore, in each round of clustering, the data point farthest from the cluster center in each cluster is pruned if it is a web article data point. Conversely, web article data points that are not farthest from the cluster center indicate a certain degree of credibility and should be retained for the next round of clustering.

[0075] For data points labeled as electronic medical records, the main reason for the lack of data credibility is that, for example, different doctors use different vocabulary to describe the same symptom or diagnosis in electronic medical records. Synonymous or near-synonymous terms are common in medical knowledge, but the data vectors show a large gap between the two data points. Therefore, such cases need to be excluded during outlier detection. When a data point labeled as an electronic medical record is farthest from its cluster center during any round of clustering, it cannot be simply pruned as with online articles. Instead, the medical knowledge data represented by the data point needs to be replaced with synonyms or near-synonyms using a medical dictionary tool. Then, it is necessary to re-evaluate whether the distance of the data point from the cluster center has decreased. If it has decreased, it indicates that the results of this round of clustering have been affected by synonyms or near-synonymous medical terms. The data point should not be identified as an outlier and pruned, but should be retained. At the same time, the synonyms or near-synonymous medical terms that reduce the distance of the data point from the cluster center need to be replaced with the original terms before entering the next round of clustering calculations. On the contrary, if the distance from the cluster center remains unchanged or even increases regardless of any synonymous or near-synonymous medical word replacement, it means that the data point needs to be pruned.

[0076] Data points labeled as speech text require different considerations when making pruning decisions. With the increasing availability of voice and video streams, medical knowledge data extracted from medical text generated by speech recognition models in the audio streams of medical podcasts and medical videos is also an important source of medical knowledge data. However, due to the varying quality of these resources, higher requirements are placed on speech recognition models. Insufficient speech recognition model capabilities can lead to the generated medical knowledge data not matching the actual content of the original video / audio. For data points labeled as speech text, when their distance from the cluster center is the farthest during any clustering round, a tolerance is set for this data point. The value of this tolerance is closely related to the accuracy of the speech recognition model. Specifically, the ratio of the average distance of other similar data points from the cluster center to the distance of the data point from the cluster center can be used. If this ratio is less than the accuracy of the speech recognition model used to convert the speech text, the outlier level of the data point exceeds the tolerance for this type of data and requires pruning.

[0077] The above-mentioned process of pruning data points in the outlier candidate set is different from the existing method of performing a unified pruning operation based on the basic numerical values of the data points after clustering. Instead, it sets specific judgment conditions for different types of data points that may be pruned based on the characteristics of medical knowledge data with different labels, thereby ensuring the computational efficiency of outlier processing and retaining the available medical knowledge data to the maximum extent.

[0078] Step S104 , performing outlier processing on the data points in the pruned outlier candidate set to determine whether the data points are outliers.

[0079] Specifically, the LOF algorithm may be used to perform outlier processing on the data points in the outlier candidate set. Figure 3 A flow chart of clustering data in a data set according to an embodiment of the present invention is shown, including the following steps:

[0080] Step S301: randomly select k data points from the original medical knowledge dataset as initial centroids, each centroid corresponding to a cluster;

[0081] Step S302: For the remaining data points in the original medical knowledge dataset, the distance from each data point to each centroid is calculated one by one, and the cluster corresponding to the centroid closest to the data point is taken as the cluster to which the data point belongs;

[0082] Step S303, calculating the mean of the data points in the cluster and using the mean as the new centroid;

[0083] Step S304: Repeat steps S302 and S303 until the calculated centroid no longer changes, and obtain k clusters corresponding to the data set.

[0084] Generally, denser clusters have higher cluster quality. However, the average distance between point pairs in cluster A is smaller than the average distance between point pairs in cluster B. This results in the erroneous judgment that sparse cluster A is of higher quality than dense cluster B. Experiments have shown that this misjudgment occurs because the average distance between point pairs in larger clusters is considered when calculating the average distance. When a point is added to a cluster, the average distance between that point and all other points in the cluster is calculated. This result is very large, causing the evaluation results to deviate from the actual results.

[0085] To this end, in another non-limiting embodiment of the method of the present invention, before performing outlier detection processing on each cluster, the clusters can be subjected to quality assessment, and clusters that do not meet the quality requirements can be discarded. In other words, the various clusters obtained by clustering are first screened out through quality assessment to better ensure data quality and further reduce the workload of subsequent calculations. In practical applications, a density-based LOF algorithm can be used to perform outlier processing on the data points in the outlier candidate set, or other outlier processing algorithms can be used, which are not specifically limited in the embodiments of the present invention.

[0086] The data processing method provided by the embodiment of the present invention is based on the respective characteristics of medical knowledge data from multiple sources. It uses improved clustering and outlier algorithms specifically suitable for this application scenario to screen and process outlier data in the original medical knowledge data, thereby improving the overall quality of the medical knowledge data and facilitating the subsequent use of the data as corpus for training artificial intelligence models in the medical field. It also uses a pruning algorithm specifically suitable for the data characteristics of this application scenario to ensure the computational efficiency of outlier processing and maximize the retention of available medical knowledge data.

[0087] Accordingly, an embodiment of the present invention further provides a data processing device, such as Figure 4 The figure shows a structural diagram of the data processing device.

[0088] In this embodiment, the data processing device includes the following modules:

[0089] Clustering module 401 is used to perform initial clustering on the medical data in the acquired original medical knowledge dataset to obtain various clusters corresponding to the original medical knowledge dataset;

[0090] Outlier detection module 402 is configured to perform outlier detection on each cluster. The outlier detection process includes: if the total number of data points in the cluster is less than a set threshold or the cluster does not contain medical data labeled as medical textbooks, then all medical data in the cluster are placed in an outlier candidate set; otherwise, the distance between each data point in the cluster and the cluster center is calculated in sequence, and medical data with a distance greater than the radius and a label other than medical textbooks are placed in an outlier candidate set; wherein the cluster center is the vectorized mean of the medical data in the cluster labeled as medical textbooks;

[0091] A pruning processing module 403 is configured to prune the outlier candidate set based on the labels of the data points in the outlier candidate set based on a clustering algorithm to obtain a pruned outlier candidate set;

[0092] The outlier processing module 404 is configured to perform outlier processing on the data points in the pruned outlier candidate set to determine whether the data points are outliers.

[0093] Furthermore, if Figure 5 As shown, a specific structure of the clustering module 401 may include the following modules:

[0094] A centroid determination module 501 is used to randomly select k data points in the data set as initial centroids, each centroid corresponding to a cluster;

[0095] A calculation module 502 is used to calculate the distance from each data point to each centroid for the remaining data points in the data set, and take the cluster corresponding to the centroid closest to the data point as the cluster to which the data point belongs;

[0096] Mean calculation module 503, used to calculate the mean of the data points in the cluster;

[0097] A second centroid determination module 504 is configured to use the mean as a new centroid;

[0098] A second calculation module 505 is configured to recalculate based on the new centroid until the calculated centroid no longer changes;

[0099] The cluster determination module 506 is configured to obtain k clusters corresponding to the data set.

[0100] like Figure 6 As shown, the pruning processing module 403 specifically includes:

[0101] The web article processing module 601 is configured to prune a data point labeled as a web article when the data point is farthest from the cluster center in any round of clustering.

[0102] The electronic medical record processing module 602 is configured to, for a data point labeled as an electronic medical record, recalculate the vector of the data point using the medical dictionary when the data point is farthest from the cluster center during any round of clustering. If the distance from the cluster center is reduced, the data point after the vector recalculation is retained; otherwise, the data point is pruned.

[0103] The speech text processing module 603 is used to calculate the ratio of the average distance of other data points of the same family from the cluster center to the distance of the data point from the cluster center when the data point is farthest from its cluster center in any round of clustering. If the ratio is less than the accuracy of the speech recognition model used to convert the speech text, the data point is pruned.

[0104] The specific implementation of the modules and units in the above embodiments can be referred to the description in the above method embodiments of the present invention, which will not be repeated here.

[0105] It should be noted that the embodiments described above are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0106] It should be noted that, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, terms such as "first" and "second" are used to distinguish between identical or similar items having substantially the same function or effect. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or order of execution. For example, the first information and the second information are used to distinguish different information, rather than to describe a specific order of information.

[0107] It should be noted that, in the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0108] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). The storage medium can also include a combination of the above-mentioned types of memory.

[0109] As used in this application, the terms "component", "module", "system" and the like are intended to refer to a computer-related entity, which can be hardware, firmware, a combination of hardware and software, software, or software in operation. For example, a component can be, but is not limited to: a process running on a processor, a processor, an object, an executable file, a thread in execution, a program and / or a computer. As an example, both an application running on a computing device and the computing device can be a component. One or more components can exist in an executing process and / or thread, and a component can be located in a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures thereon. These components can communicate in the form of local and / or remote processes, such as based on signals having one or more data packets (e.g., data from a component that interacts with another component in a local system, a distributed system, and / or interacts with other systems in the form of signals over a network such as the Internet).

[0110] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Performing initial clustering on the medical data in the acquired original medical knowledge dataset to obtain various clusters corresponding to the original medical knowledge dataset; Performing outlier detection processing on each cluster, the outlier detection processing comprising: if the total number of data points in the cluster is less than a set threshold or the cluster does not contain medical data labeled as medical textbooks, then placing all medical data in the cluster into an outlier candidate set; otherwise, sequentially calculating the distance from each data point in the cluster to the cluster center, and placing medical data with a distance greater than the radius and a label other than medical textbooks into the outlier candidate set; wherein the cluster center is the vectorized mean of the medical data in the cluster labeled as medical textbooks; Based on a clustering algorithm, the outlier candidate set is pruned according to the labels of the data points in the outlier candidate set to obtain a pruned outlier candidate set; Perform outlier processing on the data points in the pruned outlier candidate set to determine whether the data point is an outlier; The clustering algorithm is used to prune the outlier candidate set according to the labels of the data points in the outlier candidate set, specifically including: For data points labeled as online articles, when the distance from the cluster center is the farthest in any round of clustering, the data point is pruned; For data points labeled as electronic medical records, when the distance from the cluster center is the farthest in any round of clustering, the data point is vector-recalculated using the medical dictionary. If the distance from the cluster center is reduced, the data point after vector recalculation is retained; otherwise, the data point is pruned. For a data point labeled as speech text, when its distance from its cluster center is the farthest in any round of clustering, the ratio of the average distance of other data points of the same family from the cluster center to the distance of the data point to the cluster center is calculated. If the ratio is less than the accuracy of the speech recognition model used to convert the speech text, the data point is pruned.

2. The method according to claim 1, wherein The initial clustering of the medical data in the acquired original medical knowledge dataset to obtain various clusters corresponding to the original medical knowledge dataset specifically includes: (1) Randomly select k data points from the original medical knowledge dataset as the initial centroids, each centroid corresponds to a cluster; (2) For the remaining data points in the original medical knowledge dataset, the distance from each data point to each centroid is calculated one by one, and the cluster corresponding to the centroid closest to the data point is taken as the cluster to which the data point belongs; (3) Calculate the mean of the data points in the cluster and use the mean as the new centroid; (4) Repeat steps (2) and (3) until the calculated centroid no longer changes, and obtain k clusters corresponding to the data set.

3. The method according to claim 1, wherein Calculating the distance between each data point in the cluster and the cluster center specifically includes: Calculate the Euclidean distance, Mahalanobis distance, or Hamming distance from each data point in the cluster to the cluster center.

4. The method according to claim 1, wherein The method further comprises: Before performing outlier detection on each cluster, the cluster quality is evaluated and clusters that do not meet the quality requirements are discarded.

5. The method according to claim 1, wherein The outlier processing of the data points in the pruned outlier candidate set specifically includes: The density-based LOF algorithm performs outlier processing on the data points in the pruned outlier candidate set.

6. A data processing device, characterized in that: The device comprises: A clustering module, configured to perform initial clustering on the medical data in the acquired original medical knowledge dataset to obtain various clusters corresponding to the original medical knowledge dataset; An outlier detection module is configured to perform outlier detection processing on each cluster, the outlier detection processing comprising: if the total number of data points in the cluster is less than a set threshold or the cluster does not contain medical data labeled as a medical textbook, then all medical data in the cluster are placed in an outlier candidate set; otherwise, the distance from each data point in the cluster to the cluster center is calculated in sequence, and medical data with a distance greater than the radius and a label other than a medical textbook are placed in the outlier candidate set; wherein the cluster center is the vectorized mean of the medical data in the cluster labeled as a medical textbook; A pruning processing module is used to prune the outlier candidate set based on the labels of the data points in the outlier candidate set based on a clustering algorithm to obtain a pruned outlier candidate set; An outlier processing module is used to perform outlier processing on the data points in the pruned outlier candidate set to determine whether the data point is an outlier; The pruning processing module specifically includes: The web article processing module is used to prune the data point labeled as a web article when it is farthest from its cluster center in any round of clustering. The electronic medical record processing module is used to recalculate the vector of a data point labeled as an electronic medical record when it is farthest from its cluster center in any round of clustering. If the distance from the cluster center is reduced, the data point after vector recalculation is retained; otherwise, the data point is pruned. The speech-to-text processing module is used to calculate the ratio of the average distance of other data points of the same family from the cluster center to the distance of the data point from the cluster center when the data point labeled as speech text is farthest from its cluster center in any round of clustering. If the ratio is less than the accuracy of the speech recognition model used to convert the speech text, the data point is pruned.

7. The device according to claim 6, characterized in that The clustering module includes: The centroid determination module is used to randomly select k data points in the data set as the initial centroids, and each centroid corresponds to a cluster; The calculation module is used to calculate the distance from each data point to each centroid for the remaining data points in the data set, and take the cluster corresponding to the nearest centroid as the cluster to which the data point belongs; Mean calculation module, used to calculate the mean of data points in the cluster; A second centroid determination module is used to use the mean as a new centroid; A second calculation module is used to recalculate according to the new centroid until the calculated centroid no longer changes; The cluster determination module is used to obtain k clusters corresponding to the data set.

8. The device according to claim 6, wherein Calculating the distance between each data point in the cluster and the cluster center specifically includes: Calculate the Euclidean distance, Mahalanobis distance, or Hamming distance from each data point in the cluster to the cluster center.

Images