A semi-supervised clustering method and its open-ended question answering text encoding method

Abstract

This invention relates to the field of data representation technology, and discloses a semi-supervised clustering method and its open-ended question answer text encoding method. The semi-supervised clustering method includes: acquiring a dataset to be clustered, its labeled dataset, and its unlabeled dataset; mapping the dataset to be clustered to a spatial density map and / or a topological density map; and using the labeled and unlabeled datasets to cluster the data in the dataset to be clustered into several clusters, wherein each data point in each cluster has a cluster label, which can be an existing label or a new label. This invention uses a semi-supervised clustering method to efficiently and accurately cluster open-ended question answer text data, and can discover new classes with limited prior knowledge. After clustering, keywords can be extracted and encoded from the open-ended question answer text of each class, facilitating a quick and accurate understanding of the situation of the interviewed group and improving the efficiency and quality of diagnosis and treatment.

Description

Technical Field

[0001] This invention relates to the field of data representation technology, and in particular to a semi-supervised clustering method and its open-ended question answering text encoding method. Background Technology

[0002] Semi-supervised clustering has gradually become a research hotspot in fields such as machine learning. Its applications span multiple areas including road detection, image classification, information retrieval, speech recognition, bioinformatics, and healthcare. For example, in healthcare, medical staff often provide patients with open-ended questions. The responses to these questions vary widely. Semi-supervised clustering methods can be used to cluster these responses, reducing the workload of medical staff, avoiding errors, improving the efficiency and quality of processing the responses, and helping them accurately understand the patient population. However, most existing semi-supervised clustering methods (such as DBSCAN, OPTICS, DenPeak, and DENCLUE) only represent density based on spatial distance or topological relationships. They perform poorly on datasets with large density variations and can only expand based on given prior labels, failing to discover unlabeled new categories.

[0003] Therefore, there is an urgent need for a new semi-supervised clustering method and its open-ended question answering text encoding method. Summary of the Invention

[0004] This invention provides a semi-supervised clustering method to address the shortcomings of existing semi-supervised clustering methods, which suffer from poor clustering accuracy.

[0005] This invention provides a method for encoding open-ended question responses, which uses a novel semi-supervised clustering method to achieve rapid encoding of open-ended question responses, thereby helping to accurately understand the true thoughts of the respondents and improving the efficiency and quality of data processing.

[0006] This invention provides a semi-supervised clustering method, comprising:

[0007] Obtain the dataset to be clustered, as well as the labeled and unlabeled datasets obtained based on the dataset to be clustered;

[0008] Map the dataset to be clustered to a spatial density map and / or a topological density map;

[0009] Based on the spatial density map and / or topological density map, using the labeled and unlabeled datasets, the data in the dataset to be clustered are clustered into several clusters. Each data in each cluster has a cluster label, which is either the label of any data in the labeled dataset or a new label that is different from the labels of all data in the labeled dataset.

[0010] According to a semi-supervised clustering method provided by the present invention, the step of mapping the dataset to be clustered into a spatial density map and / or a topological density map includes:

[0011] For any data point i in the clustering dataset, obtain its spatial neighbor set. and topological neighbor set ;

[0012] Based on the spatial neighbor set and topological neighbor set of each data point in the dataset to be clustered, for any two data points i and j in the dataset to be clustered, determine whether the two data points i and j are adjacent and their adjacency type, and obtain the density path set between the two data points i and j. Also, determine whether the two data points i and j are density reachable and their density reachable type, and obtain the density distance between the two data points i and j.

[0013] Based on the density distance between any two data points i and j in the dataset to be clustered, the dataset to be clustered is mapped to a spatial density map and / or a topological density map.

[0014] According to the semi-supervised clustering method provided by the present invention, spatial neighbor set The expression is:

[0015] ,

[0016] In the formula, This represents the subset of data in the dataset to be clustered, centered at any data point i and with a radius of d. This represents the distance between data point i and data point j in the dataset to be clustered;

[0017] Topological neighbor set The expression is:

[0018] ,

[0019] In the formula, This represents the k nearest neighbors of any data point i in the dataset to be clustered. This represents the distance between data point i and data point j in the dataset to be clustered. This represents the distance between data point i and its k-th nearest neighbor.

[0020] According to a semi-supervised clustering method provided by the present invention, based on the spatial neighbor set and topological neighbor set of each data point in the dataset to be clustered, for any two data points i and j in the dataset to be clustered, it is determined whether the two data points i and j are adjacent and their adjacency type, and the density path set between the two data points i and j is obtained; and it is determined whether the two data points i and j are density reachable and their density reachability type, and the density distance between the two data points i and j is obtained, including:

[0021] Based on the spatial neighbor set and topological neighbor set of each data point in the dataset to be clustered, and according to the adjacency determination criteria, for any two data points i and j in the dataset to be clustered, determine whether the two data points i and j are adjacent and their adjacency type. The adjacency determination criteria include: for any two data points i and j in the dataset to be clustered, if... Then determine if data point i is spatially adjacent to data point j, denoted as... ,like Then determine the topological adjacent data point j of data point i, denoted as . ;

[0022] Based on the spatial and topological neighbor sets of each data point in the dataset to be clustered, and according to the density path determination criteria, for any two data points i and j in the dataset to be clustered, the density path set between any two data points i and j is obtained. The density path determination criteria include: for any two data points i and j in the dataset to be clustered, if there exists a sequence of data points... , making ,and Then it is called Let i be a spatial density path from data point i to data point j. For any two data points i and j in the dataset to be clustered, if there exists a sequence of data points... , making ,and Then determine Let be a topological density path from data point i to data point j, and let be the set of spatial density paths from data point i to data point j in the dataset to be clustered. Let the spatial density path set from data point i to data point j in the dataset to be clustered be denoted as . ;

[0023] Based on the density path set between any two data points i and j in the dataset to be clustered, and according to the density reachability criteria, determine whether any two data points i and j in the dataset to be clustered are density reachable and what the density reachability type is. The density reachability criteria include: if the spatial density path set from data point i to data point j in the dataset to be clustered is not empty, then data point i is determined to be spatially density reachable from data point j, denoted as... If the set of topological density paths from data point i to data point j is not empty, then data point i is topologically reachable from data point j, denoted as . ;

[0024] Based on the spatial and topological neighbor sets of each data point in the dataset to be clustered, as well as the adjacency and adjacency type, density path set, density reachability, and density reachability type between any two data points i and j in the dataset, the density distance between any two data points i and j is obtained according to the hyperparameters d and k. This density distance includes the spatial density distance. and topological density distance Spatial density distance The expression is: In the formula, The topological density distance represents the minimum d that makes the spatial density of data point i reach that of data point j. The expression is: In the formula, Make the topological density of data point i reach the minimum k of data point j.

[0025] According to a semi-supervised clustering method provided by the present invention, the expression for the spatial density map is as follows:

[0026] ,

[0027] In the formula, Representing spatial density map vertex set Representing spatial density map Edge weights, adjacency matrix , Represents the vertices in the graph. Represents an edge;

[0028] Topology density map The expression is:

[0029] ,

[0030] In the formula, Representing the topology density map The vertex set, Representing the topology density map Edge weights, adjacency matrix , Represents the vertices in the graph. Represents an edge.

[0031] According to a semi-supervised clustering method provided by the present invention, the step of clustering data in the dataset to be clustered into several clusters based on a spatial density map and / or a topological density map, using labeled and unlabeled datasets, includes:

[0032] Based on spatial density maps or topological density maps, initialization, graph partitioning, and label propagation are performed using labeled and unlabeled datasets to cluster the data in the dataset to be clustered into several clusters; or,

[0033] Based on the spatial density map and the topological density map, the data in the dataset to be clustered are clustered into several clusters by using labeled and unlabeled datasets for initialization, graph segmentation, graph merging, and label propagation.

[0034] According to a semi-supervised clustering method provided by the present invention, the step of initializing, graph segmenting, graph merging, and label propagation using labeled and unlabeled datasets based on spatial density maps and topological density maps, and clustering the data in the dataset to be clustered into several clusters includes:

[0035] The spatial density map and topological density map are initialized using labeled and unlabeled datasets as the total sample set;

[0036] By combining the labeled dataset, the same loop operation is performed on the initialized spatial density map and topological density map to obtain the segmented spatial density map and topological density map: before reaching the preset stopping condition, the longest edge of the density map is cut in a loop to divide the density map into multiple connected subgraphs. Each subgraph has only two possibilities: one is that the subgraph is a new cluster without prior label, and the other is that the subgraph belongs to the cluster with its unique value label.

[0037] The split spatial density graph and topological density graph are merged into a new graph, where the vertex set of the new graph is the union of the vertex sets of the spatial density graph and the topological density graph, and the edge set of the new graph is the intersection of the edge sets of the spatial density graph and the topological density graph.

[0038] Based on the label propagation conditions, cluster labels are assigned to all data points in the new graph. The label propagation conditions include: for a subgraph with a unique label value, all data points in the subgraph are assigned the unique label value as the cluster label; for a subgraph where all data points are unlabeled, a new label different from any label in the labeled dataset is selected as the cluster label, and all data points in the subgraph are assigned the new label.

[0039] This invention also provides a text encoding method for open-ended question answers, comprising:

[0040] Obtain the open-ended question answer text dataset to be encoded, as well as the labeled and unlabeled open-ended question answer text datasets obtained based on the open-ended question answer text dataset to be encoded. The labels of the data in the labeled open-ended question answer text dataset are the answer types.

[0041] Based on the open-ended question answer text dataset to be encoded, the labeled open-ended question answer text dataset, and the unlabeled open-ended question answer text dataset, the open-ended question answer text dataset to be encoded is clustered into several clusters using any of the semi-supervised clustering methods described above, wherein each open-ended question answer text in each cluster has a cluster label;

[0042] Encode each open-ended question answer text in the dataset based on the clustering label of each open-ended question answer text in the dataset to be encoded.

[0043] This invention also provides an open-ended question answer text encoding system, comprising:

[0044] The data acquisition module is used to: acquire the open-ended question answer text dataset to be encoded, as well as the labeled open-ended question answer text dataset and the unlabeled open-ended question answer text dataset obtained based on the open-ended question answer text dataset to be encoded, wherein the label of the data in the labeled open-ended question answer text dataset is the answer type;

[0045] The clustering module is used to: based on the open-ended question answer text dataset to be encoded, the labeled open-ended question answer text dataset, and the unlabeled open-ended question answer text dataset, cluster the open-ended question answer text dataset to be encoded into several clusters using the semi-supervised clustering method described above, wherein each open-ended question answer text in each cluster has a cluster label;

[0046] The encoding module is used to encode each open-question answer text in the open-question answer text dataset based on the clustering label of each open-question answer text in the dataset to be encoded.

[0047] The present invention also provides an electronic device, including a processor and a memory storing a computer program, wherein the processor, when executing the computer program, implements any of the above-described semi-supervised clustering methods and / or open-ended question answering text encoding methods.

[0048] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described semi-supervised clustering methods and / or open-ended question answer text encoding methods.

[0049] The present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is capable of executing any of the semi-supervised clustering methods and / or open-ended question answer text encoding methods described above.

[0050] This invention provides a semi-supervised clustering method that integrates topological distance and spatial distance, defining a new density distance to overcome the limitations of existing methods that only consider one aspect of topological or spatial distance. The new density distance can more comprehensively reflect the density attributes of points and performs better on data with uneven density. During cluster analysis, not only can spatial density maps and topological density maps be combined, but only one of them can also be used, allowing for more flexible and less restrictive application depending on the specific characteristics of the dataset. Compared with the existing semi-supervised density clustering spanning tree method, the graph partitioning clustering method has lower time complexity and higher efficiency, fundamentally solving the problem that the spanning tree method cannot discover new classes.

[0051] This invention provides an open-ended question answer text encoding method. Based on a novel semi-supervised clustering method, it performs efficient and high-precision clustering analysis on open-ended question answer text data. It can discover new classes with only a small amount of prior knowledge. After accurately clustering the open-ended question answer text data, keywords can be extracted and encoded for each class of open-ended question answer text, which facilitates a quick and accurate understanding of the situation of the interviewed group and improves data processing efficiency and quality. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating a semi-supervised clustering method provided by the present invention.

[0054] Figure 2 The image shows the visualization of the geometry dataset and the clustering effect of a semi-supervised clustering method provided by this invention. Gray dots represent unlabeled data, and colored dots represent labeled data.

[0055] Figure 3 The visualization of the Rings dataset and the clustering effect of a semi-supervised clustering method provided by this invention are shown. Gray dots represent unlabeled data, and colored dots represent labeled data.

[0056] Figure 4 This is a flowchart illustrating an open-ended question answering text encoding method provided by the present invention.

[0057] Figure 5 This is a schematic diagram of the structure of an open-ended question answering text encoding system provided by the present invention.

[0058] Figure 6This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, embodiments of this invention, and should not be construed as limiting the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. In the description of this invention, it should be understood that the terminology used is for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0060] Figure 1 This is a flowchart illustrating a semi-supervised clustering method provided by the present invention. The execution entity of the semi-supervised clustering method provided by the present invention can be any applicable terminal-side device or network-side device, such as a semi-supervised clustering apparatus.

[0061] See Figure 1 The semi-supervised clustering method provided by this invention may include:

[0062] S110. Obtain the dataset to be clustered, as well as the labeled and unlabeled datasets obtained based on the dataset to be clustered.

[0063] The dataset to be clustered can be specifically defined according to the specific application scenario. For example, the dataset to be clustered may be text data, audio data, image data, etc. In this embodiment, the dataset to be clustered is denoted as point set D. The labeled dataset can be a small portion of the data in the dataset to be clustered that has been randomly selected and manually labeled, and the remaining data is the unlabeled dataset.

[0064] S120. Map the dataset to be clustered to a spatial density map and / or a topological density map.

[0065] In one embodiment, S120 may include:

[0066] S1201. For any data point i in the dataset to be clustered, obtain its spatial neighbor set. Topological neighbor set .

[0067] Among them, the spatial neighbor set The expression is:

[0068] ,

[0069] In the formula, This represents the subset of data in the dataset to be clustered, centered at any data point i and with a radius of d. This represents the distance between data point i and data point j in the dataset to be clustered;

[0070] Topological neighbor set The expression is:

[0071] ,

[0072] In the formula, This represents the k nearest neighbors of any data point i in the dataset to be clustered. This represents the distance between data point i and data point j in the dataset to be clustered. This represents the distance between data point i and its k-th nearest neighbor.

[0073] In the definitions of spatial neighbor set and topological neighbor set, the spatial neighbor set SN of a point starts from the distance metric in space to find the set of points within a certain distance of point i; the topological neighbor set TN of a point draws on the idea of ​​k-nearest neighbors to find the k-nearest neighbors of point i based on the nearest neighbor adjacency relationship of the point. The two definitions are dual to each other, starting from the perspectives of spatial distance and topological adjacency respectively, and reflecting the density of the area around point i in the form of neighborhood.

[0074] It should be noted that this embodiment refers to the distance between two points. The metric method is not restricted and can be flexibly selected based on the dataset as an adjustable parameter. Spatial neighbor set With topological neighbor set The magnitude of varies with its hyperparameters d and k, respectively.

[0075] S1202. Based on the spatial neighbor set and topological neighbor set of each data point in the dataset to be clustered, for any two data points i and j in the dataset to be clustered, determine whether the two data points i and j are adjacent and their adjacency type, and obtain the density path set between the two data points i and j. Also, determine whether the two data points i and j are density reachable and their density reachability type, and obtain the density distance between the two data points i and j.

[0076] Specifically, S1202 may include:

[0077] Based on the spatial neighbor set and topological neighbor set of each data point in the dataset to be clustered, and according to the adjacency determination criteria, for any two data points i and j in the dataset to be clustered, determine whether the two data points i and j are adjacent and their adjacency type. The adjacency determination criteria include: for any two data points i and j in the dataset to be clustered, if... Then determine if data point i is spatially adjacent to data point j, denoted as... ,like Then determine the topological adjacent data point j of data point i, denoted as . (This is an explanatory supplement to the relationship between two points based on spatial neighbor sets and topological neighbor sets. Here, it is indicated that spatial adjacency has symmetry (i.e., for any two points i and j in the point set D, if...) but (And the topological neighbor set TN does not have symmetry).

[0078] Based on the spatial and topological neighbor sets of each data point in the dataset to be clustered, and according to the density path determination criteria, for any two data points i and j in the dataset to be clustered, the density path set between any two data points i and j is obtained. The density path determination criteria include: for any two data points i and j in the dataset to be clustered, if there exists a sequence of data points... , making ,and Then it is called Let i be a spatial density path from data point i to data point j. For any two data points i and j in the dataset to be clustered, if there exists a sequence of data points... , making ,and Then determine Let be a topological density path from data point i to data point j, and let be the set of spatial density paths from data point i to data point j in the dataset to be clustered. Let the spatial density path set from data point i to data point j in the dataset to be clustered be denoted as . (A density path starts from point i and reaches the next point by passing through the adjacency relationship between two points, eventually reaching point j. It is called a density path because the adjacency relationship between two points is derived from the spatial and topological neighbor set, and the neighbor set itself reflects the density properties around the point. Therefore, the extension of the path is based on the density standard.)

[0079] Based on the density path set between any two data points i and j in the dataset to be clustered, and according to the density reachability criteria, determine whether any two data points i and j in the dataset to be clustered are density reachable and what the density reachability type is. The density reachability criteria include: if the spatial density path set from data point i to data point j in the dataset to be clustered is not empty, then data point i is determined to be spatially density reachable from data point j, denoted as... If the set of topological density paths from data point i to data point j is not empty, then data point i is topologically reachable from data point j, denoted as . (The spatial symmetry and topological asymmetry in the definition of two-point adjacency are transferred to the definition of density reachability here, i.e., if there are Then there is But if there is ,but (Not necessarily true)

[0080] Based on the spatial and topological neighbor sets of each data point in the dataset to be clustered, as well as the adjacency and adjacency type, density path set, density reachability, and density reachability type between any two data points i and j in the dataset, the density distance between any two data points i and j is obtained according to the hyperparameters d and k. This density distance includes the spatial density distance. and topological density distance Spatial density distance The expression is: In the formula, The topological density distance represents the minimum d that makes the spatial density of data point i reach that of data point j. The expression is: In the formula, The minimum value k that makes data point i topologically reachable from data point j (in the definitions of neighbor set, adjacency, density path, and density reachability above, d and k are always hyperparameters, determining the size of the neighbor set, whether two points are adjacent, the number of density paths, and whether two points are density reachable). With the definitions of spatial density distance and topological density distance, there are two distance metrics between any two points in space. Although topological density distance does not strictly satisfy the definition of distance (because it does not satisfy symmetry), like cosine distance, it will show advantages in practical applications.

[0081] S1203. Based on the density distance between any two data points i and j in the dataset to be clustered, map the dataset to be clustered into a spatial density map and / or a topological density map.

[0082] Thus, with a point set D and distance metrics SD and TD between any two points, two undirected density graphs are naturally formed, including the spatial density graph. The expression is:

[0083] ,

[0084] In the formula, Representing spatial density map The vertex set, The adjacency matrix represents the edge weights of a spatial density graph. , Represents the vertices in the graph. Represents an edge;

[0085] Topology density map The expression is:

[0086] ,

[0087] In the formula, Representing the topology density map The vertex set, Representing the topology density map Edge weights, adjacency matrix , Represents the vertices in the graph. Represents an edge.

[0088]

[0089] S130. Based on the spatial density map and / or topological density map, using the labeled dataset and the unlabeled dataset, cluster the data in the dataset to be clustered into several clusters, wherein each data in each cluster has a cluster label, and the cluster label is the label of any data in the labeled dataset or a new label that is different from the labels of all data in the labeled dataset.

[0090] In one embodiment, S130 may include:

[0091] S1301. Based on the spatial density map or topological density map, use labeled and unlabeled datasets for initialization, graph segmentation, and label propagation to cluster the data in the dataset to be clustered into several clusters; or,

[0092] S1302. Based on the spatial density map and the topological density map, use the labeled dataset and the unlabeled dataset to perform initialization, graph segmentation, graph merging, and label propagation to cluster the data in the dataset to be clustered into several clusters.

[0093] Based on the spatial density map SG and the topological density map TG, the dataset to be clustered is mapped into two graphs. Semi-supervised clustering is based on these two graphs and consists of four parts: initialization, graph splitting, graph merging, and label propagation. The input is the unlabeled point set DN and the labeled point set DL (the union of the unlabeled point set DN and the labeled point set DL is the total sample set D). The output is a graph G containing all sample points and their labels.

[0094] S1302 may include:

[0095] The spatial density map and topological density map are initialized using the labeled and unlabeled datasets as the total sample set (the initialization definition can be set in advance).

[0096] Combining the labeled dataset, the same iterative operation is performed on the initialized spatial density map and topological density map to obtain the segmented spatial density map and topological density map: before reaching the preset stopping condition, the longest edge of the density map is cyclically cut to divide the density map into multiple connected subgraphs. Each subgraph has only two possibilities: either the subgraph is a new cluster without prior labels, or the subgraph belongs to a cluster with a unique label value. Specifically, the input is a graph G and a set of labeled points DL. Each connected subgraph in graph G is taken in turn and its intersection with the set of labeled points DL is taken. The labels of all points in the intersection are checked. If the values ​​are not unique, the stopping condition is not met. If all connected subgraphs of graph G are unlabeled or have unique label values, the stopping condition is met.

[0097] The segmented spatial density graph and topological density graph are merged into a new graph G. The merged new graph G still satisfies the property that any subgraph has no label or the label value is unique. The vertex set of the new graph is the union of the vertex set of the spatial density graph and the vertex set of the topological density graph, and the edge set of the new graph is the intersection of the edge set of the spatial density graph and the edge set of the topological density graph.

[0098] Based on the label propagation conditions, cluster labels are assigned to all data points in the new graph. The label propagation conditions include: for a subgraph with a unique label, all data points in the subgraph are assigned the unique label as the cluster label; for a subgraph where all data points are unlabeled, a new label different from any label in the labeled dataset DL is selected as the cluster label, and all data points in the subgraph are assigned the new label.

[0099] After the label propagation phase, each point has its own cluster label, and clustering ends. Algorithmically, this embodiment has three variations: the first is the standard method described above, which sequentially performs initialization, graph partitioning, graph merging, and label propagation, called ST-DBSSC; the second is a method using only the spatial density graph, which sequentially performs initialization, graph partitioning, and label propagation, called S-DBSSC; and the third is a method using only the topological density graph, which sequentially performs initialization, graph partitioning, and label propagation, called T-DBSSC.

[0100] This embodiment can perform cluster analysis on an unlabeled dataset and a small subset of labeled samples to uncover class patterns within the dataset. By adjusting parameters to account for different densities, this embodiment can be flexibly applied to datasets of varying characteristics. Overall, this embodiment is more flexible, has a wider range of applications, and is more efficient than existing semi-supervised density clustering methods.

[0101] To verify the superior performance of the semi-supervised clustering method proposed in this embodiment, the results were validated on two artificially generated datasets, geometry and rings. 3% of the sample points were randomly selected as prior information with known labels. The evaluation metric chosen was the classic clustering evaluation metric—Standardized Mutual Information (NMI) (with values ​​between 0 and 1, where higher values ​​indicate better performance). The proposed semi-supervised clustering method achieved NMI scores of 1.0 and 0.98 on the two datasets, respectively. Visualization results are attached. Figure 2 and 3 As shown.

[0102] In addition, this embodiment selects four commonly used public clustering datasets shown in Table 2 and compares them with classic unsupervised clustering methods (K-means, DBSCAN), classic semi-supervised clustering methods (Seeded K-means, Constrained K-means, SSDBSCAN), and the best-performing semi-supervised density clustering methods (SSGC, CSSC-SC).

[0103]

[0104] Table 3 shows the performance comparison on public datasets. It can be seen that the semi-supervised clustering methods (S-DBSSC, T-DBSSC, ST-DBSSC) proposed in this embodiment achieve the best results in most cases, which verifies the superiority of this embodiment.

[0105]

[0106] Specifically, this embodiment has at least the following technical effects:

[0107] 1. Comprehensive reflection of density attributes: The new density distance proposed in this embodiment combines the spatial and topological perspectives of previous methods, and can comprehensively reflect the density attributes of points.

[0108] 2. Wide range of applicable scenarios: This embodiment has three sub-methods (ST-DBSSC, S-DBSSC, T-DBSSC), which can be used in combination with spatial and topological density maps, or one of them can be used, making it flexible to be applied to different scenarios.

[0109] 3. Intermediate outputs: The spatial density map and topological density map of intermediate outputs are the results of mapping the sample space to a density map. They can be used not only for clustering but also for other analyses.

[0110] 4. New Class Discovery: The semi-supervised clustering method based on graph segmentation in this embodiment fundamentally solves the problem that previous spanning tree methods could not discover new classes.

[0111] 5. Significantly improved efficiency: The semi-supervised clustering method based on graph segmentation in this embodiment has lower time complexity and higher efficiency than the existing semi-supervised density clustering spanning tree method.

[0112] Based on the semi-supervised clustering method described above, it can be applied to open-ended question answer text encoding, forming an open-ended question answer text encoding method. Application scenarios can include road detection, image classification, information retrieval, speech recognition, bioinformatics, healthcare, and many other fields. (See also...) Figure 4 The text encoding methods for open-ended questions include:

[0113] Obtain the open-ended question answer text dataset to be encoded, as well as the labeled and unlabeled open-ended question answer text datasets obtained based on the open-ended question answer text dataset to be encoded. The labels of the data in the labeled open-ended question answer text dataset are the answer types.

[0114] Based on the open-ended question answer text dataset to be encoded, the labeled open-ended question answer text dataset, and the unlabeled open-ended question answer text dataset, the open-ended question answer text dataset to be encoded is clustered into several clusters using any of the semi-supervised clustering methods described above, wherein each open-ended question answer text in each cluster has a cluster label;

[0115] Encode each open-ended question answer text in the dataset based on the clustering label of each open-ended question answer text in the dataset to be encoded.

[0116] To comprehensively, accurately, and effectively verify the practical effects of this embodiment in real-world application scenarios, a specific open-ended questionnaire coding task was selected as the verification scenario for in-depth verification of this embodiment. The dataset used in this verification process originated from the response texts collected from a specific open-ended questionnaire. This open-ended question focused on respondents' opinions and suggestions regarding national policies concerning enterprises, and it has broad representativeness and practical significance. Through the collection process, a total of 300 questionnaire samples were obtained. These samples cover diverse feedback information from personnel from different regions, industry backgrounds, and enterprises of different sizes, and can comprehensively reflect the understanding and demands of various entities regarding national enterprise policies. Some examples are presented in tabular form, as shown in Table 4. This table records some typical questionnaire responses in detail to intuitively demonstrate the data characteristics and the effectiveness and adaptability of this invention when dealing with such data.

[0117]

[0118] First, document embedding is performed on all samples. After embedding, 10% of the samples are randomly selected and manually coded labels are assigned to these selected samples as prior knowledge (labeled dataset). Next, semi-supervised clustering is performed using this embodiment. After the clustering process, keyword extraction is performed on the different categories formed to achieve the goal of automatic encoding.

[0119] To ensure the accuracy and effectiveness of the clustering results, outliers were removed. Outliers were defined as isolated clusters with fewer than 3 samples. After outlier removal, the final clustering result consisted of 6 clusters. Subsequently, a comprehensive evaluation was conducted using the manually coded results as the benchmark for real labels. The evaluation showed that this embodiment achieved an NMI score of 0.85 in the clustering task, demonstrating its superior performance in clustering. The final automatic coding results are detailed in Table 5, which lists the key information and related coding results for each category.

[0120]

[0121] Judging from the final coding results, the semi-supervised clustering method in this embodiment can perform clustering well and complete the task of automatically coding open-ended questionnaire questions.

[0122] This invention provides an open-ended question answer text encoding method. Based on a novel semi-supervised clustering method, it performs efficient and high-precision clustering analysis on open-ended question answer text data. It can discover new classes with only a small amount of prior knowledge. After accurately clustering the open-ended question answer text data, keywords can be extracted and encoded for each class of open-ended question answer text, which facilitates a quick and accurate understanding of the situation of the interviewed group and improves the efficiency and quality of diagnosis and treatment.

[0123] The open-ended question answer text encoding system provided by the present invention is described below. The open-ended question answer text encoding system described below can be referred to in correspondence with the open-ended question answer text encoding method described above.

[0124] See Figure 5 The present invention provides an open-ended question answer text encoding system, which may include:

[0125] The data acquisition module is used to: acquire the open-ended question answer text dataset to be encoded, as well as the labeled open-ended question answer text dataset and the unlabeled open-ended question answer text dataset obtained based on the open-ended question answer text dataset to be encoded, wherein the label of the data in the labeled open-ended question answer text dataset is the answer type;

[0126] The clustering module is used to: based on the open-ended question answer text dataset to be encoded, the labeled open-ended question answer text dataset, and the unlabeled open-ended question answer text dataset, cluster the open-ended question answer text dataset to be encoded into several clusters using the semi-supervised clustering method described above, wherein each open-ended question answer text in each cluster has a cluster label;

[0127] The encoding module is used to encode each open-question answer text in the open-question answer text dataset based on the clustering label of each open-question answer text in the dataset to be encoded.

[0128] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute any of the semi-supervised clustering methods and / or open-ended question answering text encoding methods described above.

[0129] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0130] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is capable of executing any of the above-described semi-supervised clustering methods and / or open-ended question answer text encoding methods.

[0131] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform any of the above-described semi-supervised clustering methods and / or open-ended question answer text encoding methods.

[0132] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semi-supervised clustering method, characterized in that, include: Obtain the dataset to be clustered, as well as the labeled and unlabeled datasets obtained based on the dataset to be clustered; Map the dataset to be clustered to a spatial density map and / or a topological density map, where the spatial density map is based on spatial density distance. Construction, spatial density distance The expression is: In the formula, Let d represent the minimum d that makes the spatial density of data point i reachable from that of data point j, and let the topological density map be based on the topological density distance. Construction, topological density distance The expression is: In the formula, Make the topological density of data point i reach the minimum k of data point j; Based on the spatial density map and / or topological density map, using the labeled dataset and the unlabeled dataset, the data in the dataset to be clustered are clustered into several clusters. Each data in each cluster has a cluster label, which is either the label of any data in the labeled dataset or a new label that is different from the labels of all data in the labeled dataset. The step of clustering the data in the dataset to be clustered into several clusters based on the spatial density map and / or topological density map, using labeled and unlabeled datasets, includes: Based on the spatial density map and the topological density map, the labeled and unlabeled datasets are used for initialization, graph segmentation, and label propagation to cluster the data in the dataset to be clustered into several clusters. The step of clustering the data in the dataset to be clustered into several clusters based on the spatial density map and topological density map, using labeled and unlabeled datasets for initialization, graph segmentation, graph merging, and label propagation, includes: The spatial density map and topological density map are initialized using labeled and unlabeled datasets as the total sample set; By combining the labeled dataset, the same loop operation is performed on the initialized spatial density map and topological density map to obtain the segmented spatial density map and topological density map: before reaching the preset stopping condition, the longest edge of the density map is cut in a loop to divide the density map into multiple connected subgraphs. Each subgraph has only two possibilities: one is that the subgraph is a new cluster without prior label, and the other is that the subgraph belongs to the cluster with its unique value label. The split spatial density graph and topological density graph are merged into a new graph, where the vertex set of the new graph is the union of the vertex sets of the spatial density graph and the topological density graph, and the edge set of the new graph is the intersection of the edge sets of the spatial density graph and the topological density graph. Based on the label propagation conditions, cluster labels are assigned to all data points in the new graph. The label propagation conditions include: for a subgraph with a unique label value, all data points in the subgraph are assigned the unique label value as the cluster label; for a subgraph where all data points are unlabeled, a new label different from any label in the labeled dataset is selected as the cluster label, and all data points in the subgraph are assigned the new label.

2. The semi-supervised clustering method according to claim 1, characterized in that, The step of mapping the dataset to be clustered into a spatial density map and / or a topological density map includes: For any data point i in the clustering dataset, obtain its spatial neighbor set. and topological neighbor set ; Based on the spatial neighbor set and topological neighbor set of each data point in the dataset to be clustered, for any two data points i and j in the dataset to be clustered, determine whether the two data points i and j are adjacent and their adjacency type, and obtain the density path set between the two data points i and j. Also, determine whether the two data points i and j are density reachable and their density reachable type, and obtain the density distance between the two data points i and j. Based on the density distance between any two data points i and j in the dataset to be clustered, the dataset to be clustered is mapped to a spatial density map and / or a topological density map.

3. The semi-supervised clustering method according to claim 2, characterized in that, Spatial Neighbor Set The expression is: ， In the formula, This represents the subset of data in the dataset to be clustered, centered at any data point i and with a radius of d. This represents the distance between data point i and data point j in the dataset to be clustered; Topological neighbor set The expression is: ， In the formula, This represents the k nearest neighbors of any data point i in the dataset to be clustered. This represents the distance between data point i and data point j in the dataset to be clustered. This represents the distance between data point i and its k-th nearest neighbor.

4. The semi-supervised clustering method according to claim 3, characterized in that, Based on the spatial neighbor set and topological neighbor set of each data point in the dataset to be clustered, for any two data points i and j in the dataset to be clustered, it is determined whether the two data points i and j are adjacent and their adjacency type, and the density path set between the two data points i and j is obtained. Furthermore, it is determined whether the two data points i and j are density reachable and their density reachability type, and the density distance between the two data points i and j is obtained, including: Based on the spatial neighbor set and topological neighbor set of each data point in the dataset to be clustered, and according to the adjacency determination criteria, for any two data points i and j in the dataset to be clustered, determine whether the two data points i and j are adjacent and their adjacency type. The adjacency determination criteria include: for any two data points i and j in the dataset to be clustered, if... Then determine if data point i is spatially adjacent to data point j, denoted as... ,like Then determine the topological adjacent data point j of data point i, denoted as . ; Based on the spatial and topological neighbor sets of each data point in the dataset to be clustered, and according to the density path determination criteria, for any two data points i and j in the dataset to be clustered, the density path set between any two data points i and j is obtained. The density path determination criteria include: for any two data points i and j in the dataset to be clustered, if there exists a sequence of data points... , making , ,and Then it is called Let i be a spatial density path from data point i to data point j. For any two data points i and j in the dataset to be clustered, if there exists a sequence of data points... , making , ,and Then determine Let be a topological density path from data point i to data point j, and let be the set of spatial density paths from data point i to data point j in the dataset to be clustered. Let the spatial density path set from data point i to data point j in the dataset to be clustered be denoted as . ; Based on the density path set between any two data points i and j in the dataset to be clustered, and according to the density reachability criteria, determine whether any two data points i and j in the dataset to be clustered are density reachable and what the density reachability type is. The density reachability criteria include: if the spatial density path set from data point i to data point j in the dataset to be clustered is not empty, then data point i is determined to be spatially density reachable from data point j, denoted as... If the set of topological density paths from data point i to data point j is not empty, then data point i is topologically reachable from data point j, denoted as . ; Based on the spatial and topological neighbor sets of each data point in the dataset to be clustered, as well as the adjacency and adjacency type, density path set, density reachability, and density reachability type between any two data points i and j in the dataset, the density distance between any two data points i and j is obtained according to the hyperparameters d and k. This density distance includes the spatial density distance. and topological density distance Spatial density distance The expression is: In the formula, The topological density distance represents the minimum d that makes the spatial density of data point i reach that of data point j. The expression is: In the formula, Make the topological density of data point i reach the minimum k of data point j.

5. The semi-supervised clustering method according to claim 4, characterized in that, Spatial density map The expression is: ， In the formula, Representing spatial density map The vertex set, Representing spatial density map Edge weights, adjacency matrix , Represents the vertices in the graph. Represents an edge; Topology density map The expression is: ， In the formula, Representing the topology density map The vertex set, Representing the topology density map Edge weights, adjacency matrix , Represents the vertices in the graph. Represents an edge.

6. The semi-supervised clustering method according to any one of claims 1-5, characterized in that, The step involves clustering the data in the dataset to be clustered into several clusters based on spatial density maps and / or topological density maps, using labeled and unlabeled datasets, including: Based on the spatial density map or topological density map, the data in the dataset to be clustered are clustered into several clusters by using labeled and unlabeled datasets for initialization, graph segmentation, and label propagation.

7. A text encoding method for open-ended question answers, characterized in that, Obtain the open-ended question answer text dataset to be encoded, as well as the labeled and unlabeled open-ended question answer text datasets obtained based on the open-ended question answer text dataset to be encoded. The labels of the data in the labeled open-ended question answer text dataset are the answer types. Based on the open-ended question answer text dataset to be encoded, the labeled open-ended question answer text dataset, and the unlabeled open-ended question answer text dataset, the open-ended question answer text dataset to be encoded is clustered into several clusters using the semi-supervised clustering method described in any one of claims 1 to 6, wherein each open-ended question answer text in each cluster has a cluster label; Encode each open-ended question answer text in the dataset based on the clustering label of each open-ended question answer text in the dataset to be encoded.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the semi-supervised clustering method as described in any one of claims 1 to 6 and / or the open-ended question answering text encoding method as described in claim 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the semi-supervised clustering method as described in any one of claims 1 to 6 and / or the open-ended question answering text encoding method as described in claim 7.

Images