Method, device and equipment for proposal combination based on dimension reduction clustering and medium

Abstract

The invention belongs to the technical field of proposal processing, and relates to a proposal combination method and device based on dimension reduction clustering, equipment and a medium. The method comprises the following steps: acquiring data of a proposal library, and constructing a proposal set; preprocessing each proposal of the proposal set, and obtaining a document vector of each proposal to construct a feature matrix of all the proposals; performing dimension reduction processing on the feature matrixes of all the proposals, and clustering all the proposals by adopting a DBSCAN algorithm to obtain a plurality of clustering groups; for each cluster group, performing quality analysis on each proposal by adopting an NMF algorithm to obtain a quality coefficient of each proposal; and according to the quality coefficient of each proposal, combining all the proposals of each clustering group. According to the invention, full-process automatic processing of proposal and case combination can be realized, and the case combination accuracy is improved.

Description

Technical Field

[0001] The present application relates to the technical field of proposal processing, and in particular to a method, device, equipment and medium for proposal merging based on dimensionality reduction clustering. Background Art

[0002] In the proposal work, it is common for multiple people to make similar suggestions for similar issues. If this is not handled, the handling unit will have problems such as duplication of work and exhaustion. The proposal consolidation can avoid duplication of proposals and improve the overall quality of proposals, so it is of great significance in the proposal work.

[0003] In the prior art, proposals are merged by traditional methods such as manual processing or by programs to select similar proposals for auxiliary merger.

[0004] However, these two processing methods still require a lot of manual participation, have a high misjudgment rate, and are low in efficiency. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device, equipment and medium for proposal merging based on dimensionality reduction clustering to address the above technical problems, which can realize full automation of proposal merging and improve the accuracy of merging.

[0006] A proposal merging method based on dimensionality reduction clustering, comprising: Get the data of the proposal library and build a proposal set; Each proposal in the proposal set is preprocessed and the document vector of each proposal is obtained to construct the feature matrix of all proposals; the feature matrix of all proposals is subjected to dimensionality reduction processing, and the DBSCAN algorithm is used to cluster all proposals to obtain multiple cluster groups; For each cluster group, the NMF algorithm is used to analyze the quality of each proposal and obtain the quality coefficient of each proposal; All proposals in each cluster group are merged according to the quality coefficient of each proposal.

[0007] In one embodiment, each proposal in the proposal set is preprocessed and a document vector of each proposal is obtained to construct a feature matrix of all proposals, including: Each proposal in the proposal set is preprocessed and segmented using the Jieba word segmentation model to obtain multiple proposal words; Based on multiple proposed words, the FastText pre-trained model is used to obtain multiple word vectors; Based on multiple word vectors, the proposals are represented to obtain the document vector of each proposal; Based on the document vector of each proposal, a feature matrix of all proposals is constructed.

[0008] In one embodiment, the feature matrices of all proposals are subjected to dimensionality reduction processing, and the DBSCAN algorithm is used to cluster all proposals to obtain multiple cluster groups, including: Perform dimensionality reduction processing on the feature matrices of all proposals to obtain the dimensionality reduction matrix of all proposals; Determine the similarity threshold and the minimum number of samples based on the dimensionality reduction matrix; calculate the Euclidean distance for each proposal; determine the core proposal based on the Euclidean distance, similarity threshold and the minimum number of samples; Starting from the core proposal, the clustering is expanded using Euclidean distance to obtain multiple cluster groups.

[0009] In one embodiment, the feature matrices of all proposals are subjected to dimensionality reduction processing to obtain dimensionality reduction matrices of all proposals, including: For the feature matrices of all proposals, PCA technology is introduced to calculate the covariance matrix of the feature matrix; Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors; Select the eigenvectors corresponding to the largest multiple eigenvalues ​​to form an eigenvector matrix; According to the feature vector matrix, a projection matrix is ​​constructed, and the proposal set is projected into the feature space after dimensionality reduction, thereby completing the dimensionality reduction process and obtaining the dimensionality reduction matrix of all proposals.

[0010] In one embodiment, for each cluster group, the NMF algorithm is used to perform quality analysis on each proposal to obtain a quality coefficient of each proposal, including: For each cluster group, construct a document-term matrix for each proposal; Use the NMF algorithm to decompose the document-term matrix to obtain the document-topic matrix and the topic-term matrix; According to the preset standards, the coefficient ratio of each proposal is set, and the quality analysis of each proposal is carried out in combination with the document-topic matrix and the topic-term matrix to obtain the quality coefficient of each proposal.

[0011] In one embodiment, the preset criteria include: Thematic consistency, topic diversity, keyword prominence, and document coherence.

[0012] In one embodiment, all proposals of each cluster group are merged according to the quality coefficient of each proposal, including: The proposal with the highest quality coefficient in each cluster group is the main proposal, and the other proposals are similar proposals; All proposals of each cluster group are merged according to the main proposal and similar proposals.

[0013] A device for combining proposals based on dimensionality reduction clustering, comprising: The acquisition module is used to obtain the data of the proposal library and build a proposal set; The processing module is used to pre-process each proposal in the proposal set and obtain the document vector of each proposal to construct the feature matrix of all proposals; perform dimensionality reduction processing on the feature matrix of all proposals, and use the DBSCAN algorithm to cluster all proposals to obtain multiple cluster groups; The analysis module is used to perform quality analysis on each proposal using the NMF algorithm for each cluster group and obtain the quality coefficient of each proposal; The proposal merging module is used to merge all proposals of each cluster group according to the quality coefficient of each proposal.

[0014] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: Get the data of the proposal library and build a proposal set; Each proposal in the proposal set is preprocessed and the document vector of each proposal is obtained to construct the feature matrix of all proposals; the feature matrix of all proposals is subjected to dimensionality reduction processing, and the DBSCAN algorithm is used to cluster all proposals to obtain multiple cluster groups; For each cluster group, the NMF algorithm is used to analyze the quality of each proposal and obtain the quality coefficient of each proposal; All proposals in each cluster group are merged according to the quality coefficient of each proposal.

[0015] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps: Get the data of the proposal library and build a proposal set; Each proposal in the proposal set is preprocessed and the document vector of each proposal is obtained to construct the feature matrix of all proposals; the feature matrix of all proposals is subjected to dimensionality reduction processing, and the DBSCAN algorithm is used to cluster all proposals to obtain multiple cluster groups; For each cluster group, the NMF algorithm is used to analyze the quality of each proposal and obtain the quality coefficient of each proposal; All proposals in each cluster group are merged according to the quality coefficient of each proposal.

[0016] The above-mentioned proposal merging method based on dimensionality reduction clustering is a method for realizing proposal merging by combining DBSCAN algorithm and NMF algorithm. Through a series of data preprocessing, precise clustering and proposal quality analysis, automatic merging is realized to help the proposal work to be carried out efficiently. Compared with the prior art, this application improves the DBSCAN algorithm to cluster proposals, without presetting the number of clusters (i.e. clustering), that is, without manual participation in presetting the number of clusters, thereby improving the efficiency and accuracy of clustering; PCA technology is used to reduce the dimension of the high-dimensional proposal feature matrix, which can reduce the dimension of the feature and retain the most important variance component in the data; grid search is used to determine the optimal parameters, which can more accurately identify the similarities in different proposal documents, thereby improving the accuracy of clustering; after clustering is completed, the main proposal is selected by combining NMF algorithm, which can decompose a non-negative document-term matrix into a matrix representing documents and a matrix representing terms, and promote the clear interpretability of the document theme with non-negative constraints, and can better identify the content of the proposal and conduct quality analysis, and can realize the full automatic processing of proposal merging, solve the problem of repeated processing of proposals, and save time and resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is an application scenario diagram of a method for combining proposals based on dimensionality reduction clustering in one embodiment; Figure 2 A schematic diagram of a process of a proposal merging method based on dimensionality reduction clustering in one embodiment; Figure 3 The present invention is an architectural diagram of a method for combining proposals based on dimensionality reduction clustering in one embodiment; Figure 4 A structural block diagram of a device for proposal merging based on dimensionality reduction clustering in one embodiment; Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0019] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "multiple groups" means at least two groups, such as two groups, three groups, etc., unless otherwise clearly and specifically defined.

[0020] In this application, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0021] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0022] The present application provides a proposal merging method based on dimensionality reduction clustering, which can be applied to Figure 1 In the application scenario diagram shown, the terminal 102 communicates with the server 104 through the network, and the terminal 102 may include but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 may be various portal websites, servers corresponding to the backend of the work system, etc.

[0023] This application provides a method for proposal merging based on dimensionality reduction clustering, such as Figure 2 The flowchart shown in FIG. 1 shows a method for applying the method to Figure 1 The following is an example of a terminal in the example, including: Step 202, obtain the data of the proposal library and construct a proposal set.

[0024] In this step, the data of all proposals in the proposal library are collected to form a proposal set.

[0025] It should be noted that all proposals refer to all proposals submitted in the current session that need to be merged.

[0026] Step 204, preprocessing is performed on each proposal in the proposal set, and a document vector of each proposal is obtained to construct a feature matrix of all proposals; dimension reduction is performed on the feature matrix of all proposals, and the DBSCAN algorithm is used to cluster all proposals to obtain multiple cluster groups.

[0027] Specifically: Each proposal in the proposal set is preprocessed, including: removing stop words, punctuation marks, filtering non-Chinese characters, normalizing synonyms using Cilin, and using the Jieba word segmentation model to perform word segmentation to obtain multiple proposal words; based on the multiple proposal words, the FastText pre-trained model (existing technology) is used to obtain multiple word vectors; based on the multiple word vectors, the proposals are represented to obtain the document vector of each proposal.

[0028] Based on the document vector of each proposal, construct the feature matrix of all proposals; For the feature matrix (with high dimensions) of all proposals, PCA technology is introduced to calculate the covariance matrix of the feature matrix; the covariance matrix is ​​decomposed into eigenvalues ​​to obtain eigenvalues ​​and corresponding eigenvectors; the eigenvectors corresponding to the largest multiple eigenvalues ​​are selected to form an eigenvector matrix; according to the eigenvector matrix, a projection matrix is ​​constructed, and the proposal set is projected into the reduced feature space (the feature space is the dimension formed by the document vector), thereby completing the dimensionality reduction process and obtaining the reduced dimension matrix of all proposals, which not only effectively reduces the dimension of the feature space, but also retains the most important data variance, providing a more compact and easier to handle feature representation for clustering analysis, so as to improve the clustering effect in the next clustering step and improve the computational efficiency; According to the dimension reduction matrix, determine the optimal parameters (the initial value can be determined by the existing technology, and the optimal value is determined by improving the existing algorithm through grid search). The optimal parameters include the similarity threshold e (If the similarity calculation result of two proposals is less than or equal to this threshold, then they are considered close to each other in the feature space. e as the neighborhood radius) and the minimum number of samples MinPts (proposal e - the number of other proposals contained in the neighborhood at least, e -Neighborhood refers to a region centered around a specific document. e is an area defined by the radius); for each proposal, use the Euclidean distance to calculate the distance between it and all other proposals as the similarity calculation result of the two proposals; according to the Euclidean distance, similarity threshold and minimum number of samples, determine the core proposal. Specifically, if a proposal e - There are at least MinPts proposals, and these e-The Euclidean distance between the proposal in the neighborhood and this proposal is less than or equal to e , then the proposal is considered to be a core proposal; starting from the core proposal, the Euclidean distance is used to expand the clustering to obtain multiple cluster groups. Specifically, if a proposal is located in the core proposal e - In the neighborhood (i.e. the Euclidean distance between the proposal and the core proposal is less than or equal to e ), and has not been assigned to any cluster, it is added to the current cluster, that is, the core proposal and all proposals in the neighborhood of the core proposal are grouped as a cluster; a cluster label (such as 0, 1, 2, ..., n) is assigned to each proposal document, and each label represents a different cluster.

[0029] More specifically: Use FastText pre-training model to get word vector: ; In the formula, It is the proposal word obtained using the FastText pre-trained model The word vector of Document vector: ; In the formula, The document vector representing the proposal, represents the total number of proposal words in the proposal, Indicates the first word vectors of proposal words; Feature matrix: ; In the formula, represents the feature matrix of all proposals, represents the document vector of the first proposal, represents the document vector of the second proposal, Indicates n The document vector of the proposal, n Represents the total number of all proposals.

[0030] Covariance matrix: ; In the formula, is the covariance matrix, is the feature matrix after centralized processing (centralized processing is the existing technology) ( Dimension, that is OK List), is transposed; Perform eigenvalue decomposition on the covariance matrix: ; In the formula, is the eigenvalue decomposition, is the characteristic value, is the feature vector; Choose the largest The eigenvectors corresponding to the eigenvalues ​​constitute the eigenvector matrix: ; In the formula, is the eigenvector matrix, is the eigenvector corresponding to the first largest eigenvalue, is the eigenvector corresponding to the second largest eigenvalue, For the The eigenvector corresponding to the largest eigenvalue; Projection matrix: ; In the formula, is the projection matrix, which is a matrix composed of eigenvector matrices as column vectors ( dimension); Dimensionality reduction matrix: ; In the formula, is the dimension reduction matrix ( Dimension, that is OK List).

[0031] Optimal parameters: ; In the formula, is the similarity threshold, is the minimum number of samples, Grid search (the grid search algorithm of this application can automatically evaluate multiple parameter combinations and automatically find and determine the optimal parameter combination. Compared with traditional algorithms, it avoids the tediousness of manual parameter adjustment. By systematically traversing the given parameter grid and combining the clustering quality metric, it finds the global optimal solution, thereby improving the clustering quality, reducing the impact of noise, and enhancing the generalization ability of the model). is the clustering model, is the parameter grid defined; Calculate the Euclidean distance: ; In the formula, is the proposal pair distance function, used to calculate the proposal With proposal The Euclidean distance between For proposal In the feature space j The value on the dimension (that is, the proposal ), For proposal In the feature space j The value on the dimension (that is, the proposal ); for example: m =5, then the eigenvector matrix contains 5 elements, which correspond to the 5 dimensions in the feature space after dimensionality reduction; e -Neighborhood: ; In the formula, For proposal of Neighborhood, that is, with the proposal The Euclidean distance is less than or equal to e The set of all proposals for For the proposal set; The core proposals meet the following requirements: ; In the formula, For proposal of e - the number of all proposals in the neighborhood; Clustering: ; In the formula, represents the clustering result, represents the set of all proposals participating in clustering, and Indicates two different proposals, Express proposal of e -neighborhood, Express proposal of e -Neighborhood.

[0032] In this step, each cluster group includes more than two proposals. A corresponding cluster label is set for each cluster group. According to all cluster groups, a dictionary is created to store the mapping relationship between cluster labels and proposals, and each proposal is assigned to its corresponding cluster label according to the cluster group. Proposals that are not assigned to cluster groups are considered independent proposals and will not be included in the subsequent merger.

[0033] Step 206: For each cluster group, the NMF algorithm is used to perform quality analysis on each proposal to obtain a quality coefficient of each proposal.

[0034] Specifically: For each cluster group, construct the document-term matrix of each proposal; use the NMF algorithm to decompose the document-term matrix to obtain the document-topic matrix and the topic-term matrix; set the coefficient ratio of each proposal according to the preset standards, and combine the document-topic matrix and the topic-term matrix to perform quality analysis on each proposal to obtain the quality coefficient of each proposal. The preset standards include the following four dimensions: topic consistency, topic diversity, keyword prominence, and document coherence.

[0035] More specifically: For each cluster group, construct the document-term matrix of each proposal (get multiple terms of the document by word segmentation, calculate the frequency of each term, assign a unique code to each term, and then get the document-term matrix, which can be obtained according to the existing technology and will not be described in detail here); define the total number of topics, use the NMF algorithm to decompose the document-term matrix, get the initial document-topic matrix and topic-term matrix, and set the objective function (provide a clear optimization goal to find the low-dimensional feature representation that best represents the intrinsic structure of the data by minimizing the difference between the original data and the decomposition result, thereby achieving data dimensionality reduction, feature extraction, and enhancing the accuracy of the results), iteratively optimize the document-topic matrix and topic-term matrix, and when the preset conditions (existing technology) are met, get the final document-topic matrix and topic-term matrix; for the document-topic matrix, each row represents a pending topic, according to the existing technology, get the weight of each pending topic, and define the total number of topics as , then the one with the largest weight To be determined topics are the topics; for each topic in the topic-term matrix, according to the existing technology, the weight of each term in the topic is obtained, and the total number of keywords is defined as , then the one with the largest weight The terms are keywords of the corresponding topics; according to the document-topic matrix, the topic consistency coefficient, topic diversity coefficient and document coherence coefficient are calculated; according to the topic-term matrix, the keyword prominence coefficient is calculated; according to the topic consistency coefficient, topic diversity coefficient, document coherence coefficient and keyword prominence coefficient, and the coefficient ratio of the topic consistency coefficient, topic diversity coefficient, document coherence coefficient and keyword prominence coefficient is set, the quality coefficient of the proposal is obtained.

[0036] Among them, the document-term matrix: ; Document-topic matrix: ; Topic-Term Matrix: ; Objective function of NMF: ; Iterative Optimization: ; In the formula, represents the document-term matrix, which is a non-negative matrix to be decomposed; Representing terms In the The weight of the proposals, represents the total number of proposals in the cluster group, Indicates the total number of distinct terms in the vocabulary; Represents the document-topic matrix, which is a non-negative basis matrix; Indicates Proposal and The degree of relevance of the topics, represents the total number of topics; represents the topic-term matrix, which is a non-negative coefficient matrix; Indicates Term in topic The weight of Represents the Frobenius norm, which is used to measure the size of the reconstruction error.

[0037] For example: Proposal list documents = ["Natural language processing is an important direction in the field of artificial intelligence.", "Deep learning has achieved remarkable results in image recognition and speech recognition."].

[0038] The document-term matrix constructed is: [[1 0 0 0 1 0 0 0 0 0] [0 1 1 1 0 1 0 0 0 0]] Define the total number of topics as 3, use the NMF algorithm to decompose the document-term matrix, and obtain two non-negative matrices, where: Document-topic matrix: [[ 0.707106780.29289322] [ 0.707106780.29289322 ]] Topic-Term Matrix: [[ 0.499999990.] [ 0.0.49999999 ]] Set the coefficient ratio, for example: topic consistency 0.3, topic diversity 0.2, keyword prominence 0.3, document coherence 0.2.

[0039] Perform quality analysis: 1) The topic consistency coefficient is calculated by evaluating the weights of the topics in the proposal. If the weight of a proposal is mainly concentrated on one or two topics, the proposal is considered to have high consistency; ; in, Indicates The theme consistency coefficient of the proposals; Indicates The weight distribution vector of proposals on all topics (which can be obtained by using the existing technology based on the document-topic matrix); express The maximum value among The highest weight of proposals on a topic; Indicates The sum of the weights of proposals on all topics, represents the total number of topics; 2) The topic diversity coefficient is measured by the entropy of all topic weights in the proposal; the lower the entropy value, the more concentrated the topics covered by the proposal; the higher the entropy value, the more diverse the topics; ; in, represents the topic diversity coefficient, The proposal is in The weights on the topics (which can be obtained by using the existing technology based on the document-topic matrix), ; 3) The keyword prominence coefficient is calculated by analyzing the frequency and distribution of keywords in the proposal; keywords should appear frequently in the proposal and be clearly distinguishable from other terms; ; in, represents the topic diversity coefficient, It is l The frequency of keywords in the proposals, is the total number of keywords; 4) The document coherence coefficient is obtained by analyzing the coherence of the topics in the proposal; if the paragraphs or sentences in the proposal revolve around one or several closely related topics, the proposal is considered to have high coherence; ; in, represents the document coherence coefficient, and is the topic distribution vector of adjacent paragraphs or sentences in the proposal (which can be obtained by using the existing technology based on the document-topic matrix), is the similarity measurement function (i.e., cosine similarity), which can be obtained according to the prior art. is the number of paragraphs or sentences in the proposal; 5) The quality coefficient of each proposal is calculated through the above four dimensions and their coefficient ratios: ; in, S is the quality coefficient.

[0040] In this step, the coefficient ratio can also be specifically set according to actual conditions, which will not be described in detail here.

[0041] Step 208 , combining all proposals of each cluster group according to the quality coefficient of each proposal.

[0042] Specifically: The proposal with the highest quality coefficient in each cluster group is the main proposal, and the other proposals are similar proposals; based on the main proposal and similar proposals, all proposals in each cluster group are merged.

[0043] More specifically: The proposal documents of each cluster group are sorted in descending order according to the quality coefficient, and the proposal ranked first in each cluster group is determined to be the main proposal; all other proposals of each cluster group are regarded as similar proposals, and the main proposal and similar proposals in the same group are merged according to the selected main proposal; in the subsequent processing process, for a cluster group, only the main proposal needs to be processed to avoid repeated processing.

[0044] In this embodiment, if Figure 3 As shown, the present application combines the DBSCAN algorithm and the NMF algorithm to realize proposal merging. Specifically: all proposals in the proposal set are preprocessed and segmented, and then vectorized (including: obtaining word vectors and document vectors), and all proposals are clustered using the DBSCAN algorithm, similar proposals are clustered into a cluster group, and cluster labels are assigned; taking the cluster group as a unit, the NMF (non-negative matrix factorization) algorithm is used to perform quality analysis on each proposal, and the proposal with the highest quality coefficient in each cluster group is determined as the main merged proposal, and the main merged proposal and similar proposals in the same cluster group are merged to complete the automatic merging.

[0045] The above-mentioned proposal merging method based on dimensionality reduction clustering is a method for realizing proposal merging by combining DBSCAN algorithm and NMF algorithm. Through a series of data preprocessing, precise clustering and proposal quality analysis, automatic merging is realized to help the proposal work to be carried out efficiently. Compared with the prior art, this application improves the DBSCAN algorithm to cluster proposals, without presetting the number of clusters (i.e. clustering), that is, without manual participation in presetting the number of clusters, thereby improving the efficiency and accuracy of clustering; PCA technology is used to reduce the dimension of the high-dimensional proposal feature matrix, which can reduce the dimension of the feature and retain the most important variance component in the data; grid search is used to determine the optimal parameters, which can more accurately identify the similarities in different proposal documents, thereby improving the accuracy of clustering; after clustering is completed, the main proposal is selected by combining NMF algorithm, which can decompose a non-negative document-term matrix into a matrix representing documents and a matrix representing terms, and promote the clear interpretability of the document theme with non-negative constraints, and can better identify the content of the proposal and conduct quality analysis, and can realize the full automatic processing of proposal merging, solve the problem of repeated processing of proposals, and save time and resources.

[0046] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0047] The present application also provides a device for combining proposals based on dimensionality reduction clustering, such as Figure 4 As shown, in one embodiment, it includes: an acquisition module 402, a processing module 404, an analysis module 406 and a case combination module 408, wherein: The acquisition module 402 is used to acquire data from the proposal library and construct a proposal set; The processing module 404 is used to pre-process each proposal in the proposal set and obtain the document vector of each proposal to construct a feature matrix of each proposal; perform dimensionality reduction processing on the feature matrix of each proposal, and cluster all proposals using the DBSCAN algorithm to obtain multiple cluster groups; An analysis module 406 is used to perform quality analysis on each proposal using the NMF algorithm for each cluster group to obtain a quality coefficient of each proposal; The proposal combining module 408 is used to combine all proposals of each cluster group according to the quality coefficient of each proposal.

[0048] For the specific definition of a device for combining proposals based on dimensionality reduction clustering, please refer to the definition of a method for combining proposals based on dimensionality reduction clustering mentioned above, which will not be repeated here. Each module in the above device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0049] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for proposal merging based on dimensionality reduction clustering is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0050] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0051] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0052] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0053] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0054] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0055] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for combining proposals based on dimensionality reduction clustering, characterized in that: include: Get the data of the proposal library and build a proposal set; Each proposal in the proposal set is preprocessed and the document vector of each proposal is obtained to construct the feature matrix of all proposals; the feature matrix of all proposals is subjected to dimensionality reduction processing, and the DBSCAN algorithm is used to cluster all proposals to obtain multiple cluster groups; For each cluster group, the NMF algorithm is used to perform quality analysis on each proposal and obtain the quality coefficient of each proposal; All proposals in each cluster group are merged according to the quality coefficient of each proposal.

2. According to claim 1, a method for combining proposals based on dimensionality reduction clustering is characterized in that: Each proposal in the proposal set is preprocessed and the document vector of each proposal is obtained to construct the feature matrix of all proposals, including: Each proposal in the proposal set is preprocessed and segmented using the Jieba word segmentation model to obtain multiple proposal words; Based on multiple proposed words, the FastText pre-trained model is used to obtain multiple word vectors; Based on multiple word vectors, the proposals are represented to obtain the document vector of each proposal; Based on the document vector of each proposal, a feature matrix of all proposals is constructed.

3. A method for combining proposals based on dimensionality reduction clustering according to claim 1 or 2, characterized in that: The feature matrix of all proposals is subjected to dimensionality reduction processing, and the DBSCAN algorithm is used to cluster all proposals to obtain multiple clustering groups, including: Perform dimensionality reduction processing on the feature matrices of all proposals to obtain the dimensionality reduction matrix of all proposals; Determine the similarity threshold and the minimum number of samples based on the dimensionality reduction matrix; calculate the Euclidean distance for each proposal; determine the core proposal based on the Euclidean distance, similarity threshold and the minimum number of samples; Starting from the core proposal, the clustering is expanded using Euclidean distance to obtain multiple cluster groups.

4. The method for combining proposals based on dimensionality reduction clustering according to claim 3, characterized in that: Perform dimensionality reduction on the feature matrices of all proposals to obtain the dimensionality reduction matrices of all proposals, including: For the feature matrices of all proposals, PCA technology is introduced to calculate the covariance matrix of the feature matrix; Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors; Select the eigenvectors corresponding to the largest multiple eigenvalues ​​to form an eigenvector matrix; According to the feature vector matrix, a projection matrix is ​​constructed, and the proposal set is projected into the feature space after dimensionality reduction, thereby completing the dimensionality reduction process and obtaining the dimensionality reduction matrix of all proposals.

5. A method for combining proposals based on dimensionality reduction clustering according to claim 1 or 2, characterized in that: For each cluster group, the NMF algorithm is used to analyze the quality of each proposal and obtain the quality coefficient of each proposal, including: For each cluster group, construct a document-term matrix for each proposal; Use the NMF algorithm to decompose the document-term matrix to obtain the document-topic matrix and the topic-term matrix; According to the preset standards, the coefficient ratio of each proposal is set, and the quality analysis of each proposal is carried out in combination with the document-topic matrix and the topic-term matrix to obtain the quality coefficient of each proposal.

6. The method for combining proposals based on dimensionality reduction clustering according to claim 5, characterized in that: The preset standards include: Thematic consistency, topic diversity, keyword prominence, and document coherence.

7. A method for combining proposals based on dimensionality reduction clustering according to claim 1 or 2, characterized in that: All proposals in each cluster group are combined according to the quality coefficient of each proposal, including: The proposal with the highest quality coefficient in each cluster group is the main proposal, and the other proposals are similar proposals; All proposals of each cluster group are merged according to the main proposal and similar proposals.

8. A device for combining proposals based on dimensionality reduction clustering, characterized in that: include: The acquisition module is used to obtain the data of the proposal library and build a proposal set; The processing module is used to pre-process each proposal in the proposal set and obtain the document vector of each proposal to construct the feature matrix of all proposals; perform dimensionality reduction processing on the feature matrix of all proposals, and use the DBSCAN algorithm to cluster all proposals to obtain multiple cluster groups; The analysis module is used to perform quality analysis on each proposal using the NMF algorithm for each cluster group and obtain the quality coefficient of each proposal; The proposal merging module is used to merge all proposals of each cluster group according to the quality coefficient of each proposal.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images