Training text data acquisition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a training text data acquisition method and device, electronic equipment and a storage medium, and the method comprises the steps: building a text vector corresponding to each candidate document, and drawing a hypercube comprising each text vector; averagely dividing the hypercube into a plurality of sub-cubes; determining the number of clusters for clustering, and determining an initial centroid based on the number of the clusters and the number of the text vectors in each sub-cube; and clustering each text vector based on the cluster number and the initial centroid to obtain a plurality of clustering result clusters, and determining training text data based on the plurality of clustering result clusters. According to the embodiment of the invention, training text data with high professionality and high quality can be generated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for acquiring training text data. Background Art

[0002] With the continuous development of Internet technology, technologies such as big data and cloud computing are constantly accumulating and improving. Public data resource acquisition channels headed by the Internet and a large number of enterprises have accumulated massive data resources. These data have a significant effect on the training of artificial intelligence models in various fields, including the financial field. Using a large amount of general data obtained from public channels combined with professional field data accumulated within the enterprise to pre-train and fine-tune large models can significantly improve the application efficiency of large models in professional fields.

[0003] However, due to the uneven quality of data obtained from public channels, it is unable to meet the quality and professional requirements of artificial intelligence training in various professional fields. Therefore, there is an urgent need for a training text generation solution that can generate highly professional and high-quality training text data. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, electronic device and storage medium for acquiring training text data, which can generate highly professional and high-quality training text data.

[0005] In a first aspect, an embodiment of the present invention provides a method for acquiring training text data, comprising:

[0006] Establishing text vectors corresponding to each candidate document and drawing a hypercube including each text vector;

[0007] Dividing the hypercube into a plurality of sub-cubes on average;

[0008] Determining the number of clusters for clustering and determining initial centroids based on the number of clusters and the number of text vectors in each sub-cube; and

[0009] The text vectors are clustered based on the number of clusters and the initial centroids to obtain a plurality of clustering result clusters, and training text data is determined based on the plurality of clustering result clusters.

[0010] In a second aspect, an embodiment of the present invention provides a training text data acquisition device, comprising:

[0011] A hypercube acquisition module, used to establish text vectors corresponding to each candidate document and draw a hypercube including each text vector;

[0012] A sub-cube acquisition module, used for evenly dividing the hypercube into a plurality of sub-cubes;

[0013] A cluster number and centroid acquisition module, used to determine the number of clusters for clustering and determine the initial centroid based on the number of clusters and the number of text vectors in each sub-cube; and

[0014] The clustering result and training text data acquisition module is used to cluster the text vectors based on the number of clusters and the initial centroid to obtain multiple clustering result clusters, and determine the training text data based on the multiple clustering result clusters.

[0015] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a training text data acquisition method as described in any one of the embodiments of the present invention is implemented.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for acquiring training text data as described in any one of the embodiments of the present invention.

[0017] The embodiments of the present invention provide a method, device, electronic device and storage medium for acquiring training text data. By establishing a text vector corresponding to each candidate document and drawing a hypercube including each text vector, and then evenly dividing the hypercube, the initial centroid of the cluster is determined based on the number of text vectors in the sub-cubes obtained by the division. This can effectively improve the quality of selecting the initial centroid and reduce the sensitivity to the selection of the initial centroid. By clustering each text vector based on the initial centroid, the stability and consistency of the clustering can be improved, the efficiency of capturing text semantic information can be improved, and the text clustering effect can be optimized. Furthermore, by determining the training text data based on the clustering result cluster, the efficiency of acquiring the training text data can be improved and the quality and professionalism of the acquired training text data can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of a method for acquiring training text data provided by an embodiment of the present invention;

[0020] Figure 2 is another flow chart of the method for acquiring training text data provided by an embodiment of the present invention;

[0021] Figure 3 is another flow chart of the method for acquiring training text data provided by an embodiment of the present invention;

[0022] Figure 4 is another flow chart of the method for acquiring training text data provided by an embodiment of the present invention;

[0023] Figure 5 It is a flow chart of a training text data acquisition device provided by an embodiment of the present invention;

[0024] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Figure 1 A flowchart of a training text data acquisition method, device, electronic device, and storage medium provided in an embodiment of the present invention. This embodiment can be applied to scenarios of acquiring training data for artificial intelligence models in the financial field. The method can be executed by a training text data acquisition device provided in an embodiment of the present invention, and the device can be implemented in software and / or hardware. In a specific embodiment, the device can be integrated in an electronic device, such as a computer, a server, etc. The following embodiments will be described by taking the device integrated in an electronic device as an example. Reference Figure 1 , the method may specifically include the following steps:

[0028] Step 101: Create text vectors corresponding to each candidate document and draw a hypercube including each text vector. This step can help evenly divide the hypercube into multiple sub-cubes and determine the initial centroid of the cluster based on the amount of text in each sub-cube.

[0029] Specifically, before step 101, initial text data may be prepared, and each initial document in the initial text data may be preprocessed to obtain the above-mentioned candidate documents. Specifically, the above-mentioned initial text data may be obtained from an open source Chinese data set or crawled from a web page.

[0030] Specifically, the process of preprocessing the initial text data may include: cleaning messy characters and punctuation marks, filtering harmful character texts, and screening and removing colloquial texts and general texts. The above-mentioned harmful character texts include pornographic and social phobia texts, the above-mentioned colloquial texts include texts such as you, me, etc., and the above-mentioned general texts include texts such as of, land, etc.

[0031] Specifically, the process of establishing the text vector corresponding to each candidate document can be based on algorithms such as Bag-of-Words (BoW), Term Frequency-Inverse Document Frequency (TF-IDF) or Word Embedding.

[0032] Specifically, the process of drawing a hypercube including various text vectors may include: determining the dimension of the hypercube based on the various text vectors; determining the vertices of the hypercube based on the dimension of the hypercube and the element values ​​of the various text vectors; and establishing the hypercube based on the vertices of the hypercube.

[0033] Step 102: divide the hypercube into multiple sub-cubes on average. This step can help determine the initial centroid of the cluster based on the number of texts in each sub-cube.

[0034] Optionally, the process of evenly dividing the hypercube into a plurality of sub-cubes includes: determining the number of sub-cubes, and evenly dividing the hypercube based on the number of sub-cubes to obtain the plurality of sub-cubes.

[0035] Specifically, the number of the sub-cubes can be set based on empirical data.

[0036] In a specific example, for an m-dimensional hypercube, when the number of sub-cubes is M, and M=k1×k2...k j ...k m ,k iis an integer, j = 1, 2…m. It can be divided along the m dimensions, dividing the side length of each dimension into k equal parts. i In this way, M sub-hypercubes can be obtained. The side length of each sub-hypercube in each dimension is the side length of the original hypercube in that dimension.

[0037] Step 103, determine the number of clusters to be clustered and determine the initial centroid based on the number of clusters and the number of text vectors in each sub-cube. Compared with the random selection of the initial centroid method in the prior art, this step can effectively improve the quality of the initial centroid selection, reduce the sensitivity to the initial centroid selection, improve the stability and consistency of clustering, and improve the efficiency of capturing text semantic information, and optimize the text clustering effect.

[0038] Specifically, in clustering algorithms, a cluster refers to a collection of data points, and the number of clusters is the number of different groups into which the data set is divided.

[0039] Optionally, the process of determining the number of clusters to be clustered includes: determining the number of clusters to be clustered based on the elbow method.

[0040] Specifically, the process of determining the number of clusters to be clustered may also be performed based on experience or based on methods such as the silhouette coefficient method.

[0041] Optionally, the above process of determining the initial centroid based on the number of clusters and the number of text vectors in each sub-cube includes: determining the probability that each sub-cube contains the initial centroid based on the weighted average probability of the number of text vectors in the corresponding sub-cube; and determining the initial centroid based on the probability that each sub-cube contains the initial centroid and the number of clusters.

[0042] Specifically, the weighted average probability of the number of text vectors in each sub-cube can be calculated based on the following formula:

[0043]

[0044] Where N represents the total number of text vectors, H i represents the i-th subcube, Represents the H in the hypercube i The number of inner text vectors.

[0045] Specifically, the weighted average probability of the number of text vectors in the corresponding sub-cube can be determined as the probability that each sub-cube contains the initial centroid.

[0046] Specifically, other common processing may be performed on the corresponding weighted average probability, such as multiplying it by a corresponding correction coefficient determined based on other influencing factors to obtain the probability that each of the above sub-cubes contains the initial centroid.

[0047] The process of determining the initial centroid based on the probability of each sub-cube containing the initial centroid and the number of clusters includes: obtaining the number of sub-cubes with the largest probability of containing the initial centroid from multiple sub-cubes to obtain the target sub-cube; and determining the geometric center of the target sub-cube as the initial centroid.

[0048] Specifically, each sub-cube is determined to contain an initial centroid based on the weighted average probability of the number of text vectors in the corresponding sub-cube, and the center of the target sub-cube obtained from the cluster number of sub-cubes with the largest probability of the initial centroid is further determined as the initial centroid, which can help ensure the rationality of the selection of the initial centroid.

[0049] Specifically, it is also possible to obtain candidate sub-cubes whose number exceeds the number of clusters, and then determine the initial centroid based on the geometric centers of the candidate sub-cubes.

[0050] Specifically, other points in the corresponding target sub-cube may also be determined as the initial centroid.

[0051] Step 104, clustering each text vector based on the number of clusters and the initial centroid to obtain multiple clustering result clusters, and determining the training text data based on the multiple clustering result clusters. This step, based on steps 101 to 103, establishes the text vector corresponding to each candidate document, draws a hypercube including each text vector, and then evenly divides the hypercube and determines the initial centroid of the cluster based on the number of text vectors in the sub-cubes obtained by the division, which can effectively improve the selection quality of the initial centroid and reduce the sensitivity to the selection of the initial centroid. By clustering each text vector based on the initial centroid, the stability and consistency of the clustering can be improved, and the efficiency of capturing text semantic information can be improved, and the text clustering effect can be optimized; further, by determining the training text data based on the clustering result cluster, the efficiency of obtaining the training text data can be improved and the quality and professionalism of the obtained training text data can be improved.

[0052] Optionally, the process of determining training text data based on multiple clustering result clusters includes: establishing clustering result data sets corresponding to each clustering result cluster based on candidate documents corresponding to each clustering result cluster; and determining the clustering result data set most relevant to the target field among the clustering result data sets as training text data.

[0053] Specifically, the above target field may be one target field or multiple target fields.

[0054] Optionally, the process of determining the clustering result data set most relevant to the target field among the clustering result data sets as the training text data includes:

[0055] For a current target field among multiple target fields, a clustering result data set most relevant to the current target field among various clustering result data sets is obtained to obtain the current target data set, and the current target data set is determined as the training text data corresponding to the current target field.

[0056] Optionally, the training text data acquisition method provided by the embodiment of the present invention also includes: performing dimensionality reduction processing on the hypercube and each text vector obtained in step 101 to obtain a hypercube and text vector after dimensionality reduction, and executing steps 102 to 104 based on the hypercube and text vector after dimensionality reduction.

[0057] Specifically, the training text data acquisition method provided in the embodiment of the present invention can decompose the corresponding computing task into multiple subtasks by utilizing a distributed computing framework, and process the above multiple subtasks in parallel on multiple computing nodes to speed up the computing speed and improve the efficiency of training text data acquisition.

[0058] The following further describes the method for obtaining training text data provided by an embodiment of the present invention. Figure 2 As shown, Figure 1 Step 101 in the method may include the following steps:

[0059] Step 1011, determining the text vector corresponding to each candidate document based on the term frequency-inverse document frequency algorithm.

[0060] Specifically, the above process of determining the text vector corresponding to each candidate document based on the term frequency-inverse document frequency (TF-IDF) algorithm may include: traversing all text data, extracting the words appearing therein to construct a vocabulary; for each candidate document, calculating the term frequency of each word in the vocabulary in the current candidate document, and calculating the inverse document frequency of each word; for each candidate document, calculating the term frequency-inverse document frequency value of each word corresponding to the current candidate document based on the term frequency and corresponding inverse document frequency of each word corresponding to the current candidate document; and constructing the text vector of the current candidate document based on the term frequency-inverse document frequency value of each word corresponding to the current candidate document.

[0061] Step 1012: draw a hypercube based on the number of dimensions of each text vector and the extreme value of each text vector in each dimension.

[0062] Optionally, the process based on the number of dimensions of each text vector includes: determining the dimension of each text vector as the number of dimensions of a hypercube.

[0063] Optionally, the process of drawing a hypercube using the extreme values ​​of each text vector in each dimension includes: determining the extreme values ​​of each text vector in each dimension as vertices of the hypercube.

[0064] It can be understood that since the number of dimensions of each text vector determined based on the word frequency-inverse document frequency algorithm is equal to the number of dimensions of the constructed vocabulary, the dimension of each text vector is determined as the number of dimensions of the hypercube and the extreme values ​​of each text vector in each dimension are determined as the vertices of the hypercube, a hypercube including each text vector can be drawn.

[0065] Specifically, the value obtained by adding a correction value to the extreme value of each text vector in each dimension may be determined as the vertex of the hypercube to ensure that the hypercube completely includes each text vector.

[0066] The embodiment of the present invention can simply and conveniently draw a hypercube that meets the requirements of the embodiment of the present invention, and can facilitate subsequent candidate document clustering and acquisition of training text data.

[0067] The following further describes the method for obtaining training text data provided by an embodiment of the present invention. Figure 3 As shown, Figure 1 Step 104 in may include the following steps:

[0068] Step 1041 , performing this round of clustering on each text vector based on the number of clusters and the initial centroid to obtain multiple current round result clusters.

[0069] Step 1042 , verify the effect of the clustering results of this round based on the intra-cluster error square sum and the intra-cluster square sum of each result cluster of this round.

[0070] Specifically, the above-mentioned Sum of Squared Error within a Cluster (SSE) and Within-Cluster Sum of Squares (WCSS) can be used to evaluate the clustering effect. The smaller the values ​​of the SSE and WCSS, the closer the corresponding data points are to the corresponding centroid, and the better the clustering effect.

[0071] Optionally, the process of verifying the effect of the clustering results of this round based on the intra-cluster error square sum and the intra-cluster square sum of each result cluster of this round includes:

[0072] It is judged whether the intra-cluster error square and the cluster internal square of each result cluster of this round are less than the preset corresponding threshold value. If the intra-cluster error square and the cluster internal square of each result cluster of this round are less than the preset corresponding threshold value, it is judged that the clustering result of this round has passed the effect verification; otherwise, it is judged that the clustering result of this round has not passed the effect verification.

[0073] Step 1043, when the clustering result of this round fails to pass the effect verification, restart the execution to evenly divide the hypercube into multiple sub-cubes, and perform the next round of clustering on each text vector until the clustering result of this round passes the effect verification.

[0074] Step 1044, when the clustering result of this round passes the effect verification, multiple result clusters of this round are determined as multiple clustering result clusters.

[0075] Optional, such as Figure 4 As shown, step 1041 may include the following steps:

[0076] Step 1041A: perform clustering on each text vector based on the number of clusters and the initial centroid to obtain multiple clusters of the current process.

[0077] Specifically, the process of performing clustering on each text vector based on the number of clusters and the initial centroid to obtain multiple clusters of the current round process includes:

[0078] Calculate the Euclidean distance between each text vector and each initial centroid, and assign each text vector to the cluster where the initial centroid with the closest Euclidean distance is located; recalculate the centroid of each cluster and reassign each text vector based on the new centroid until each centroid no longer changes significantly, and determine the corresponding clusters as clusters of this round of process.

[0079] Step 1041B, calculating the silhouette coefficient of each text vector based on multiple clusters of this round of process, and verifying the rationality of the number of clusters based on the silhouette coefficient of each text vector.

[0080] Optionally, the above process of verifying the rationality of the number of clusters based on the contour coefficient of each text vector includes: calculating the difference between 1 and the contour coefficient of each text vector, and when the corresponding differences are all less than a preset difference threshold, determining that the number of clusters has passed the rationality verification, otherwise determining that the number of clusters has not passed the rationality verification.

[0081] It can be understood that the Silhouette Coefficient is used to measure the closeness of a data point to other data points in its cluster during clustering, as well as the degree of separation from data points in other clusters. The value of the Silhouette Coefficient is between [-1, 1]. The closer the Silhouette Coefficient is to 1, the more reasonable the sample clustering is. Therefore, the rationality of the number of clusters can be verified based on whether the difference between 1 and the Silhouette Coefficient of each text vector is less than the preset difference threshold.

[0082] Step 1041C, when the number of clusters fails to pass the rationality verification, the number of clusters is redetermined, and the initial centroid is determined based on the number of clusters and the number of text vectors in each sub-cube again, until the number of clusters passes the rationality verification.

[0083] Step 1041D: when the number of clusters passes the rationality verification, multiple current round process clusters are determined as current round result clusters.

[0084] It is understandable that when performing text clustering, the distribution is often too broad or too narrow. For example, texts related to finance, sports and entertainment English vocabulary are often clustered into one cluster during actual clustering. The embodiment of the present invention performs multiple rounds of clustering, verifies the rationality of the number of clusters in each round of clustering, and verifies the effect of the corresponding clustering results after each round of clustering, and then adjusts the number of clusters based on the rationality verification results, and adjusts the number of sub-cubes and clusters based on the effect verification results, which can help obtain the optimal clustering results, thereby making the training text data finally obtained have the highest quality.

[0085] Figure 5 is a structural diagram of a training text data acquisition device provided in an embodiment of the present invention, and the device is suitable for executing the training text data acquisition method provided in an embodiment of the present invention. Figure 5 As shown, the device may specifically include:

[0086] The hypercube acquisition module 501 is used to establish text vectors corresponding to each candidate document and draw a hypercube including each text vector, which can be used to evenly divide the hypercube to obtain multiple sub-cubes and determine the initial centroid of the cluster based on the amount of text in each sub-cube.

[0087] Optionally, the hypercube acquisition module 501 can be specifically used to determine the text vector corresponding to each candidate document based on a term frequency-inverse document frequency algorithm; and

[0088] A hypercube is drawn based on the number of dimensions of each text vector and the extreme value of each text vector in each dimension.

[0089] The sub-cube acquisition module 502 is used to evenly divide the hypercube into a plurality of sub-cubes, so as to facilitate determining the initial centroid of the cluster based on the number of texts in each sub-cube.

[0090] The cluster number and centroid acquisition module 503 is used to determine the number of clusters to be clustered and determine the initial centroid based on the number of clusters and the number of text vectors in each sub-cube. It can effectively improve the quality of the initial centroid selection, reduce the sensitivity to the initial centroid selection, improve the stability and consistency of clustering, and improve the efficiency of capturing text semantic information, and optimize the text clustering effect.

[0091] Optionally, the cluster number and centroid acquisition module 503 can be specifically used to determine the number of clusters for clustering based on the elbow method;

[0092] Determining the probability that each sub-cube contains the initial centroid based on the weighted average probability of the number of text vectors in the corresponding sub-cube; and

[0093] The initial centroid is determined based on the probability that each sub-cube contains the initial centroid and the number of clusters.

[0094] Optionally, the cluster number and centroid acquisition module 503 can be specifically used to obtain the number of cluster sub-cubes with the highest probability of containing the initial centroid from multiple sub-cubes to obtain a target sub-cube; and

[0095] The geometric center of the target subcube is determined as the initial centroid.

[0096] Clustering result and training text data acquisition module 504 is used to cluster each text vector based on the number of clusters and the initial centroid to obtain multiple clustering result clusters, and determine the training text data based on the multiple clustering result clusters. In combination with modules 501 to 503, by establishing the text vector corresponding to each candidate document and drawing a hypercube including each text vector, and then evenly dividing the hypercube and determining the initial centroid of the cluster based on the number of text vectors in the sub-cubes obtained by the division, the quality of the selection of the initial centroid can be effectively improved, and the sensitivity to the selection of the initial centroid can be reduced. By clustering each text vector based on the initial centroid, the stability and consistency of the clustering can be improved, and the efficiency of capturing text semantic information can be improved, and the text clustering effect can be optimized; further, by determining the training text data based on the clustering result cluster, the efficiency of obtaining the training text data can be improved and the quality and professionalism of the obtained training text data can be improved.

[0097] Optionally, the clustering result and training text data acquisition module 504 can be specifically used to perform current round clustering on each text vector based on the number of clusters and the initial centroid to obtain multiple current round result clusters;

[0098] The clustering results of this round are verified based on the intra-cluster error square and the intra-cluster sum of squares of each cluster of results in this round;

[0099] When the clustering result of this round fails to pass the effect verification, restart the execution to evenly divide the hypercube into multiple sub-cubes, and perform the next round of clustering on each text vector until the clustering result of this round passes the effect verification; and

[0100] When the clustering results of this round pass the effect verification, multiple result clusters of this round are determined as multiple clustering result clusters.

[0101] Optionally, the clustering result and training text data acquisition module 504 can be specifically used to perform current round clustering on each text vector based on the number of clusters and the initial centroid to obtain multiple current round process clusters;

[0102] Calculate the silhouette coefficient of each text vector based on multiple clusters of this round of processes, and verify the rationality of the number of clusters based on the silhouette coefficient of each text vector;

[0103] When the number of clusters fails to pass the rationality verification, the number of clusters is re-determined, and the initial centroid is determined based on the number of clusters and the number of text vectors in each sub-cube again, until the number of clusters passes the rationality verification; and

[0104] When the number of clusters passes the rationality verification, multiple current-round process clusters are determined as current-round result clusters.

[0105] Optionally, the clustering result and training text data acquisition module 504 can be specifically used to:

[0106] Establishing clustering result data sets corresponding to each clustering result cluster based on the candidate documents corresponding to each clustering result cluster; and

[0107] The clustering result data set most relevant to the target field among the clustering result data sets is determined as the training text data.

[0108] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0109] An embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the training text data acquisition method provided in any of the above embodiments is implemented.

[0110] An embodiment of the present invention further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method for acquiring training text data provided in any of the above embodiments is implemented.

[0111] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the training text data acquisition method as described in any one of the embodiments of the present invention.

[0112] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of an electronic device suitable for implementing an embodiment of the present invention. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0113] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0114] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0115] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0116] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0117] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0118] The modules and / or units involved in the embodiments of the present invention may be implemented in software or hardware. The modules and / or units described may also be arranged in a processor, for example, may be described as: a processor including a hypercube acquisition module, a subcube acquisition module, a cluster number and centroid acquisition module, and a clustering result and training text data acquisition module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.

[0119] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device implements: establishing text vectors corresponding to each candidate document, and drawing a hypercube including each text vector; dividing the hypercube into multiple sub-cubes on average; determining the number of clusters to be clustered and determining the initial centroid based on the number of clusters and the number of text vectors in each sub-cube; and clustering each text vector based on the number of clusters and the initial centroid to obtain multiple clustering result clusters, and determining training text data based on the multiple clustering result clusters.

[0120] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for acquiring training text data, characterized in that: include: Establishing text vectors corresponding to each candidate document and drawing a hypercube including each text vector; Dividing the hypercube into a plurality of sub-cubes on average; Determine the number of clusters to be clustered and determine the initial centroid based on the number of clusters and the number of text vectors in each sub-cube; as well as The text vectors are clustered based on the number of clusters and the initial centroids to obtain a plurality of clustering result clusters, and training text data is determined based on the plurality of clustering result clusters.

2. The method for acquiring training text data according to claim 1, characterized in that: The step of establishing text vectors corresponding to each candidate document and drawing a hypercube including each text vector includes: Determining the text vector corresponding to each candidate document based on a term frequency-inverse document frequency algorithm; and The hypercube is drawn based on the number of dimensions of each text vector and the extreme value of each text vector in each dimension.

3. The method for acquiring training text data according to claim 1, characterized in that: The step of determining the number of clusters to be clustered and determining the initial centroid based on the number of clusters and the number of text vectors in each sub-cube comprises: Determine the number of clusters for clustering based on the elbow method; Determining the probability that each sub-cube contains an initial centroid based on the weighted average probability of the number of text vectors in the corresponding sub-cube; and The initial centroid is determined based on the probability that each sub-cube contains the initial centroid and the number of clusters.

4. The method for acquiring training text data according to claim 3, characterized in that: The determining the initial centroid based on the probability that each sub-cube contains the initial centroid and the number of clusters includes: Obtaining the number of sub-cubes of clusters with the highest probability of containing the initial centroid from the multiple sub-cubes to obtain a target sub-cube; and The geometric center of the target sub-cube is determined as the initial centroid.

5. The method for acquiring training text data according to claim 1, characterized in that: The clustering of the text vectors based on the number of clusters and the initial centroid to obtain a plurality of clustering result clusters includes: Performing current round clustering on each text vector based on the number of clusters and the initial centroid to obtain multiple current round result clusters; The clustering results of this round are verified based on the intra-cluster error square and the intra-cluster sum of squares of each cluster of results in this round; When the clustering result of this round fails to pass the effect verification, restarting the even division of the hypercube into a plurality of sub-cubes and performing the next round of clustering on each text vector until the clustering result of this round passes the effect verification; and When the clustering results of this round pass the effect verification, the multiple result clusters of this round are determined as the multiple clustering result clusters.

6. The method for acquiring training text data according to claim 5, characterized in that: The clustering of each text vector based on the number of clusters and the initial centroid to obtain multiple result clusters of this round includes: Performing current round clustering on each text vector based on the number of clusters and the initial centroid to obtain a plurality of current round process clusters; Calculating the silhouette coefficient of each text vector based on the multiple current process clusters, and verifying the rationality of the number of clusters based on the silhouette coefficient of each text vector; When the number of clusters fails to pass the rationality verification, the number of clusters is redetermined, and the determination of the initial centroid based on the number of clusters and the number of text vectors in each sub-cube is restarted until the number of clusters passes the rationality verification; and When the number of clusters passes the rationality verification, the multiple current-round process clusters are determined as the current-round result clusters.

7. The method for acquiring training text data according to claim 1, characterized in that: The step of determining the training text data based on the plurality of clustering result clusters comprises: Establishing clustering result data sets corresponding to each clustering result cluster based on the candidate documents corresponding to each clustering result cluster; and The clustering result data set most relevant to the target field among the clustering result data sets is determined as the training text data.

8. A training text data acquisition device, characterized in that: include: A hypercube acquisition module, used to establish text vectors corresponding to each candidate document and draw a hypercube including each text vector; A sub-cube acquisition module, used for evenly dividing the hypercube into a plurality of sub-cubes; A cluster number and centroid acquisition module, used to determine the number of clusters for clustering and determine the initial centroid based on the number of clusters and the number of text vectors in each sub-cube; as well as The clustering result and training text data acquisition module is used to cluster the text vectors based on the number of clusters and the initial centroid to obtain multiple clustering result clusters, and determine the training text data based on the multiple clustering result clusters.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the training text data acquisition method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the training text data acquisition method as described in any one of claims 1 to 7 is implemented.

Images