Document processing method, device and server

Abstract

The embodiment of the present application provides a document processing method, device and server, the method comprising: obtaining a first policy document set, the first policy document set including at least one policy document; preprocessing the first policy document set to obtain a word set, the word set including multiple words; extracting a label set for each industry category under multiple industry categories from the first policy document set according to the word set and preset labels; obtaining a second policy document set, the second policy document set including at least one policy document corresponding to each industry category; according to the second policy document set, obtaining the word frequency features of each label in the label set of each industry category, and performing data processing according to the word frequency features of each label in the label set of each industry category. The present application can improve the reliability of data processing using the word frequency features of policy documents.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a document processing method, device and server. Background Art

[0002] The development of technologies such as big data and artificial intelligence has provided high-quality data services to a large number of users. Among these technologies, the word frequency features of documents are widely used for data processing, such as document classification. At present, word frequency analysis tools can usually be used to obtain the word frequency features of documents. However, in some cases, such as policy documents issued by the whole country and local governments, if word frequency analysis tools are used to obtain the word frequency features of policy documents, the word frequency features obtained will be less targeted, which will lead to lower reliability of the data processing process. Summary of the invention

[0003] The embodiments of the present application provide a document processing method, device, and server, which can improve the reliability of data processing using word frequency features of policy documents.

[0004] In a first aspect, an embodiment of the present application provides a document processing method, including:

[0005] Acquire a first policy document set, wherein the first policy document set includes at least one policy document;

[0006] Preprocessing the first policy document set to obtain a word set, wherein the word set includes a plurality of words;

[0007] According to the word set and the preset labels, extracting a label set for each industry category under multiple industry categories from the first policy document set;

[0008] Acquire a second policy document set, where the second policy document set includes at least one policy document corresponding to each industry category;

[0009] According to the second policy document set, the word frequency characteristics of each tag in the tag set of each industry category are obtained, and data processing is performed according to the word frequency characteristics of each tag in the tag set of each industry category.

[0010] Optionally, the data processing according to the word frequency feature of each tag in the tag set of each industry category includes:

[0011] The preset model is trained using the second policy document set and the word frequency characteristics of each label in the label set of each industry category to obtain a policy document classification model for industry category prediction.

[0012] Optionally, extracting a label set for each industry category under multiple industry categories from the first policy document set according to the word set and the preset label includes:

[0013] According to the label group of each industry category under the multiple industry categories included in the preset label, determine a plurality of texts corresponding to the label group of the industry category in the first policy document set;

[0014] Determining a text set for clustering corresponding to the industry category from the multiple texts according to the word set;

[0015] Clustering the text set of each industry category using a topic model to obtain a clustering result, wherein the clustering result includes a word distribution under each topic in a plurality of topics;

[0016] A tag set for each industry category is constructed according to the word distribution under each topic in the multiple topics.

[0017] Optionally, determining a text set for clustering corresponding to the industry category from the multiple texts according to the word set includes:

[0018] Performing vector representation on each of the multiple words included in the word set to obtain a word vector set, wherein the word vector set includes the word vector of each word;

[0019] Using the word vector set to perform vector representation on each of the multiple texts to obtain a text vector for each text;

[0020] Calculating the similarity between the multiple texts using the text vector of each text;

[0021] According to the similarities between the multiple texts, determining texts whose similarities are greater than or equal to a preset similarity from the multiple texts;

[0022] The texts with similarity greater than or equal to the preset similarity are used as the text set for clustering corresponding to the industry category.

[0023] Optionally, clustering the text set corresponding to each industry category using a topic model to obtain a clustering result includes:

[0024] Determine a subword set corresponding to the text set of each industry category from the word set;

[0025] The topic model is used to cluster the subword sets corresponding to the text sets of each industry category to obtain clustering results.

[0026] Optionally, the method further comprises:

[0027] Obtain the policy document to be predicted;

[0028] The policy document to be predicted is used as input data of the policy document classification model. The policy document prediction model predicts the industry category of the policy document to be predicted to obtain a prediction result, which indicates the industry category corresponding to the policy document to be predicted.

[0029] Optionally, the prediction result includes the probability of the policy document to be predicted in each industry category among the multiple industry categories, and the method further includes:

[0030] Get the threshold for successful prediction of industry categories;

[0031] Calculating the difference between the probability of the policy document to be predicted in the first industry category and the probability of other industry categories except the first industry category among the multiple industry categories; the first industry category refers to the industry category with the highest probability among the multiple industry categories;

[0032] Select at least one industry category whose difference is less than the threshold value from the other industry categories;

[0033] The first industry category and the at least one industry category are determined as industry categories corresponding to the policy document to be predicted.

[0034] Optionally, before obtaining the threshold for successful prediction of the industry category, the method further includes:

[0035] When the first number of policy documents included in the second policy document set are predicted to have a correct industry category, calculating the difference between the maximum probability value of the correct industry category of each policy document in the first number of policy documents and the maximum probability value of the incorrect industry category of the policy document;

[0036] The difference corresponding to each policy document is averaged and the average of the differences is obtained as the threshold for successful industry category prediction.

[0037] In a second aspect, an embodiment of the present application provides a document processing device, including:

[0038] An acquiring unit, configured to acquire a first policy document set, wherein the first policy document set includes at least one policy document;

[0039] A processing unit, configured to pre-process the first policy document set to obtain a word set, wherein the word set includes a plurality of words;

[0040] The processing unit is further configured to extract a label set for each industry category under a plurality of industry categories from the first policy document set according to the word set and the preset label;

[0041] The acquisition unit is further configured to acquire a second policy document set, wherein the second policy document set includes at least one policy document corresponding to each industry category;

[0042] The processing unit is further used to obtain the word frequency characteristics of each tag in the tag set of each industry category according to the second policy document set, and perform data processing according to the word frequency characteristics of each tag in the tag set of each industry category.

[0043] In a third aspect, an embodiment of the present application provides a server comprising a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method described in the first aspect.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method described in the first aspect.

[0045] To summarize, the server can obtain a first policy document set, and preprocess the first policy document set to obtain a word set, so as to extract a label set for each industry category under multiple industry categories from the first policy document set based on the word set and preset labels; the server can also obtain a second policy document set, and obtain the word frequency features of each label in the label set of each industry category based on the second policy document set, so as to perform data processing based on the word frequency features of each label in the label set of each industry category, thereby improving the effectiveness of the word frequency features of the obtained policy documents, and further improving the reliability of using the word frequency features of the policy documents for data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1a It is a flowchart of a document processing method provided by an embodiment of the present application;

[0048] Figure 1b This is a schematic diagram of a label extraction process provided by an embodiment of the present application;

[0049] Figure 1cThis is a schematic diagram of a label set of grains in agriculture provided by an embodiment of the present application;

[0050] Figure 1d This is a schematic diagram of a label set of agricultural grains with a word frequency feature added provided in an embodiment of the present application;

[0051] Figure 2a It is a flowchart of another document processing method provided by an embodiment of the present application;

[0052] Figure 2b is a schematic diagram of parameters of a policy document classification model provided in an embodiment of the present application;

[0053] Figure 2c It is a schematic diagram of a document processing process provided by an embodiment of the present application;

[0054] Figure 3 is a structural schematic diagram of a document processing device provided in an embodiment of the present application;

[0055] Figure 4 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0057] See also Figure 1a , is a flowchart of a document processing method provided in an embodiment of the present application. The document processing method can be applied to a server. The server can be a server or a server cluster. Specifically, the method may include the following steps:

[0058] S101. Obtain a first policy document set, where the first policy document set includes at least one policy document.

[0059] The at least one is one or more, and the policy document may refer to a policy-related document issued by local governments across the country. It should be noted that the document processing method described in the embodiment of the present application can be applied to other types of documents in addition to policy documents.

[0060] In one embodiment, the server obtains the first policy document set by obtaining the first policy document set stored in the server itself.

[0061] In one embodiment, the server may obtain the first policy document set by obtaining the first policy document set from a database.

[0062] In one embodiment, the server may obtain the first policy document set by receiving the first policy document set uploaded by a terminal, including but not limited to a tablet computer, a laptop computer, a desktop computer, and other intelligent terminals.

[0063] S102: Preprocess the first policy document set to obtain a word set, where the word set includes a plurality of words.

[0064] The server can pre-process the policy documents included in the first policy document set through natural language processing (NLP) technology, such as performing word segmentation to obtain a word set. For example, the word segmentation method can be "adjusting the tax amount standard of urban land use tax in the whole province is an important measure for the provincial party committee and the provincial government to promote the "three cuts, one reduction and one supplement", develop advantageous and characteristic industries, and do a good job in the "three major articles". "

[0065] Among them, the NLP technology includes but is not limited to any of the following: a dictionary-based word segmentation method, a statistics-based word segmentation method, a rule-based word segmentation method, a Chinese word segmentation method based on character annotation, and a Chinese word segmentation method based on artificial intelligence technology.

[0066] In one embodiment, the server may obtain a preset dictionary, which may include professional terms in policy-related fields. Accordingly, the server pre-processes each policy document included in the first policy document set by using NLP technology to obtain a word set. The server pre-processes each policy document included in the first policy document set by using the dictionary and the NLP technology to obtain a word set.

[0067] S103: extracting a label set for each industry category under multiple industry categories from the first policy document set according to the word set and the preset labels.

[0068] In an embodiment of the present application, the server can extract a label set of each industry category under multiple industry categories from the first policy document set according to the word set and the preset label, so as to achieve the purpose of extracting labels. Among them, the label mentioned in the embodiment of the present application can be a word. The industry category can refer to an industry category or an industry classification. In one embodiment, the industry category can also refer to a category or classification determined according to the industry category or industry classification. For example, the industry category can be a first industry category and / or at least one corresponding second industry category. Among them, the second industry category is a sub-industry category under the first industry category. For example, the first industry category is agriculture, forestry, animal husbandry, and fishery. The second industry category can be any one of agriculture, forestry, animal husbandry, and fishery. In one embodiment, the industry category can also include a third industry category and the like on the basis of the first industry category and / or the second industry category. Among them, the third industry category is a sub-industry category under the second industry category, and so on. For example, the second industry category is agriculture, and the third industry category can be grain or planting, etc. See Table 1, which is an industry category table provided in an embodiment of the present application. It should be noted that Table 1 is only an example of an industry category classification. The specific industry category classification can also adopt other classification forms, which are not listed one by one in the embodiments of the present application.

[0069] Table 1

[0070]

[0071] In one embodiment, the server can perform entity clustering on the first policy document set according to the word set and the preset labels to obtain a label set for each industry category under multiple industry categories. Among them, the entity clustering process can be understood as a label extraction process for industry categories. One of the entity clustering methods will be explained below. It should be noted that the embodiments of the present application can also use other entity clustering methods to obtain a label set for each industry category in multiple industry categories, which are not listed one by one in the embodiments of the present application.

[0072] In one embodiment, Figure 1b As shown, the server extracts a tag set for each industry category under multiple industry categories from the first policy document set according to the word set and the preset tags, which may include the following steps:

[0073] S1031. Determine, according to a label group of each industry category under a plurality of industry categories included in a preset label, a plurality of texts corresponding to the label group of the industry category in the first policy document set.

[0074] The tag group may include at least one tag. For example, when the multiple industry categories include grains under agriculture, the tag group of grains under agriculture may include broad beans, corn, wheat, etc. The text refers to text including one or more tags in the tag group.

[0075] In one embodiment, the server may use a keyword matching method to determine a plurality of texts corresponding to the label group of each industry category under a plurality of industry categories included in the preset label in the first policy document set.

[0076] S1032: Determine a text set for clustering corresponding to the industry category from the multiple texts according to the word set.

[0077] The server may determine a text set for clustering corresponding to the industry category from the multiple texts according to the word set, so as to perform cluster analysis.

[0078] In one embodiment, the server determines the text set for clustering corresponding to the industry category from the multiple texts according to the word set by the server performing vector representation on each of the multiple words included in the word set to obtain a word vector set, and the word vector set includes the word vector of each word; the server uses the word vector set to perform vector representation on each of the multiple texts to obtain the text vector of each text, and uses the text vector of each text to calculate the similarity between the multiple texts, so as to determine the text with a similarity greater than or equal to a preset similarity from the multiple texts according to the similarity between the multiple texts, so as to use the text with a similarity greater than or equal to the preset similarity as the text set for clustering corresponding to the industry category. In this way, some texts with high similarity can be determined for each industry category for clustering analysis.

[0079] In one embodiment, the server includes but is not limited to using a word vector model or a pre-trained bidirectional language understanding model or the like to perform vector representation on each of the multiple words included in the word set to obtain a word vector set.

[0080] S1033. Cluster the text set of each industry category using a topic model to obtain a clustering result, wherein the clustering result includes word distribution under each topic in multiple topics.

[0081] The server can cluster the text set of each industry category using the topic model to obtain a clustering result, which includes the word distribution under each topic in multiple topics, so as to construct a tag set for each industry category. The topic model can be a topic model such as a latent Dirichlet distribution model, which is not limited in the embodiment of the present application.

[0082] For example, the multiple industry categories include industry category 1, industry category 2, industry category 3, industry category 4, and industry category 5. The server can use the topic model to cluster the text set of industry category 1, the text set of industry category 2, the text set of industry category 3, the text set of industry category 4, and the text set of industry category 5 to obtain clustering results.

[0083] In one embodiment, the server uses the topic model to cluster the text set corresponding to each industry category, and the clustering result can be obtained by the server determining the subword set corresponding to the text set of each industry category from the word set, and clustering the subword set corresponding to the text set of each industry category using the topic model to obtain the clustering result. In one embodiment, the server determines the subword set corresponding to the text set of each industry category from the word set, and the server determines the words corresponding to each text in the text set of each industry category from the word set to construct the word set corresponding to the text set. In this way, the server can directly determine the subword set corresponding to the text set of each industry category based on the word set for clustering, which can effectively improve the efficiency of cluster analysis.

[0084] In one embodiment, in addition to clustering in the above manner, the text set of each industry category can also be preprocessed, such as performing word segmentation to obtain a target word set corresponding to the text set, and the target word set is input into the topic model. The target word set is clustered through the topic model to obtain a clustering result.

[0085] S1034. Construct a tag set for each industry category according to the word distribution under each topic in the multiple topics.

[0086] After obtaining the clustering results, the server can construct a label set for each industry category based on the word distribution under each of the multiple topics included in the clustering results. In this way, for example, a label set for grains under agriculture as shown in FIG1c can be constructed. It should be noted that Figure 1c Shown are only some examples of labels from the label set for grains under Agriculture.

[0087] In one embodiment, the server constructs a tag set for each industry category based on the word distribution under each topic in multiple topics, so that the server can determine the industry category corresponding to each topic in multiple topics, and determine the word distribution under each topic as the tag set of the industry category corresponding to the topic.

[0088] In one embodiment, considering that the word distribution obtained by the above method may not be accurate enough, the server can also filter the word distribution under each topic, and determine the word distribution under each topic after filtering as a label set of the industry category corresponding to the topic. The filtering method includes but is not limited to receiving a deletion instruction, deleting the corresponding words according to the deletion instruction; or using a preset model to identify whether each word in the word distribution under each topic belongs to the topic, and deleting the words that do not belong to the topic. The preset model includes but is not limited to models such as convolutional neural network models.

[0089] S104: Obtain a second policy document set, where the second policy document set includes at least one policy document corresponding to each industry category.

[0090] The second policy document set may be different from the first policy document set. In one embodiment, the second policy document set may be completely the same as or partially the same as the second policy document set. For example, referring to Table 2, an example of the second policy document set may be as follows:

[0091] Table 2

[0092]

[0093] In one embodiment, in order to ensure the balance of industry categories, the number of at least one policy document corresponding to each industry category may be the same.

[0094] In one embodiment, the server obtains the second policy document set by obtaining the second policy document set stored in the server itself.

[0095] In one embodiment, the server may obtain the second policy document set by obtaining the second policy document set from a database.

[0096] In one embodiment, the server may obtain the second policy document set by receiving the second policy document set uploaded by the terminal.

[0097] S105. According to the second policy document set, the word frequency features of each tag in the tag set of each industry category are obtained, and according to the word frequency features of each tag in the tag set of each industry category.

[0098] In an embodiment of the present application, the server can obtain the word frequency characteristics of each tag in the tag set of each industry category based on the second policy document set, and perform data processing based on the word frequency characteristics of each tag in the tag set of each industry category. In one embodiment, the word frequency characteristics can be word frequency information, such as word frequency. In one embodiment, the word frequency can refer to the word frequency in the second policy document set. The data processing includes but is not limited to any of the following: storage processing, text processing, and image processing.

[0099] In one embodiment, the server obtains the word frequency features of each tag in the tag set of each industry category according to the second policy document set by calculating the word frequency features corresponding to each tag in the tag set of each industry category in the second policy document set.

[0100] In one embodiment, the server may generate a tag set for each industry category with a word frequency feature added. For example, the server may generate Figure 1d The label set of grains under agriculture with the word frequency feature added is shown. It should be noted that Figure 1d Shown are only some examples of labels in the label set for grains under agriculture with word frequency features added.

[0101] In one embodiment, when the data processing includes image processing, the server performs data processing according to the word frequency features of each tag in the tag set of each industry category by generating a picture including each tag in the tag set, such as a tag cloud, according to the word frequency features of each tag in the tag set of each industry category; or, the server obtains the tag set of each industry category with the word frequency features added, and generates a picture including each tag in the tag set, such as a tag cloud, according to the tag set of each industry category with the word frequency features added. In this way, the image content is enriched through the tag set, and the image content is made more targeted, accurate and reliable.

[0102] In one embodiment, the server may obtain drawing parameters, and draw pictures including each tag in the tag set according to the drawing parameters and the word frequency characteristics of each tag in the tag set of each industry category; or, draw pictures including each tag in the tag set according to the drawing parameters and the tag set of each industry category with the word frequency characteristics added. The drawing parameters include at least one of the following: font, arrangement shape, background picture, font color.

[0103] In one embodiment, when the data processing includes text processing, the server performs data processing according to the tag set of each industry category to which the word frequency feature is added. The server can train a preset model according to the word frequency features of each tag in the second policy document set and the tag set of each industry category to obtain a policy document classification model for industry category prediction. In one embodiment, each policy document included in the second policy document set can carry its own industry category. The preset model includes but is not limited to a naive Bayes model or a classification regression model (or a classification and regression algorithm model) and other models that can be used for text classification.

[0104] visible, Figure 1a In the illustrated embodiment, the server can obtain a first policy document set, and preprocess the first policy document set to obtain a word set, and extract a label set for each industry category under multiple industry categories from the first policy document set based on the word set and preset labels; the server can obtain a second policy document set, and based on the second policy document set, count the word frequency features of each label in the label set of each industry category, and generate a label set for each industry category with the word frequency features added, and perform data processing based on the label set of each industry category with the word frequency features added, thereby improving the effectiveness of the word frequency features of the obtained policy documents, and further improving the reliability when using the word frequency features of the policy documents for data processing.

[0105] See also Figure 2a , is a flow chart of another document processing method provided in an embodiment of the present application. The document processing method is applied to a server. The server may be a server or a server cluster. Specifically, the method may include the following steps:

[0106] S201. Obtain a first policy document set, where the first policy document set includes at least one policy document.

[0107] S202: Preprocess the first policy document set to obtain a word set, where the word set includes a plurality of words.

[0108] S203: extracting a label set for each industry category under multiple industry categories from the first policy document set according to the word set and the preset labels.

[0109] S204: Obtain a second policy document set, where the second policy document set includes at least one policy document corresponding to each industry category.

[0110] Among them, steps S201 to S204 can be found in Figure 1a Steps S101 to S204 in the embodiment are not described in detail in the embodiment of the present application.

[0111] S205. According to the second policy document set, the word frequency characteristics of each label in the label set of each industry category are obtained, and the preset model is trained using the word frequency characteristics of each label in the second policy document set and the label set of each industry category to obtain a policy document classification model for industry category prediction.

[0112] When the data processing includes text processing, the server performs data processing according to the tag set of each industry category to which the word frequency feature is added. The server can train the preset model according to the word frequency features of each tag in the second policy document set and the tag set of each industry category to obtain a policy document classification model for industry category prediction. In one embodiment, each policy document included in the second policy document set can carry its own industry category. The preset model includes but is not limited to a naive Bayesian model, a classification regression model (or a classification and regression algorithm model), a random forest, and the like, which can be used for text classification. The training process will be described below using the preset model as a Bayesian model.

[0113] In one embodiment, the process of obtaining the word frequency characteristics of each tag in the tag set of each industry category according to the second policy document set in step S205 can be included as a module in the training process. In one embodiment, this process may not be included as a module in the training process, and the embodiment of the present application does not limit it.

[0114] In one embodiment, when the preset model is a naive Bayes classification model, the above training process may include at least the following three modules: calculating the word frequency characteristics of the label, calculating the probability of the industry category, and calculating the statistical probability of the label under each industry category. Specifically, it can be as follows:

[0115] 1. Calculate the word frequency feature Count (Word k |DOC):

[0116] Count(Word k |DOC)=Frq(Word k |DOC1,…,DOC m ) Formula 1.1;

[0117] Among them, Word k represents label k, Frq represents the word frequency statistics of all policy documents in the second policy document set, and DOC represents the document set {DOC1,…,DOC m}, m represents the number of documents.

[0118] 2. The probability P(CATEi ):

[0119] P(CATE i )=NUM(CATE i ) / m Formula 1.2;

[0120] Among them, NUM(CATE i ) represents the number of industry categories i.

[0121] 3. Statistical probability P (Word k |CATE i ), which may include calculating tf_idf features, word vector similarity metrics, such as word vector similarity, etc., and adding statistically significant feature values ​​(features must be normalized in advance). Specifically, the statistical probability P (Word k) of label k under industry category i can be calculated by the following formula k |CATE i ):

[0122] P(Word k |CATE i )=TF_IDF(Word k |CATE i )+SIM_VEC(Word k |CATE i ) +……

[0123] Formula 1.3;

[0124] Among them, TF_IDF represents the tf_idf feature, and SIM_VEC represents the word vector similarity measure.

[0125] A. The tf_idf feature can be calculated by the following formula:

[0126] TF_IDF(Word k |CATE i )=Count(Word k |CATE i )*INV(CATE i )=Count (Word k |DOC1,…,DOC t )*INV(DOC1,…,DOC t |CATE i ) Formula 1.4;

[0127] Among them, INV(CATE i ) represents the inverse document frequency of all documents in industry category i, DOC1,…,DOC trepresents the set of all documents in industry category i, and t represents the number of all documents in industry category i.

[0128] B. The SIM_VEC word vector similarity metric can be calculated using the following formula:

[0129] SIM_VEC(Word k |CATE i )=COS(VEC(Word k ),VEC(CATE i ))

[0130] Formula 1.5;

[0131] Among them, COS represents cosine distance and VEC represents word vector.

[0132] In one embodiment, the above method can be used to obtain Figure 2b The parameter diagram of the policy document classification model shown in Figure 1 is as follows: Figure 2b The parameters of the policy document classification model shown can be used in the subsequent industry category prediction process.

[0133] S206: Obtain the policy document to be predicted.

[0134] S207, using the policy document to be predicted as input data of the policy document classification model, and having the policy document prediction model predict the industry category of the policy document to be predicted to obtain a prediction result, wherein the prediction result indicates the industry category corresponding to the document to be predicted.

[0135] Among them, the prediction result may include the industry category corresponding to the policy document to be predicted. In one embodiment, the prediction result may also include a probability value corresponding to the industry category corresponding to the policy document to be predicted. In one embodiment, the industry category corresponding to the policy document to be predicted may be a first industry category. The first industry category refers to an industry category with the highest probability among multiple industry categories.

[0136] In one embodiment, the prediction result includes the probability of the policy document to be predicted in each industry category among the multiple industry categories. The server can obtain the threshold for successful industry category prediction; the server calculates the difference between the probability of the policy document to be predicted in the first industry category and the probability of other industry categories except the first industry category among the multiple industry categories; the server selects at least one industry category from the other industry categories whose difference is less than the threshold, and determines the first industry category and the at least one industry category as the industry category corresponding to the policy document to be predicted. In this way, in addition to finding the industry category with the largest probability value as the industry category corresponding to the policy document to be predicted, other qualified industry categories can also be found as the industry category corresponding to the policy document to be predicted, thereby achieving the purpose of multi-industry mapping of the policy document to be predicted.

[0137] For example, the multiple industry categories include industry category 1, industry category 2, industry category 3, industry category 4 and industry category 5. The probability of the document to be predicted in industry category 1 is 98%, the probability in industry category 2 is 90%, the probability in industry category 3 is 60%, the probability in industry category 4 is 33%, and the probability in industry category 5 is 10%. At this time, the server can determine the industry category with the highest probability among the five industry categories. For industry category 1, the server calculates the difference between the probability of industry category 1 and each industry category in the other four industry categories. After calculating the four differences, if the threshold for successful industry category prediction is 10%, the server can determine to select industry category 2 from the four industry categories with a difference less than 10% (through calculation, it is found that the difference between industry category 1 and industry category 2 is 8%), so the server can determine industry category 1 and industry category 2 as the industry category corresponding to the document to be predicted.

[0138] In one embodiment, assuming that the preset model is a naive Bayes classification model, and the policy document to be predicted is DOC x , then DOC x CATE in industry category i i The probability can be obtained according to the Bayesian formula and the statistical probability calculated above. Specifically, the policy document classification model can calculate DOC using the following formula: x CATE in industry category i i Probability:

[0139] P(CATE i |DOC x )

[0140] =P(DOC x CATE i )*P(CATE i ) / P(DOC x)

[0141] =P(DOC x |CATE i )*P(CATE i ) / 1

[0142] =P(DOC x |CATE i )*P(CATE i )

[0143] =P(Word1,…,Word k |CATE i )*P(CATE i )

[0144] =P(Word k |Word k-1 ,…,Word1,CATE i )*…*P(Word2|Word1,CATE i )* P(Word1|CATE i )*P(CATE i )

[0145] =P(Word k |CATE i )*…*P(Word2|CATE i )*P(Word1|CATE i )*P(CATE i )

[0146] Formula 1.6;

[0147] Since it is assumed that there is a DOC x Calculation is based on the premise that P(DOC x ) is always 1. Word1,…, Word k Indicates DOC x The tags that appear in .

[0148] In one embodiment, when the first number of policy documents included in the second policy document set are predicted to have the correct industry category, the server can calculate the difference between the maximum probability value of the correct industry category of each policy document in the first number of policy documents and the maximum probability value of the probability value of the incorrect industry category of the policy document; average the difference values ​​corresponding to each policy document, and obtain the average of the differences as the threshold for successful industry category prediction. Among them, the "first" and "second" mentioned in the embodiments of the present application do not represent a sequential relationship.

[0149] For example, the threshold for successful industry category prediction is set to THRESHOLD. For the policy document DOC in the first number of policy documents, the server can obtain the maximum probability value of the correct industry category of DOC in the following manner:

[0150] max{P r (CATE i |DOC),P r2 (CATE i+1 |DOC),…}

[0151] Formula 1.7;

[0152] The server can obtain the maximum probability value among the probability values ​​of incorrect industry categories of DOC in the following way:

[0153] max{P n (CATE i |DOC),P n2 (CATE i+1 |DOC),…}

[0154] Formula 1.8;

[0155] The server can calculate the difference between the maximum probability value of the correct industry category and the maximum probability value of the incorrect industry category in the following way:

[0156] ERR i =max{P r (CATE i |DOC),P r2 (CATE i+1 |DOC),…}-max{P n (CATE i |DOC),P n2 (CATE i+1 |DOC),…}

[0157] Formula 1.9;

[0158] Assuming the first quantity is q, q ERRs can be obtained as follows: i The mean of :

[0159] MEAN=(ERR i,1 +ERR i,2 +…+ERR i,k ) / q

[0160] Formula 2.0;

[0161] THRESHOLD=MEAN

[0162] Formula 2.1; Accordingly, if the first industry category is industry category i, when DOC x The probability P of being in industry category i max (CATE i |DOC x ) and the probability value P(CATE j |DOC x ) is less than THRESHOLD, industry category j can be determined as DOC x Corresponding industry categories; or industry category i and industry category j can be determined as DOC x Corresponding industry categories. The process can be expressed as follows:

[0163] P max (CATE i |DOC x )-P(CATE j |DOC x ) <THRESHOLD

[0164] Formula 2.2;

[0165] In one embodiment, when receiving a search request sent by a terminal, the server can generate search results based on the keywords carried in the search request and return the search results to the terminal, wherein the search results carry information of policy documents matching the keywords, such as link addresses or summary information.

[0166] visible, Figure 2a In the illustrated embodiment, the server trains a preset model based on the word frequency characteristics of each tag in the second policy document set and the tag set of each industry category, and obtains a policy document classification model for predicting the industry category of the policy document to be predicted, thereby realizing rapid division of industry categories for the policy document. And when querying or searching for policies, it also has good industry differentiation and classification categories, and can quickly retrieve according to industry categories to meet the needs of querying policies. This can help users save time when querying policies, thereby improving the quality of policy search products and improving user satisfaction. At the same time, the proposed policy document classification model and method have good scalability, and can complete the training of classification models and industry mapping for different corpora.

[0167] In one embodiment, the present solution embodiment also provides a document processing process, which can be referred to in Figure 2c .exist Figure 2cIn one aspect, the server can perform text preprocessing on the policy document. On the other hand, the server can also perform industry standard construction after text preprocessing. On the other hand, the server can also perform text classification after industry standard construction to obtain the industry category of the policy document to be predicted. The above text preprocessing process can be mainly divided into: domain vocabulary accumulation (such as accumulation of domain entities, such as financial domain entities) and text formatting (such as word segmentation / NLP technology). In the text preprocessing process, the accumulated rain vocabulary, that is, the preset dictionary and NLP technology can be used to preprocess each policy document included in the first policy document set to obtain a word set. The above industry standard construction can include industry category determination and entity clustering. The label extraction process can be realized by entity clustering. Among them, when performing entity clustering, the label set of each industry category under multiple industry categories can be extracted from the first policy document set according to the word set and the preset label. Specifically, entity clustering can be performed on the first policy document set according to the word set and the preset label to obtain a label set of each industry category under multiple industry categories. The above-mentioned text classification process can be one of the applications for data processing based on the word frequency characteristics of each label in the label set of each industry category. The text classification process can include industry classification and multi-industry mapping. The industry classification can refer to the process of training a preset model using the word frequency characteristics of the second policy document set and each label in the label set of each industry category to obtain a policy document classification model for industry category prediction of the policy document to be predicted and obtain a prediction result. The multi-industry mapping refers to the process of determining the first industry category and the at least one industry category as the industry category corresponding to the policy document to be predicted when the prediction result includes the probability of each industry category of the policy document to be predicted in multiple industry categories.

[0168] See also Figure 3 , is a schematic diagram of the structure of a document processing device provided in an embodiment of the present application. The device can be applied to a server. Specifically, the device may include:

[0169] The acquisition unit 301 is configured to acquire a first policy document set, where the first policy document set includes at least one policy document.

[0170] The processing unit 302 is configured to pre-process the first policy document set to obtain a word set, where the word set includes a plurality of words.

[0171] The processing unit 302 is further configured to extract a label set for each industry category under the plurality of industry categories from the first policy document set according to the word set and the preset label.

[0172] The acquisition unit 301 is further used to acquire a second policy document set, where the second policy document set includes at least one policy document corresponding to each industry category.

[0173] The processing unit 302 is further configured to obtain the word frequency characteristics of each tag in the tag set of each industry category according to the second policy document set, and perform data processing according to the word frequency characteristics of each tag in the tag set of each industry category.

[0174] In an optional embodiment, the processing unit 302 performs data processing according to the word frequency characteristics of each label in the label set of each industry category, specifically using the word frequency characteristics of each label in the second policy document set and the label set of each industry category to train a preset model to obtain a policy document classification model for industry category prediction.

[0175] In an optional implementation, the processing unit 302 extracts a label set for each industry category under multiple industry categories from the first policy document set based on the word set and preset labels, specifically determining a plurality of texts corresponding to the label group of each industry category under the multiple industry categories included in the preset labels in the first policy document set; determining a text set for clustering corresponding to the industry category from the multiple texts based on the word set; clustering the text set of each industry category using a topic model to obtain a clustering result, the clustering result including the word distribution under each topic in the multiple topics; and constructing a label set for each industry category based on the word distribution under each topic in the multiple topics.

[0176] In an optional embodiment, the processing unit 302 determines a text set for clustering corresponding to the industry category from the multiple texts based on the word set, specifically by performing vector representation on each of the multiple words included in the word set to obtain a word vector set, wherein the word vector set includes the word vector of each word; performing vector representation on each of the multiple texts using the word vector set to obtain a text vector of each text; using the text vector of each text, calculating the similarity between the multiple texts; based on the similarity between the multiple texts, determining texts whose similarity is greater than or equal to a preset similarity from the multiple texts; and using the texts whose similarity is greater than or equal to the preset similarity as the text set for clustering corresponding to the industry category.

[0177] In an optional implementation, the processing unit 302 uses a topic model to cluster the text set corresponding to each industry category to obtain a clustering result, specifically determining a sub-word set corresponding to the text set of each industry category from the word set; and uses a topic model to cluster the sub-word set corresponding to the text set of each industry category to obtain a clustering result.

[0178] In an optional implementation, the acquisition unit 301 is further configured to acquire a policy document to be predicted.

[0179] In an optional embodiment, the processing unit 302 is also used to use the policy document to be predicted as input data of the policy document classification model, and the policy document prediction model predicts the industry category of the policy document to be predicted to obtain a prediction result, wherein the prediction result indicates the industry category corresponding to the policy document to be predicted.

[0180] In an optional implementation, the prediction result includes the probability of the policy document to be predicted in each of the multiple industry categories.

[0181] In an optional implementation, the acquisition unit 301 is further used to acquire a threshold value for successful prediction of the industry category.

[0182] In an optional embodiment, the processing unit 302 is also used to calculate the difference between the probability of the policy document to be predicted in the first industry category and the probabilities of other industry categories among the multiple industry categories except the first industry category; the first industry category refers to the industry category with the highest probability among the multiple industry categories; at least one industry category whose difference is less than the threshold is selected from the other industry categories; and the first industry category and the at least one industry category are determined as the industry categories corresponding to the policy document to be predicted.

[0183] In an optional embodiment, the processing unit 302 is also used to calculate the difference between the maximum probability value of the correct industry category of each policy document in the first number of policy documents and the maximum probability value of the incorrect industry category of the policy document before the acquisition unit 301 acquires the threshold value for successful industry category prediction, when a first number of policy documents included in the second policy document set are predicted to have a correct industry category; perform mean calculation on the difference corresponding to each policy document, and obtain the mean of the differences as the threshold value for successful industry category prediction.

[0184] visible, Figure 3In the illustrated embodiment, the text processing device can obtain a first policy document set, and preprocess the first policy document set to obtain a word set, and extract a label set of each industry category under multiple industry categories from the first policy document set according to the word set and preset labels; the text processing device can obtain a second policy document set, and according to the second policy document set, count the word frequency features of each label in the label set of each industry category to perform data processing. The effectiveness of the word frequency features of the obtained policy documents is improved, and the reliability of using the word frequency features of the policy documents for data processing is improved.

[0185] See also Figure 4 , is a schematic diagram of a server structure provided in an embodiment of the present application. The server described in this embodiment may include: a processor 1000, an input device 2000, an output device 3000 and a memory 4000. The processor 1000, the input device 2000, the output device 3000 and the memory 4000 may communicate with each other through the following method: Figure 4 The input device 2000 and / or the output device 3000 are connected by a bus or other means as shown. In one embodiment, the input device 2000 and / or the output device 3000 are optional devices. In one embodiment, the input device 2000 and the output device 3000 may also include a standard wired or wireless communication interface.

[0186] The processor 1000 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0187] The memory 4000 may be a high-speed RAM memory or a non-volatile memory, such as a disk memory. The memory 4000 is used to store a set of program codes, and the processor 1000, the input device 2000, and the output device 3000 may call the program codes stored in the memory 4000. Specifically:

[0188] Processor 1000 is used to obtain a first policy document set, the first policy document set includes at least one policy document; preprocess the first policy document set to obtain a word set, the word set includes multiple words; extract a label set for each industry category under multiple industry categories from the first policy document set based on the word set and preset labels; obtain a second policy document set, the second policy document set includes at least one policy document corresponding to each industry category; based on the second policy document set, obtain the word frequency characteristics of each label in the label set of each industry category, and perform data processing based on the word frequency characteristics of each label in the label set of each industry category.

[0189] Optionally, the processor 1000 performs data processing according to the word frequency characteristics of each label in the label set of each industry category, specifically using the second policy document set and the word frequency characteristics of each label in the label set of each industry category to train a preset model to obtain a policy document classification model for industry category prediction.

[0190] Optionally, the processor 1000 extracts a label set for each industry category under multiple industry categories from the first policy document set based on the word set and the preset label, specifically determining a plurality of texts corresponding to the label group of the industry category in the first policy document set based on the label group of each industry category under the multiple industry categories included in the preset label; determining a text set for clustering corresponding to the industry category from the multiple texts based on the word set; clustering the text set of each industry category using a topic model to obtain a clustering result, the clustering result including the word distribution under each topic in the multiple topics; and constructing a label set for each industry category based on the word distribution under each topic in the multiple topics.

[0191] Optionally, the processor 1000 determines the text set for clustering corresponding to the industry category from the multiple texts according to the word set, specifically by performing vector representation on each of the multiple words included in the word set to obtain a word vector set, the word vector set including the word vector of each word; performing vector representation on each of the multiple texts using the word vector set to obtain a text vector of each text; calculating the similarity between the multiple texts using the text vector of each text; determining texts with a similarity greater than or equal to a preset similarity from the multiple texts according to the similarity between the multiple texts; and using the texts with a similarity greater than or equal to the preset similarity as the text set for clustering corresponding to the industry category.

[0192] Optionally, the processor 1000 uses a topic model to cluster the text set corresponding to each industry category to obtain a clustering result, specifically determining a sub-word set corresponding to the text set of each industry category from the word set; and uses a topic model to cluster the sub-word set corresponding to the text set of each industry category to obtain a clustering result.

[0193] Optionally, processor 1000 is also used to obtain a policy document to be predicted; use the policy document to be predicted as input data of the policy document classification model, and use the policy document prediction model to predict the industry category of the policy document to be predicted to obtain a prediction result, wherein the prediction result indicates the industry category corresponding to the policy document to be predicted.

[0194] Optionally, the prediction result includes the probability of the policy document to be predicted in each industry category among the multiple industry categories.

[0195] Optionally, processor 1000 is also used to obtain a threshold for successful industry category prediction; calculate the difference between the probability of the policy document to be predicted in the first industry category and the probability of other industry categories other than the first industry category among the multiple industry categories; the first industry category refers to the industry category with the highest probability among the multiple industry categories; select at least one industry category from the other industry categories whose difference is less than the threshold; and determine the first industry category and the at least one industry category as the industry categories corresponding to the policy document to be predicted.

[0196] Optionally, processor 1000 is also used to calculate the difference between the maximum probability value of the correct industry category of each policy document in the first number of policy documents and the maximum probability value of the incorrect industry category of the policy document before obtaining the threshold for successful industry category prediction, when a first number of policy documents included in the second policy document set are predicted to have a correct industry category; perform mean calculation on the difference corresponding to each policy document, and obtain the mean of the differences as the threshold for successful industry category prediction.

[0197] In a specific implementation, the processor 1000, the input device 2000, and the output device 3000 described in the embodiments of the present application may execute Figure 1a and Figure 2a The implementation method described in the embodiment can also execute the implementation method described in the embodiment of the present application, which will not be repeated here.

[0198] Each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of sampling hardware or in the form of sampling software functional unit.

[0199] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The medium is a computer-readable storage medium, and the computer-readable storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0200] What is disclosed above is only a preferred embodiment of the present application, and it certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiment and equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A document processing method, characterized in that: include: Acquire a first policy document set, wherein the first policy document set includes at least one policy document; Preprocessing the first policy document set to obtain a word set, wherein the word set includes a plurality of words; According to the label group of each industry category under the multiple industry categories included in the preset label, determine a plurality of texts corresponding to the label group of the industry category in the first policy document set; Performing vector representation on each of the multiple words included in the word set to obtain a word vector set, wherein the word vector set includes the word vector of each word; Using the word vector set to perform vector representation on each of the multiple texts to obtain a text vector for each text; Calculating the similarity between the multiple texts using the text vector of each text; According to the similarities between the multiple texts, determining texts whose similarities are greater than or equal to a preset similarity from the multiple texts; The texts with similarity greater than or equal to the preset similarity are used as the text set for clustering corresponding to the industry category; Clustering the text set of each industry category using a topic model to obtain a clustering result, wherein the clustering result includes a word distribution under each topic in a plurality of topics; Constructing a tag set for each industry category according to the word distribution under each of the multiple topics; Acquire a second policy document set, where the second policy document set includes at least one policy document corresponding to each industry category; According to the second policy document set, the word frequency characteristics of each label in the label set of each industry category are obtained, and data processing is performed according to the word frequency characteristics of each label in the label set of each industry category; the data processing includes generating a tag cloud including each label in the label set according to the word frequency characteristics of each label in the label set of each industry category; the word frequency characteristics of each label in the label set are also used to train a policy document classification model, and the policy document classification model is used to perform industry category prediction for the policy document to be predicted, and the industry category prediction includes: obtaining a threshold value for successful industry category prediction; calculating the probability of the policy document to be predicted in the first industry category and the probability of other industry categories in the multiple industry categories except the first industry category. Difference, the first industry category refers to the industry category with the highest probability among the multiple industry categories; at least one industry category with a difference less than the threshold is selected from the other industry categories; the first industry category and the at least one industry category are determined as the industry categories corresponding to the policy document to be predicted; wherein the threshold for successful prediction of the industry category is determined as follows: when a first number of policy documents included in the second policy document set are predicted to have a correct industry category, the difference between the maximum probability value of the correct industry category of each policy document in the first number of policy documents and the maximum probability value of the probability value of the incorrect industry category of the policy document is calculated; the mean of the difference corresponding to each policy document is calculated to obtain the mean of the difference as the threshold for successful prediction of the industry category.

2. The method according to claim 1, characterized in that The data processing according to the word frequency characteristics of each tag in the tag set of each industry category includes: The preset model is trained using the second policy document set and the word frequency characteristics of each label in the label set of each industry category to obtain a policy document classification model for industry category prediction.

3. The method according to claim 1, characterized in that The text set corresponding to each industry category is clustered using the topic model to obtain a clustering result, including: Determine a subword set corresponding to the text set of each industry category from the word set; The topic model is used to cluster the subword sets corresponding to the text sets of each industry category to obtain clustering results.

4. The method according to claim 2, characterized in that: The method further comprises: Obtain the policy document to be predicted; The policy document to be predicted is used as input data of the policy document classification model, and the policy document classification model predicts the industry category of the policy document to be predicted to obtain a prediction result, which indicates the industry category corresponding to the policy document to be predicted.

5. A document processing device, characterized in that: include: An acquiring unit, configured to acquire a first policy document set, wherein the first policy document set includes at least one policy document; A processing unit, configured to pre-process the first policy document set to obtain a word set, wherein the word set includes a plurality of words; The processing unit is further used to determine, based on the label group of each industry category under the multiple industry categories included in the preset label, multiple texts corresponding to the label group of the industry category in the first policy document set; perform vector representation on each of the multiple words included in the word set to obtain a word vector set, wherein the word vector set includes the word vector of each word; perform vector representation on each of the multiple texts using the word vector set to obtain a text vector of each text; calculate the similarity between the multiple texts using the text vector of each text; and determine, based on the similarity between the multiple texts, texts whose similarity is greater than or equal to the preset similarity from the multiple texts; The texts with similarities greater than or equal to the preset similarity are used as the text set for clustering corresponding to the industry category; the text set of each industry category is clustered using a topic model to obtain a clustering result, wherein the clustering result includes the word distribution under each of the multiple topics; and a label set of each industry category is constructed according to the word distribution under each of the multiple topics; The acquisition unit is further configured to acquire a second policy document set, wherein the second policy document set includes at least one policy document corresponding to each industry category; The processing unit is also used to obtain the word frequency characteristics of each label in the label set of each industry category according to the second policy document set, and perform data processing according to the word frequency characteristics of each label in the label set of each industry category; the data processing includes generating a tag cloud including each label in the label set according to the word frequency characteristics of each label in the label set of each industry category; the word frequency characteristics of each label in the label set are also used to train a policy document classification model, and the policy document classification model is used to perform industry category prediction on the policy document to be predicted, and the industry category prediction includes: obtaining a threshold value for successful industry category prediction; calculating the probability of the policy document to be predicted in the first industry category and the probability of the policy document to be predicted in other industry categories except the first industry category among the multiple industry categories The difference between probabilities, the first industry category refers to the industry category with the highest probability among the multiple industry categories; at least one industry category whose difference is less than the threshold is selected from the other industry categories; the first industry category and the at least one industry category are determined as the industry categories corresponding to the policy document to be predicted; wherein the threshold for successful industry category prediction is determined as follows: when a first number of policy documents included in the second policy document set are predicted to have a correct industry category, the difference between the maximum probability value of the correct industry category of each policy document in the first number of policy documents and the maximum probability value of the probability value of the incorrect industry category of the policy document is calculated; the mean of the difference corresponding to each policy document is calculated to obtain the mean of the difference as the threshold for successful industry category prediction.

6. A server, characterized in that: The method comprises a processor and a memory, wherein the processor and the memory are connected to each other, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 4.

Images