Document processing method and device

Abstract

A document processing method and apparatus. The method comprises: determining the frequency x of the qth sentence in the N sentences in the i-th document based on the similarity between the sentence in the i-th document and N sentences in M ​​documents. i,q , where M and N are both integers greater than 1, q is a positive integer not greater than N, and i is a positive integer; based on the frequency x corresponding to the qth sentence in the i-th document i,q , determine the document representation of the i-th document. This can improve the efficiency of document representation.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and more specifically, to a method and apparatus for document processing. Background Art

[0002] Natural language processing focuses on representing natural language as data that computers can process. For example, a document written in natural language can be represented as a vector. This allows computers to process the information in the document and provide intelligent feedback based on the results.

[0003] Natural language processing can be applied in a variety of scenarios, such as text classification, search engines, recommendation systems, diagnostic systems, and fault handling systems. For example, fault handling systems can use historical fault description documents to provide solutions to newly occurring faults and prevent future failures. For example, this can involve processing the text within historical fault description documents. In these applications, document representation is a key component, determining the effectiveness of subsequent post-processing.

[0004] However, document representation often requires processing a large amount of data, which affects its efficiency. Therefore, how to improve the efficiency of document representation has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The present application provides a document processing method and apparatus, which can improve the efficiency of document representation.

[0006] In a first aspect, a document processing method is provided, comprising: determining, based on the similarity between a sentence in an i-th document and N sentences in M ​​documents, a frequency x corresponding to the q-th sentence in the N sentences in the i-th document; i,q , where M and N are both integers greater than 1, q is a positive integer not greater than N, and i is a positive integer; based on the frequency x corresponding to the qth sentence in the i-th document i,q , determine the document representation of the i-th document.

[0007] The technical solution of the embodiment of the present application determines the corresponding frequency of sentences in a document based on the similarity between sentences, and determines the document representation of the document based on the corresponding frequency of the sentences in the document. This can reduce the dimension of the data and the resource requirements required for processing, thereby improving the efficiency of document representation.

[0008] In some possible implementations, when i is not greater than M, the i-th document is a document among the M documents.

[0009] In some possible implementations, when i is greater than M, the i-th document is a document outside the M documents.

[0010] In some possible implementations, determining the document representation of the i-th document includes: determining the inverse document frequency (IDF) corresponding to the q-th sentence based on the frequency of the q-th sentence in the M documents. q ; According to the frequency x corresponding to the qth sentence in the i-th document i,q and the inverse document frequency idf corresponding to the qth sentence q , determine the document representation of the i-th document.

[0011] In some possible implementations, the inverse document frequency (IDF) corresponding to the qth sentence is q It is negatively correlated with the number of documents containing the qth sentence in the M documents.

[0012] In some possible implementations, if the frequency of the qth sentence in the pth document among the M documents is greater than zero, then the pth document contains the qth sentence, and p is a positive integer not greater than M.

[0013] In some possible implementations, the frequency x corresponding to the qth sentence in the i-th document is i,q is the sum of the similarities between the qth sentence and each sentence in the i-th document.

[0014] In some possible implementations, the determining of the frequency x corresponding to the qth sentence in the N sentences in the i-th document is performed. i,q , including: determining the frequency x corresponding to the qth sentence in the i-th document based on the K most similar sentences in the N sentences of the sentence in the i-th document i,q , where K is a positive integer less than N.

[0015] Using the most similar K sentences can reduce the amount of calculation and ensure the efficiency of document representation.

[0016] In some possible implementations, K is positively correlated with N.

[0017] In some possible implementations, the frequency x corresponding to the qth sentence in the i-th document is i,qis the sum of the K nearest neighbor similarities between the qth sentence and each sentence in the i-th document; wherein, if the qth sentence belongs to the K most similar sentences to the j-th sentence in the i-th document, then the K nearest neighbor similarity between the qth sentence and the j-th sentence in the i-th document is the similarity between the qth sentence and the j-th sentence in the i-th document; if the qth sentence does not belong to the K most similar sentences to the j-th sentence in the i-th document, then the K nearest neighbor similarity between the qth sentence and the j-th sentence in the i-th document is zero, and j is not greater than n i A positive integer, n i is the number of sentences in the i-th document.

[0018] In some possible implementations, the frequency x corresponding to the qth sentence in the i-th document is i,q is an N-dimensional vector x i The qth element of , where Among them, for vector N dimension If the lth sentence among the N sentences belongs to The K most similar sentences to the sentence, then The lth element is the lth sentence and the lth element The similarity of the sentences, otherwise, The lth element of is zero, wherein the lth element The sentence is the jth sentence of the i-th document, n i is the number of sentences in the i-th document, j is no more than n i is a positive integer, and l is a positive integer not greater than N.

[0019] For the i-th document n i The sentences are summed up, so that n i The similarity information of the sentences is combined to reflect the structural information of the document.

[0020] In some possible implementations, the determining of the inverse document frequency (IDF) corresponding to the qth sentence is performed. q , including: determining the inverse document frequency (idf) corresponding to the qth sentence according to the following formula q , Here, |*| represents the cardinality of the set.

[0021] Using the above-mentioned inverse document frequency, whether the sentence appears in the document is indicated by whether the corresponding frequency of the sentence in the document is greater than zero. Moreover, the more documents a sentence appears in, the smaller the inverse document frequency is, and the fewer documents a sentence appears in, the larger the inverse document frequency is. This can reflect the distinctiveness of the sentence, thereby improving the effect of document representation.

[0022] In some possible implementations, determining the document representation of the i-th document includes: determining the document representation z of the i-th document according to the following formula: i , Among them, ||*|| represents the 2 norm, the N-dimensional vector y i The qth element y i,q For, y i,q =x i,q *idf q .

[0023] The product of the frequency of the sentence in the document obtained based on similarity and the inverse document frequency of the sentence is used as the weight of the sentence in the document. The document representation obtained based on this can reflect the structural information of the document and the structural information between documents, thereby effectively representing the document.

[0024] In some possible implementations, the method further includes: obtaining a sentence representation of a sentence according to a first model; determining the similarity between sentences according to the sentence representation of the sentence; wherein the first model is obtained based on multiple word embedding models, the first model connects multiple vectors to obtain the sentence representation of the sentence, and the multiple vectors are vectors of words of the sentence obtained according to the multiple word embedding models.

[0025] Different embedding models can bring compensatory information, and different P values ​​can also introduce rich information. Therefore, the above model can improve the representation effect.

[0026] In some possible implementations, the document is a Chinese document, and the method further includes: performing word segmentation on the sentence to obtain words of the sentence.

[0027] In some possible implementations, the word segmentation processing of the sentence includes: obtaining an initial word sequence of the sentence; performing reverse maximum matching on the initial word sequence according to a general dictionary and a professional dictionary of the field to which the document belongs, to obtain general words in the sentence and words specific to the field.

[0028] After the above reverse maximum matching, word segmentations that are consistent with the understanding of general and professional fields can be obtained, thereby improving the quality of word segmentation.

[0029] In some possible implementations, the method further includes: performing document processing according to the document representation.

[0030] In some possible implementations, the i-th document is a fault description document corresponding to an unresolved fault; and the document processing based on the document representation includes: determining, based on the document representation, a reference document among the M documents that has the highest similarity to the i-th document, wherein the reference document is a fault description document corresponding to a resolved fault, and the solution to the fault corresponding to the reference document is used to process the fault corresponding to the i-th document.

[0031] By adopting the technical solution of the embodiment of the present application for troubleshooting, a solution can be automatically obtained based on historical documents effectively without manual processing, thereby saving time and labor costs and improving processing efficiency.

[0032] In some possible implementations, the similarity is cosine similarity.

[0033] In a second aspect, a document processing apparatus is provided, comprising a module for executing the method in the first aspect or any possible implementation thereof.

[0034] In a third aspect, a document processing device is provided, comprising: a memory for storing a program; a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is used to execute the above-mentioned document processing method.

[0035] In a fourth aspect, the present application also provides a computer-readable storage medium storing program code for execution by a device, wherein the program code includes instructions for executing the steps in the above-mentioned document processing method.

[0036] In a fifth aspect, the present application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned document processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of a system architecture of an embodiment of the present application.

[0038] Figure 2 It is a schematic diagram of the document processing process of an embodiment of the present application.

[0039] Figure 3 It is a schematic flow chart of the document processing method of an embodiment of the present application.

[0040] Figure 4It is a schematic block diagram of a document processing device according to an embodiment of the present application.

[0041] Figure 5 It is a schematic block diagram of a document processing device according to another embodiment of the present application.

[0042] Figure 6 It is a structural diagram of a document processing device according to an embodiment of the present application.

[0043] List of reference numerals:

[0044] 110, processing device;

[0045] 111, processing module;

[0046] 112, communication interface;

[0047] 113, storage system;

[0048] 114, preprocessing module;

[0049] 115, model;

[0050] 120, data generating device;

[0051] 130, training equipment;

[0052] 140, database;

[0053] 150, corpus;

[0054] 201, Documentation;

[0055] 202, word segmentation tool;

[0056] 203, General Dictionary;

[0057] 204, initial word sequence;

[0058] 205, professional dictionary;

[0059] 206, reverse maximum matching;

[0060] 207, words;

[0061] 208, sentence embedding model;

[0062] 209, sentence vector;

[0063] 210, document representation;

[0064] 211, document representation-based processing;

[0065] 310, based on the similarity between the sentence in the i-th document and the N sentences in the M documents, determine the frequency x of the q-th sentence in the N sentences in the i-th document. i,q , where M and N are both integers greater than 1, q is a positive integer not greater than N, and i is a positive integer;

[0066] 320, according to the frequency x corresponding to the qth sentence in the i-th document i,q , determine the document representation of the i-th document;

[0067] 400, a device for document processing;

[0068] 410, acquiring unit;

[0069] 420, document representation unit;

[0070] 430, sentence representation unit;

[0071] 440, participle unit;

[0072] 450, processing unit;

[0073] 600, a device for document processing;

[0074] 601, memory;

[0075] 602, processor;

[0076] 603, communication interface;

[0077] 604, bus. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present application are described below in conjunction with the accompanying drawings. It should be understood that the specific examples in this specification are only intended to help those skilled in the art better understand the embodiments of the present application, and are not intended to limit the scope of the embodiments of the present application.

[0079] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0080] It should also be understood that the various implementation methods described in this specification can be implemented individually or in combination, and the embodiments of the present application are not limited to this.

[0081] Unless otherwise specified, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art in the art of the present application. The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit the scope of this application.

[0082] Figure 1 It is a schematic diagram of a system architecture of an embodiment of the present application.

[0083] exist Figure 1 In the illustrated system architecture, data generation device 120 is a device that generates data to be processed, for example, a document to be processed. There can be multiple data generation devices 120. Documents generated by data generation device 120 can be directly transmitted to processing device 110 or first stored in database 140 and then retrieved by processing device 110. For example, a fault description document generated by data generation device 120 can be transmitted to processing device 110 in real time; processing device 110 then retrieves historical documents from database 140.

[0084] The processing device 110 is in communication with the data generating device 120. Specifically, the processing device 110 may include a communication interface 112 to enable communication with other devices. The communication connection may be wired or wireless.

[0085] The processing device 110 can be an electronic device or system with data processing capabilities, such as a computer. The processing device 110 can include a processing module 111 for implementing data processing, for example, document representation using the technical solutions of the embodiments of this application. The processing module 111 can specifically be one or more processors. The processor can be any type of processor and is not limited in this embodiment of the application.

[0086] The processing device 110 may further include a storage system 113. The storage system 113 may be used to store data and instructions, such as computer-executable instructions for implementing the technical solutions of the embodiments of the present application. The processing device 110 may access data, instructions, etc. from the storage system 113, or store data, instructions, etc. in the storage system 113. The storage system 113 may specifically be one or more memories. The memories may be of any type, and the embodiments of the present application are not limited thereto.

[0087] The storage system 113 may be disposed within the processing device 110 or outside the processing device 110. In the case where the storage system 113 is disposed outside the processing device 110, the processing device 110 may access the storage system 113 through a data interface.

[0088] The processing device 110 may also include other general devices, such as an output device for outputting processed data to a user.

[0089] In some possible implementations, the processing module 111 may include a pre-processing module 114 for pre-processing the acquired data, such as performing word segmentation on the document.

[0090] In some possible implementations, the processing device 110 may be configured with a trained model 115. In this case, the processing module 111 may use the model 115 to perform corresponding processing.

[0091] For example, the model 115 may be a sentence embedding model for sentence representation. The training device 130 may train the sentence embedding model based on the training data in the corpus 150. In this way, the processing module 111 may use the sentence embedding model to obtain a sentence representation.

[0092] In some possible implementations, the input document can first be segmented by the preprocessing module 114 to obtain the words of the sentence; then it can be input into the model 115 to obtain the representation of the sentence; and then the document can be represented using the technical solution of the embodiment of the present application described below.

[0093] In some possible implementations, the processing device 110 can also be a data generating device. In this case, the processing device 110 can generate a document to be processed and process the document.

[0094] In some possible implementations, the processing device 110 can also be a training device. In this case, the processing device 110 can first train the model 115 and then use the model 115, or can simultaneously train the model 115 during use.

[0095] It should be understood that Figure 1 It is only a schematic diagram of a system architecture provided in an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation.

[0096] In some possible implementations, the model 115 trained by the training device 130 can be a model obtained based on deep learning, for example, it can be a model built based on a neural network, where the neural network can be a convolutional neural network (CNN), a recurrent neural network (RNN), a deep convolutional neural network (DCNN), and the like.

[0097] The following combination Figure 2 , briefly introduces the main process of the document processing method of the embodiment of the present application.

[0098] Figure 2 A schematic diagram showing the document processing process of an embodiment of the present application is shown.

[0099] exist Figure 2 In the example, for the document 201 to be processed, the word segmentation tool 202 can be used to segment the words to obtain the initial word sequence 203; and then the processed words 207 can be obtained through the inverse maximum matching 206. The above process can be called a preprocessing process. Since there are many special words in the documents in the industrial field, the general dictionary obtained from the external data source and the professional dictionary of the document field configured on the client side can be used for inverse maximum matching. The preprocessed words 207 can obtain sentence vectors 209 (sentence representations) through the sentence embedding model 208. Optionally, the sentence embedding model 208 can be a pre-trained model. The sentence embedding model will be described in detail below.

[0100] Based on the sentence vector 209, a document representation can be performed to obtain a document representation 210. The detailed scheme of document representation will be described below. Finally, based on the document representation 210, subsequent processing 211 can be performed, such as document analysis, comparison, classification, training a classifier, etc.

[0101] A commonly used method for document representation is the Term Frequency-Inverse Document Frequency (TFIDF) method. TFIDF is used to assess the importance of a term to a document within a document set or corpus. A term's importance increases proportionally with its occurrence in a document, but decreases inversely with its frequency in the corpus. If a term has a high frequency in one document and rarely appears in other documents, it is considered to have good class distinction ability and is suitable for classification. TFIDF is actually: TF*IDF, where TF stands for Term Frequency and IDF stands for Inverse Document Frequency. TF represents the frequency of a term within a document. The key idea behind IDF is that the fewer documents containing a term, the greater its IDF, indicating that the term has good class distinction ability. Specifically, the IDF of a term can be calculated by dividing the total number of documents by the number of documents containing the term and taking the base-10 logarithm of the quotient.

[0102] The TFIDF method uses the TFIDF values ​​of words to form a document representation (vector). Therefore, the dimensionality of the vector in the TFIDF method is determined by the number of words. However, due to the large number of words in a document, this method is prone to the curse of dimensionality, which affects the efficiency of document representation.

[0103] In light of this, the present embodiment provides an improved technical solution that obtains the "TFIDF" of a sentence based on sentence similarity. Different from the TFIDF of the aforementioned words, the technical solution of the present embodiment can be referred to as a pseudo-TFIDF document representation method. The technical solution of the present embodiment is described in detail below.

[0104] Figure 3 A schematic flow chart of a document processing method according to an embodiment of the present application is shown. Figure 1 The processing device 110 in is executed.

[0105] 310, based on the similarity between the sentence in the i-th document and the N sentences in the M documents, determine the frequency x of the q-th sentence in the N sentences in the i-th document. i,q , where M and N are both integers greater than 1, q is a positive integer not greater than N, and i is a positive integer.

[0106] In one embodiment, when i is not greater than M, the i-th document is a document among the M documents.

[0107] In another embodiment, when i is greater than M, the i-th document is a document outside the M documents.

[0108] When performing document characterization, multiple documents can be characterized together during the first characterization; when new documents appear subsequently, the new documents can be characterized based on the already characterized documents (for example, when there are fewer new documents), or the new documents and the already characterized documents can be re-characterized together (for example, when there are more new documents).

[0109] Accordingly, the technical solutions of the embodiments of the present application include technical solutions for two cases. One case is to characterize M documents together, that is, when i is not greater than M, the i-th document among the M documents is characterized; the other case is to characterize new documents based on the M documents that have been characterized, that is, when i is greater than M, documents outside the M documents are characterized.

[0110] The documents mentioned above can be a category of documents in a specific field. For example, for an industrial fault handling system, this could be a fault description document for a specific type of equipment. This fault description document can include a description of the fault event and, for resolved faults, the cause, solution, or classification. When representing the documents, focus on the common components of all documents, such as the fault event description.

[0111] For example, the i-th fault description document can be represented as D i , or, (D i ,oi ), where D i The event description part indicating the fault, o i Indicates the cause, solution, or classification of the fault. For unresolved faults, the document contains the former; for resolved faults, the document contains both of the above. When characterizing the document, you can only focus on D i In subsequent applications, you can use D i Get the corresponding o i .

[0112] It should be understood that the technical solutions of the embodiments of the present application can be applied to various documents and are not limited to the above examples. In addition, the embodiments of the present application do not limit the language type of the document. For example, it can be Chinese or non-Chinese. In some optional embodiments, corresponding technical solutions are further provided for Chinese documents.

[0113] In the embodiments of the present application, document representation is performed based on the frequency of sentences in the document. The frequency of sentences in the document is obtained by the similarity between sentences. In other words, the frequency is not obtained by the actual number of times a sentence appears in the document, but is a weighted value obtained by the similarity between sentences.

[0114] Optionally, the frequency x corresponding to the qth sentence in the i-th document i,q is the sum of the similarities between the qth sentence and each sentence in the i-th document. In other words, the frequency of a sentence in a document is the sum of the similarities between the sentence and all the sentences in the document.

[0115] Optionally, the similarity is cosine similarity, but this embodiment of the present application is not limited to this.

[0116] Cosine similarity, also known as cosine similarity, evaluates the similarity between two vectors by calculating the cosine of the angle between them. For a document, each sentence can be represented as a multidimensional vector. The cosine similarity between two sentences is the cosine of the angle between the two vectors representing the two sentences.

[0117] Optionally, in an embodiment of the present application, to reduce the amount of computation, a K-Nearest Neighbor (KNN) algorithm may be used. That is, the frequency of a sentence in a document may be determined based on the K most similar sentences in the sentence space (the N sentences mentioned above), where K is a positive integer less than N.

[0118] The K most similar sentences to a sentence are the K nearest neighbors of the sentence in the sentence space, that is, the K sentences with the highest similarity to the sentence.

[0119] In the embodiment of the present application, the sentence space consists of N sentences in M ​​documents. When characterizing M documents as a whole, the sentence of the i-th document is a sentence in the N sentences, and the frequency x is determined by using the K most similar sentences in the N sentences. i,q When characterizing a new document outside of the M documents, the sentence of the i-th document is not one of the N sentences, but the frequency x is still determined by using the most similar K sentences among the N sentences. i,q .

[0120] Optionally, the value of K can be predetermined based on the total number of sentences and can be continuously adjusted. For example, K can be positively correlated with N, that is, when N is large, K can take a larger value. For another example, if there are already sentences with relatively low similarity among the K sentences, the value of K can be reduced, and vice versa, the value of K can be increased. As an example, the value range of K can be 5-30, but this is not limited to this in the embodiments of the present application.

[0121] The similarity between a sentence and every sentence in a document can be set using the KNN algorithm. Specifically, if the sentence is one of the K most similar sentences to a sentence in the document, the similarity can be the similarity of the two sentences; otherwise, it can be set to zero.

[0122] For ease of description, the similarity after using the KNN algorithm is expressed as K nearest neighbor similarity. Accordingly, if the qth sentence belongs to the K most similar sentences to the jth sentence of the i-th document, then the K nearest neighbor similarity between the qth sentence and the jth sentence of the i-th document is the similarity between the qth sentence and the jth sentence of the i-th document; if the qth sentence does not belong to the K most similar sentences to the jth sentence of the i-th document, then the K nearest neighbor similarity between the qth sentence and the jth sentence of the i-th document is zero, and j is not greater than n. i A positive integer, n i is the number of sentences in the i-th document.

[0123] In this case, the frequency x of the qth sentence in the i-th document is i,q is the sum of the K nearest neighbor similarities between the qth sentence and each sentence in the i-th document.

[0124] Optionally, the frequency of a sentence in a document may be determined in the following manner.

[0125] For the jth sentence in the i-th document, it is represented as Sentences, get The K most similar sentences to the sentence;

[0126] Introducing N-dimensional vectors Among them, if the lth sentence in the N sentences belongs to the The K most similar sentences to the sentence, then The lth element is the lth sentence and the lth element The similarity of sentences (or the lth element is 1), otherwise, The lth element of is zero;

[0127] For the i-th document,

[0128]

[0129] N-dimensional vector x i The qth element x i,q , is the frequency of the qth sentence in the N sentences in the i-th document. i is the number of sentences in the i-th document, j is no more than n i is a positive integer, and l is a positive integer not greater than N.

[0130] In formula 1, the n of the i-th document i The sentences are summed up, so that n i The similarity information of the sentences is combined to reflect the structural information of the document.

[0131] 320, according to the frequency x corresponding to the qth sentence in the i-th document i,q , determine the document representation of the i-th document.

[0132] By the method in 310, the frequency of the qth sentence in each of the M documents can be obtained. In this way, the inverse document frequency (idf) corresponding to the qth sentence can be further determined based on the frequency of the qth sentence in the M documents. q .

[0133] That is to say, after obtaining the frequency of the sentence in the document based on the similarity, the inverse document frequency of the sentence can be obtained next.

[0134] The inverse document frequency (idf) corresponding to the qth sentence q It can be negatively correlated with the number of documents containing the qth sentence in the M documents. That is, the greater the number of documents containing the qth sentence in the M documents, the higher the inverse document frequency (idf) corresponding to the qth sentence. q The smaller the number of documents containing the qth sentence in the M documents, the smaller the inverse document frequency (idf) corresponding to the qth sentence. q The larger the value, the better. In this way, using the inverse document frequency idfq It can reflect the distinctiveness of the sentence.

[0135] In one embodiment, if the frequency of the qth sentence in the pth document among the M documents is greater than zero, then the pth document contains the qth sentence, and p is a positive integer not greater than M. In other words, whether a document contains a sentence can be determined by whether the frequency of the sentence in the document is greater than zero.

[0136] Optionally, based on formula (1), the inverse document frequency (idf) corresponding to the qth sentence can be determined according to the following formula: q ,

[0137]

[0138] Here, |*| represents the cardinality of the set.

[0139] In formula (2), x i,q > 0, indicating that the qth sentence has appeared in i documents. Correspondingly, formula (2) shows that if the qth sentence has appeared in more documents, the inverse document frequency (idf) q The smaller the value, the smaller the inverse document frequency (IDF). q Therefore, the inverse document frequency (idf) q It can reflect the distinctiveness of the sentence.

[0140] Get the frequency x of the qth sentence in the i-th document i,q and the inverse document frequency idf corresponding to the qth sentence q Then, we can calculate the frequency x of the qth sentence in the i-th document. i,q and the inverse document frequency idf corresponding to the qth sentence q , determine the document representation of the i-th document.

[0141] As can be seen from the above, the greater the inverse document frequency of a sentence, the greater the discriminability of the sentence. Therefore, the inverse document frequency of a sentence can be combined with the frequency of the sentence in the document to represent the weight of the sentence in the document, thereby obtaining a representation of the document.

[0142] Optionally, the product of the frequency of the sentence in the document and the inverse document frequency of the sentence can be used as the weight of the sentence in the document.

[0143] For example, based on formulas (1) and (2), the weight of the qth sentence in the i-th document can be,

[0144] y i,q =x i,q*idf q (3)

[0145] y i,q As the N-dimensional vector y corresponding to N sentences i The qth element of y. i Normalization can be performed to obtain the document representation z of the i-th document i ,

[0146]

[0147] Here, ||*|| represents the 2-norm.

[0148] Through the above method, an N-dimensional vector representation of the i-th document can be obtained. In other words, the dimension of the document representation obtained by the technical solution of the embodiment of the present application is the number of sentences. Compared with the representation method based on the number of words, the data dimension of the technical solution of the embodiment of the present application is reduced, thereby reducing the amount of data calculation and improving the efficiency of document representation.

[0149] As an example, when characterizing M documents at a time (i.e., i is not greater than M), the document characterization may adopt the following algorithm:

[0150] Algorithm: Pseudo TFIDF Document Representation

[0151] Input:

[0152] sentence vectors, or if o i exist(document Or, if o i exist, sentence vectors);

[0153] an integer K of KNN;

[0154] Output:

[0155] representation feature matrix z for all documents (representation feature matrix z for all documents);

[0156] Comment:

[0157] D i :the ith document (the i-th document);

[0158] :the jth sentence for D i (D i The jth sentence of

[0159] M: number of documents; (number of documents);

[0160] n i :number of sentences in ith document (the number of sentences in the ith document);

[0161] N: number of sentences

[0162] Process:

[0163] step1(step 1):Sequentially assign an index number index from 1toNfor every sentence (assign indexes to each sentence from 1 to N in sequence );

[0164] step 2: for every do(for each ):

[0165] Find the K nearest neighbors in the sentence space via distance metric, such as cosine similarity (through the distance matrix, for example, cosine similarity, find the K nearest neighbors in the sentence space);

[0166] Involve a vector to represent this sentence,and the lthentry in is cosine similarity if index(s)==l and s is in the K nearestneighbors for (Introducing vector Indicates that the sentence, if the index s == l and s is in Among the K nearest neighbors of The lth element of is the cosine similarity);

[0167] step3: for every D i do(for each D i ):

[0168] The spare representation for D i is(redundant representation is)

[0169] Step 4: Let x i,q represent the qth component in x i (x i,q Represents x i So we can define

[0170]

[0171] Step 5: for every D i do(for each D i )

[0172] update y i by(update y i )y i,q =x i,q *idf q ;

[0173] normalize

[0174] step6(step 6):Based on do the clustering task (based on or train a classifier for classification task using (or use training a classifier for a classification task).

[0175] For example, the fault description document D includes three sentences s1, s2, s3, and the corresponding sentence vectors are v1, v2, v3. Through KNN, the K (for ease of understanding, set to 3) most similar sentences to s1, s2, s3 are {s1, s}4s{, s6}s, {s2,}. Assuming the corresponding similarities are {1, 0.8, 0.75}, {1, 0.90, 0.88}, {1, 0.95, 0.82}, according to formula (1), the pseudo TF representation of document D can be obtained as follows: [1, 1, 1, 0.8, 0, 2.47, 0.95, 0, 0, 0.88..... 0, 0, 0, 0, 0] N According to the above algorithm, pseudo TFIDF document representation can be further obtained.

[0176] The technical solution of the embodiment of the present application performs document representation by obtaining the corresponding frequency of sentences in the document based on similarity and the inverse document frequency corresponding to the sentences. The obtained document representation can reflect the structured information of the document and the structured information between documents, and the document representation is sparse, so it can effectively represent the document.

[0177] In addition, compared with the current TFIDF characterization scheme, the dimension of the pseudo-TFIDF characterization scheme in the embodiment of the present application is greatly reduced. Therefore, a faster document processing speed can be achieved without too many computing and storage resources, thereby improving the efficiency of document processing.

[0178] In addition, compared with document characterization schemes based on deep learning, the document characterization scheme of the embodiment of the present application does not require a large amount of training data, thereby avoiding the problem of lack of training data due to the confidentiality of industrial documents.

[0179] Optionally, for each document, a representation of each sentence, that is, a vector of each sentence, can be obtained by various methods, and then the above method can be used to obtain the representation of the document.

[0180] Optionally, as an embodiment of the present application, a sentence representation of a sentence can be obtained based on the first model (sentence embedding model).

[0181] The sentence embedding model may be a pre-trained model. For example, the sentence embedding model may be pre-trained using a deep learning method.

[0182] Alternatively, instead of training a sentence embedding model, a sentence embedding model based on multiple word embedding models may be directly used.

[0183] Optionally, as an embodiment of the present application, the first model can be obtained based on multiple word embedding models.

[0184] Specifically, the simplest method for finding sentence embeddings is to average the embeddings of all the words in the sentence, but this approach yields poor results. To improve performance, the word embedding averaging operation can be generalized to a type of p-mean operation, where different p values ​​can be used to generate different features.

[0185] To u i The p-mean can be expressed as:

[0186]

[0187] When p = 1, p-mean is the average operation; when p = +∞, it is the maximum (max) operation; when p = -∞, it is the minimum (min) operation. The above three operations (average, maximum, and minimum) can be used together to improve the effect.

[0188] In addition, to further improve the effect, you can use a variety of word embedding models, such as word2vec, fasttext, glove, etc., perform p-mean operation on the embedding (embedding, that is, vector) of each model, and then connect the results.

[0189] For example, the following formula can be used for sentence representation (sentence embedding):

[0190]

[0191] In the above formula, u, v, w represent the word vectors obtained from different word embedding models, s p Represents the sentence vector for p∈{1,±∞}, n represents the number of words in the sentence, and the symbol Connect different s p as a joint sentence vector.

[0192] For example, assuming that the dimension of each word embedding model is 100, according to formula (6), s1, s +∞ , s -∞ The dimensions of are all 300. According to formula (7), a 900-dimensional vector s can be obtained by vector concatenation.

[0193] Based on the above-mentioned multiple word embedding models, a sentence embedding model can be obtained. According to the multiple word embedding models, multiple vectors of the words of the sentence are obtained. The sentence embedding model can connect the multiple vectors to obtain the sentence representation (vector) of the sentence. In this way, when representing the document, the sentence representation of each sentence can be obtained according to the sentence embedding model. Then, based on the sentence representation of each sentence, the similarity between the sentences is determined, and the document representation is obtained using the above-mentioned method.

[0194] Different embedding models can bring compensation information, and different P values ​​can also introduce rich information. Therefore, the above-mentioned model connecting P average can improve the representation effect.

[0195] For some languages, such as Chinese, sentences need to be segmented to obtain their individual words. The quality of this segmentation process is crucial for subsequent processing. Industrial documents often contain specialized terms, making it difficult for conventional segmentation methods to identify these terms, impacting segmentation quality.

[0196] In view of this, in one embodiment of the present application, the following method can be used to perform word segmentation on the sentence:

[0197] Obtain an initial word sequence of the sentence; perform reverse maximum matching on the initial word sequence based on a general dictionary and a specialized dictionary of the field to which the document belongs, to obtain general words in the sentence and words specific to the field.

[0198] Specifically, an initial word sequence for a sentence can be obtained using existing word segmentation tools, such as Jieba. This initial word sequence is then subjected to reverse maximum matching using an automatically established dictionary, which includes a general dictionary obtained from an external data source and a specialized dictionary configured on the client side for the document's domain. This reverse maximum matching yields word segments that align with both general and specialized domain understandings, thereby improving the quality of the segmentation.

[0199] The words of the sentence obtained in the above manner can be used for the aforementioned sentence representation and document representation, thereby improving the efficiency of Chinese document representation.

[0200] Optionally, after obtaining the document representation of the document through the aforementioned method, document processing may be performed based on the document representation of the document, such as document analysis, comparison, classification, classifier training, etc.

[0201] Optionally, in one embodiment of the present application, for the fault description document, document processing related to fault handling can be performed based on the document representation of the document.

[0202] Specifically, when the i-th document is a fault description document corresponding to an unresolved fault, the document representation can be used to determine the reference document with the highest similarity to the i-th document among the M documents. The reference document is a fault description document corresponding to a resolved fault, and the solution to the fault corresponding to the reference document is used to resolve the fault corresponding to the i-th document. In other words, for a newly occurring fault, the historical document with the highest similarity can be found based on its document, and the new fault can be resolved using the solution to the fault corresponding to that historical document.

[0203] By adopting the technical solution of the embodiment of the present application for troubleshooting, a solution can be automatically obtained based on historical documents effectively without manual processing, thereby saving time and labor costs and improving processing efficiency.

[0204] The above describes in detail the method embodiments of the embodiments of the present application. The following describes the device embodiments of the embodiments of the present application. The device embodiments and the method embodiments correspond to each other. Therefore, for the parts that are not described in detail, please refer to the previous method embodiments. The device can implement any possible implementation method of the above methods.

[0205] Figure 4 A schematic block diagram of a document processing apparatus 400 according to an embodiment of the present application is shown. The apparatus 400 can execute the document processing method according to the embodiment of the present application, for example, the apparatus 400 can be the aforementioned processing apparatus 110.

[0206] like Figure 4 As shown, the apparatus 400 may include:

[0207] An acquisition unit 410 is configured to acquire M documents;

[0208] The document representation unit 420 is configured to determine the frequency x of the qth sentence in the N documents in the ith document based on the similarity between the sentence in the ith document and the N sentences in the M documents. i,q , where M and N are both integers greater than 1, and q is a positive integer not greater than N; according to the frequency x corresponding to the qth sentence in the i-th document i,q , determine the document representation of the i-th document.

[0209] Optionally, in one embodiment of the present application, when i is not greater than M, the i-th document is a document among the M documents.

[0210] Optionally, in one embodiment of the present application, when i is greater than M, the i-th document is a document outside the M documents.

[0211] Optionally, in one embodiment of the present application, the document representation unit 420 is specifically configured to:

[0212] According to the frequency of the qth sentence in the M documents, the inverse document frequency (idf) corresponding to the qth sentence is determined. q ;

[0213] According to the frequency x of the qth sentence in the i-th document i,q and the inverse document frequency idf corresponding to the qth sentence q , determine the document representation of the i-th document.

[0214] Optionally, in one embodiment of the present application, the inverse document frequency (IDF) corresponding to the qth sentence is q It is negatively correlated with the number of documents containing the qth sentence in the M documents.

[0215] Optionally, in one embodiment of the present application, if the frequency corresponding to the qth sentence in the pth document among the M documents is greater than zero, then the pth document contains the qth sentence, and p is a positive integer not greater than M.

[0216] Optionally, in one embodiment of the present application, the frequency x corresponding to the qth sentence in the i-th document is i,q is the sum of the similarities between the qth sentence and each sentence in the i-th document.

[0217] Optionally, in one embodiment of the present application, the document representation unit 420 is specifically configured to:

[0218] According to the K most similar sentences of the sentence of the i-th document in the N sentences, determine the frequency x corresponding to the q-th sentence in the i-th document i,q , where K is a positive integer less than N.

[0219] Optionally, in one embodiment of the present application, K is positively correlated with N.

[0220] Optionally, in one embodiment of the present application, the frequency x corresponding to the qth sentence in the i-th document is i,q is the sum of the K nearest neighbor similarities between the qth sentence and each sentence in the i-th document;

[0221] Wherein, if the qth sentence belongs to the K most similar sentences to the jth sentence of the i-th document, then the K nearest neighbor similarity between the qth sentence and the jth sentence of the i-th document is the similarity between the qth sentence and the jth sentence of the i-th document;

[0222] If the qth sentence does not belong to the K most similar sentences to the jth sentence of the ith document, then the K nearest neighbor similarity between the qth sentence and the jth sentence of the ith document is zero, and j is not greater than n i A positive integer, n i is the number of sentences in the i-th document.

[0223] Optionally, in one embodiment of the present application, the frequency x corresponding to the qth sentence in the i-th document is i,q is an N-dimensional vector x i The qth element of

[0224] in,

[0225] Among them, for vector N dimension If the lth sentence among the N sentences belongs to The K most similar sentences to the sentence, then The lth element is the lth sentence and the lth element The similarity of the sentences, otherwise, The lth element of is zero, wherein the lth element The sentence is the jth sentence of the i-th document, n i is the number of sentences in the i-th document, j is no more than n i is a positive integer, and l is a positive integer not greater than N.

[0226] Optionally, in one embodiment of the present application, the document representation unit 420 is specifically configured to:

[0227] Determine the inverse document frequency (idf) corresponding to the qth sentence according to the following formula q ,

[0228]

[0229] Here, |*| represents the cardinality of the set.

[0230] Optionally, in one embodiment of the present application, the document representation unit 420 is specifically configured to:

[0231] The document representation z of the i-th document is determined according to the following formula: i ,

[0232]

[0233] Among them, ||*|| represents the 2 norm, the N-dimensional vector y i The qth element y i,q for,

[0234] yi,q =x i,q *idf q .

[0235] Alternatively, as Figure 5 As shown, the apparatus 400 may further include:

[0236] a sentence representation unit 430 for obtaining a sentence representation of the sentence according to the first model;

[0237] The document representation unit is further configured to determine the similarity between sentences based on the sentence representations of the sentences;

[0238] The first model is obtained based on multiple word embedding models. The first model connects multiple vectors to obtain sentence representation of the sentence. The multiple vectors are vectors of words in the sentence obtained according to the multiple word embedding models.

[0239] Optionally, in one embodiment of the present application, the document is a Chinese document, such as Figure 5 As shown, the apparatus 400 may further include:

[0240] The word segmentation unit 440 is used to perform word segmentation processing on the sentence to obtain common words and domain-specific words in the sentence.

[0241] Optionally, in one embodiment of the present application, the word segmentation unit 440 is specifically used to:

[0242] Get the initial word sequence of the sentence;

[0243] According to a general dictionary and a professional dictionary in the field to which the document belongs, the initial word sequence is subjected to reverse maximum matching to obtain the words of the sentence.

[0244] Optionally, in one embodiment of the present application, Figure 5 As shown, the apparatus 400 further includes:

[0245] The processing unit 450 is configured to perform document processing according to the document representation.

[0246] Optionally, in one embodiment of the present application, the i-th document is a fault description document corresponding to an unresolved fault;

[0247] The processing unit 450 is specifically configured to:

[0248] Based on the document representation, determine the reference document with the highest similarity to the i-th document among the M documents, wherein the reference document is a fault description document corresponding to a resolved fault, and the solution to the fault corresponding to the reference document is used to handle the fault corresponding to the i-th document.

[0249] Optionally, in one embodiment of the present application, the similarity is cosine similarity.

[0250] Figure 6 It is a schematic diagram of the hardware structure of the document processing device of an embodiment of the present application. Figure 6 The document processing apparatus 600 shown includes a memory 601 , a processor 602 , a communication interface 603 and a bus 604 . The memory 601 , the processor 602 and the communication interface 603 are connected to each other via the bus 604 .

[0251] The memory 601 may be a read-only memory (ROM), a static storage device, or a random access memory (RAM). The memory 601 may store programs. When the program stored in the memory 601 is executed by the processor 602, the processor 602 and the communication interface 603 are used to perform the various steps of the document processing method of the embodiment of the present application.

[0252] The processor 602 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the document processing device of the embodiment of the present application, or to execute the document processing method of the embodiment of the present application.

[0253] The processor 602 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the document processing method of the embodiment of the present application may be completed by hardware integrated logic circuits in the processor 602 or software instructions.

[0254] The processor 602 may also be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or the like. The storage medium is located in the memory 601, and the processor 602 reads the information in the memory 601 and, in combination with its hardware, completes the functions required to be executed by the units included in the document processing apparatus of the embodiments of the present application, or executes the document processing method of the embodiments of the present application.

[0255] The communication interface 603 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 600 and other devices or a communication network. For example, the document to be characterized can be obtained through the communication interface 603 .

[0256] The bus 604 may include a path for transmitting information between various components of the device 600 (eg, the memory 601 , the processor 602 , and the communication interface 603 ).

[0257] It should be noted that although the above-mentioned device 600 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the device 600 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 600 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the device 600 may also include only the devices necessary to implement the embodiments of the present application, and does not necessarily include Figure 6 All devices shown in .

[0258] An embodiment of the present application further provides a computer-readable storage medium storing program code for execution by a device, wherein the program code includes instructions for executing the steps in the above-mentioned document processing method.

[0259] An embodiment of the present application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned document processing method.

[0260] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0261] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0262] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0263] The terms used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates, the singular forms "a" and "the" are intended to also include plural forms. Similarly, the terms "and / or" as used in this application refer to any and all possible combinations of one or more associated listings. In addition, when used in this application, the term "comprising" refers to the existence of stated features, wholes, steps, operations, elements, and / or components, but does not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or these groups.

[0264] The various aspects, implementations, implementations, or features of the described embodiments can be used individually or in any combination. The various aspects of the described embodiments can be implemented by software, hardware, or a combination of software and hardware. The described embodiments can also be embodied by a computer-readable medium storing computer-readable code, the computer-readable code comprising instructions executable by at least one computing device. The computer-readable medium can be associated with any data storage device capable of storing data that can be read by a computer system. Example computer-readable media can include read-only memory, random access memory, compact disc read-only memory (CD-ROM), hard disk drive (HDD), digital video disc (DVD), magnetic tape, and optical data storage devices. The computer-readable medium can also be distributed among computer systems connected via a network so that the computer-readable code can be stored and executed in a distributed manner.

[0265] The above technical description may refer to the accompanying drawings, which form a part of this application and illustrate implementation methods in accordance with the described embodiments in the drawings. Although these embodiments are described in sufficient detail to enable those skilled in the art to implement these embodiments, these embodiments are non-limiting; other embodiments can be used and changes can be made without departing from the scope of the described embodiments. For example, the order of operations described in the flowchart is non-limiting, so the order of two or more operations illustrated in the flowchart and described according to the flowchart can be changed according to several embodiments. As another example, in several embodiments, one or more operations illustrated in the flowchart and described according to the flowchart are optional or deletable. In addition, certain steps or functions can be added to the disclosed embodiments, or the order of two or more steps can be replaced. All these changes are considered to be included in the disclosed embodiments and the claims.

[0266] In addition, terms are used in the above technical description to provide a thorough understanding of the described embodiments. However, overly detailed details are not required to implement the described embodiments. Therefore, the above description of the embodiments is presented for the purpose of illustration and description. The embodiments presented in the above description and the examples disclosed based on these embodiments are provided separately to add context and help understand the described embodiments. The above description is not intended to be exhaustive or to limit the described embodiments to the precise form of the present application. Based on the above teachings, several modifications, selective applications and variations are feasible. In some cases, well-known processing steps are not described in detail to avoid unnecessarily affecting the described embodiments.

[0267] The above description is merely a specific implementation of the embodiments of the present application, but the scope of protection of the embodiments of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the embodiments of the present application should be included in the scope of protection of the embodiments of the present application. Therefore, the scope of protection of the embodiments of the present application should be based on the scope of protection of the claims.

Claims

1. A document processing method, characterized in that: include: Based on the similarity between the sentence in the i-th document and the N sentences in the M documents, determine the frequency x of the q-th sentence in the N documents in the i-th document. i,q , where M and N are both integers greater than 1, q is a positive integer not greater than N, and i is a positive integer; According to the frequency x of the qth sentence in the i-th document i,q , determine the document representation of the i-th document; The step of determining the document representation of the i-th document includes: According to the frequency of the qth sentence in the M documents, the inverse document frequency (idf) corresponding to the qth sentence is determined. q ; According to the frequency x of the qth sentence in the i-th document i,q The inverse document frequency (idf) corresponding to the qth sentence q , determine the document representation of the i-th document; Among them, the frequency x corresponding to the qth sentence in the i-th document i,q is an N-dimensional vector x i The qth element of in, Among them, for vector N dimension If the lth sentence among the N sentences belongs to The K most similar sentences to the sentence, then The lth element is the lth sentence and the lth element The similarity of the sentences, otherwise, The lth element of is zero, wherein the lth element The sentence is the jth sentence of the i-th document, n i is the number of sentences in the i-th document, j is no more than n i A positive integer, l is a positive integer not greater than N; Wherein, the inverse document frequency (idf) corresponding to the qth sentence is determined q ,include: Determine the inverse document frequency (idf) corresponding to the qth sentence according to the following formula q , Among them, |*| represents the cardinality of the set; The step of determining the document representation of the i-th document includes: The document representation z of the i-th document is determined according to the following formula: i , Among them, ||*|| represents the 2 norm, the N-dimensional vector y i The qth element y i,q for, y i,q =x i,q *idf q 。 2. The method according to claim 1, characterized in that When i is not greater than M, the i-th document is a document among the M documents.

3. The method according to claim 1, characterized in that When i is greater than M, the i-th document is a document outside the M documents.

4. The method according to claim 1, wherein The inverse document frequency (idf) corresponding to the qth sentence q It is negatively correlated with the number of documents containing the qth sentence in the M documents.

5. The method according to claim 4, characterized in that If the frequency of the qth sentence in the pth document among the M documents is greater than zero, then the pth document contains the qth sentence, and p is a positive integer not greater than M.

6. The method according to claim 1, characterized in that The frequency x of the qth sentence in the i-th document i,q is the sum of the similarities between the qth sentence and each sentence in the i-th document.

7. The method according to claim 1, characterized in that The determination of the frequency x corresponding to the qth sentence in the N sentences in the i-th document i,q ,include: According to the K most similar sentences of the sentence of the i-th document in the N sentences, determine the frequency x corresponding to the q-th sentence in the i-th document i,q , where K is a positive integer less than N.

8. The method according to claim 7, characterized in that K is positively correlated with N.

9. The method according to claim 7 or 8, characterized in that The frequency x of the qth sentence in the i-th document i,q is the sum of the K nearest neighbor similarities between the qth sentence and each sentence in the i-th document; Wherein, if the qth sentence belongs to the K most similar sentences to the jth sentence of the i-th document, then the K nearest neighbor similarity between the qth sentence and the jth sentence of the i-th document is the similarity between the qth sentence and the jth sentence of the i-th document; If the qth sentence does not belong to the K most similar sentences to the jth sentence of the ith document, then the K nearest neighbor similarity between the qth sentence and the jth sentence of the ith document is zero, and j is not greater than n i A positive integer, n i is the number of sentences in the i-th document.

10. The method according to claim 1, characterized in that The method further comprises: According to the first model, obtaining sentence representation of the sentence; Determine the similarity between sentences based on their sentence representations; Among them, the first model is obtained based on multiple word embedding models, the first model connects multiple vectors to obtain sentence representation of the sentence, and the multiple vectors are vectors of words in the sentence obtained according to the multiple word embedding models.

11. The method according to claim 10, characterized in that The document is a Chinese document, and the method further includes: Perform word segmentation on the sentence to obtain the words of the sentence.

12. The method according to claim 11, characterized in that The word segmentation processing of the sentence includes: Get the initial word sequence of the sentence; According to a general dictionary and a specialized dictionary in the field to which the document belongs, the initial word sequence is subjected to reverse maximum matching to obtain general words in the sentence and words specific to the field.

13. The method according to claim 1, wherein The method further comprises: Document processing is performed according to the document representation.

14. The method according to claim 13, characterized in that The i-th document is a fault description document corresponding to the unresolved fault; The performing document processing according to the document representation includes: Based on the document representation, determine the reference document with the highest similarity to the i-th document among the M documents, wherein the reference document is a fault description document corresponding to a resolved fault, and the solution to the fault corresponding to the reference document is used to handle the fault corresponding to the i-th document.

15. The method according to claim 1, wherein The similarity is cosine similarity.

16. A document processing device, characterized in that: include: An acquisition unit (410), configured to acquire M documents; The document representation unit (420) is used to determine the frequency x of the qth sentence in the N documents in the ith document based on the similarity between the sentence in the ith document and the N sentences in the M documents. i,q , where M and N are both integers greater than 1, q is a positive integer not greater than N, and i is a positive integer; based on the frequency x corresponding to the qth sentence in the i-th document i,q , determine the document representation of the i-th document; The document representation unit (420) is specifically used for: According to the frequency of the qth sentence in the M documents, the inverse document frequency (idf) corresponding to the qth sentence is determined. q ; According to the frequency x of the qth sentence in the i-th document i,q The inverse document frequency (idf) corresponding to the qth sentence q , determine the document representation of the i-th document; Among them, the frequency x corresponding to the qth sentence in the i-th document i,q is an N-dimensional vector x i The qth element of in, Among them, for vector N dimension If the lth sentence among the N sentences belongs to The K most similar sentences to the sentence, then The lth element is the lth sentence and the lth element The similarity of the sentences, otherwise, The lth element of is zero, wherein the lth element The sentence is the jth sentence of the i-th document, n i is the number of sentences in the i-th document, j is no more than n i A positive integer, l is a positive integer not greater than N; The document representation unit (420) is specifically used for: Determine the inverse document frequency (idf) corresponding to the qth sentence according to the following formula q , Among them, |*| represents the cardinality of the set; The document representation unit (420) is specifically used for: The document representation z of the i-th document is determined according to the following formula: i , Among them, ||*|| represents the 2 norm, the N-dimensional vector y i The qth element y i,q for, y i,q =x i,q *idf q 。 17. The device according to claim 16, characterized in that When i is not greater than M, the i-th document is a document among the M documents.

18. The device according to claim 16, characterized in that When i is greater than M, the i-th document is a document outside the M documents.

19. The device according to claim 16, characterized in that The inverse document frequency (idf) corresponding to the qth sentence q It is negatively correlated with the number of documents containing the qth sentence in the M documents.

20. The device according to claim 19, characterized in that If the frequency of the qth sentence in the pth document among the M documents is greater than zero, then the pth document contains the qth sentence, and p is a positive integer not greater than M.

21. The device according to claim 16, characterized in that The frequency x of the qth sentence in the i-th document i,q is the sum of the similarities between the qth sentence and each sentence in the i-th document.

22. The device according to claim 16, characterized in that The document representation unit (420) is specifically used for: According to the K most similar sentences of the sentence of the i-th document in the N sentences, determine the frequency x corresponding to the q-th sentence in the i-th document i,q , where K is a positive integer less than N.

23. The device according to claim 22, characterized in that K is positively correlated with N.

24. The device according to claim 22 or 23, characterized in that The frequency x of the qth sentence in the i-th document i,q is the sum of the K nearest neighbor similarities between the qth sentence and each sentence in the i-th document; Wherein, if the qth sentence belongs to the K most similar sentences to the jth sentence of the i-th document, then the K nearest neighbor similarity between the qth sentence and the jth sentence of the i-th document is the similarity between the qth sentence and the jth sentence of the i-th document; If the qth sentence does not belong to the K most similar sentences to the jth sentence of the ith document, then the K nearest neighbor similarity between the qth sentence and the jth sentence of the ith document is zero, and j is not greater than n i A positive integer, n i is the number of sentences in the i-th document.

25. The device according to claim 16, characterized in that The device further comprises: A sentence representation unit (430) is used to obtain a sentence representation of the sentence according to the first model; Wherein, the document representation unit (420) is further used to determine the similarity between sentences based on the sentence representations of the sentences; The first model is obtained based on multiple word embedding models. The first model connects multiple vectors to obtain sentence representation of the sentence. The multiple vectors are vectors of words in the sentence obtained according to the multiple word embedding models.

26. The device according to claim 25, characterized in that The document is a Chinese document, and the device further includes: The word segmentation unit (440) is used to perform word segmentation processing on the sentence to obtain the words of the sentence.

27. The device according to claim 26, characterized in that The word segmentation unit (440) is specifically used for: Get the initial word sequence of the sentence; According to a general dictionary and a specialized dictionary in the field to which the document belongs, the initial word sequence is subjected to reverse maximum matching to obtain general words in the sentence and words specific to the field.

28. The device according to claim 16, characterized in that The device further comprises: The processing unit (450) is used for performing document processing according to the document representation.

29. The device according to claim 28, characterized in that The i-th document is a fault description document corresponding to the unresolved fault; The processing unit (450) is specifically configured to: Based on the document representation, determine the reference document with the highest similarity to the i-th document among the M documents, wherein the reference document is a fault description document corresponding to a resolved fault, and the solution to the fault corresponding to the reference document is used to handle the fault corresponding to the i-th document.

30. The device according to claim 16, wherein The similarity is cosine similarity.

31. A document processing device, characterized in that: include: Memory (601), for storing programs; A processor (602) is configured to execute a program stored in the memory (601); when the program stored in the memory (601) is executed, the processor (602) is configured to execute the document processing method according to any one of claims 1 to 15.

32. A computer-readable storage medium, characterized in that The computer-readable storage medium is used for program code executed by a device, wherein the program code includes instructions for executing the steps of the document processing method according to any one of claims 1 to 15.

Images