Communication engineering record file information processing method, device and storage medium

Abstract

The present invention relates to a communication engineering record file information processing method, device and storage medium, wherein the method comprises: performing OCR processing on the communication engineering record file to obtain the original corpus text; performing sentence processing based on the original corpus text to obtain multiple original sentences; processing the original sentences to obtain multiple word vectors; generating the influence weights between the word vectors according to the distance and part of speech between the word vectors in the same original sentence; updating the word vectors of the word vectors based on the influence weights between all the word vectors in the same original sentence; calculating the distance between the word vectors of all the updated word vectors to obtain the adjacency matrix of the original sentence; inputting the adjacency matrix into the trained detection model to obtain the risk output by the detection model; if the risk is greater than the pre-configured threshold risk, the corresponding original sentence is highlighted. Compared with the prior art, the present invention has the advantages of high detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a communication engineering record file information processing method, device and storage medium. Background Art

[0002] During the implementation of communication engineering projects, process supervision is required. During this process, a large amount of natural text needs to be read, which is inefficient.

[0003] Therefore, those skilled in the art are studying some machine learning techniques to pre-process these natural language corpora. In the prior art, generally, the natural language text is first subjected to OCR processing to obtain text input, and then the input text is subjected to word segmentation processing to obtain word vectors, and then some keywords are extracted using the similarity between each word vector, and then the word vectors of the extracted keywords are input into the diagnosis model to obtain the importance diagnosis result, and finally a preliminary diagnosis result is formed based on the importance obtained and the threshold judgment method to prompt the expert's attention.

[0004] However, when such models are applied to communication engineering projects, since many words in communication engineering projects have high similarities, the extracted keywords are often inaccurate, which makes it impossible to reduce the workload of experts. Summary of the invention

[0005] The purpose of the present invention is to provide a communication engineering record file information processing method, device and storage medium, which can fully express the contextual meaning of the word segmentation by generating multiple original sentences from the original corpus text, and then correcting the word vector of the word segmentation using the word vectors of the previous and next word segmentations, and then calculating the keywords and obtaining the final result, thereby improving the semantic processing results of the overall original sentence and realizing intelligent assisted review.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A communication engineering record file information processing method, comprising:

[0008] Perform OCR processing on the communication engineering record files to obtain the original corpus text;

[0009] Sentence processing is performed based on the original corpus text to obtain multiple original sentences;

[0010] Process the original sentence to obtain multiple word vectors;

[0011] Generate the influence weights between the participles according to the distance and part of speech between the participles in the same original sentence;

[0012] Update the word vector of each word based on the influence weights between all words in the same original sentence;

[0013] Calculate the distance between the word vectors of all updated word segments to obtain the adjacency matrix of the original sentence;

[0014] Input the adjacency matrix into the trained detection model to obtain the risk level output by the detection model;

[0015] If the risk level is greater than the pre-configured threshold risk, the corresponding original statement will be highlighted.

[0016] The sentence segmentation process based on the original corpus text obtains a plurality of original sentences, including:

[0017] The original corpus text is cut into multiple sentence units according to periods, commas and semicolons;

[0018] Process each sentence unit to obtain the word vectors of all the word segments in the sentence unit;

[0019] Based on the word vectors of all the word segments in the sentence unit, the fusion vector of the sentence unit is obtained;

[0020] The similarity of the fusion vectors of the two previous and next sentence units is calculated, and the sentence units are combined based on the calculated similarity to obtain the original sentence.

[0021] The calculating the similarity of the fusion vectors of the two preceding and following sentence units, and combining the sentence units based on the calculated similarity to obtain the original sentence, includes:

[0022] Step S2-1: Calculate the similarity of the fusion vectors of the two previous and next sentence units;

[0023] Step S2-2: creating a first statement set as the current statement set, selecting a first statement unit and placing it in the first statement set;

[0024] Step S2-3: Determine whether the similarity between the fusion vector of the currently selected sentence unit and the next sentence unit exceeds a pre-configured threshold. If yes, execute step S2-4; otherwise, execute step S2-6;

[0025] Step S2-4: taking the latter sentence unit as the currently selected sentence unit, placing it in the current sentence set, and determining whether the number of sentence units in the current sentence set exceeds a pre-configured threshold number, if yes, executing step S2-5, otherwise, executing step S2-7;

[0026] Step S2-5: split the current sentence set using the two adjacent sentence units with the lowest fusion vector similarity in the current sentence set as the dividing line, select the latter split set as the new current sentence unit, and execute step S2-7;

[0027] Step S2-6: creating a new statement set as the current statement set, placing the latter statement unit as the currently selected statement unit in the current statement set, and executing step S2-7;

[0028] Step S2-7: Determine whether there are any untraversed sentence units, if yes, execute step S2-3, otherwise execute step S2-8;

[0029] Step S2-8: concatenate the sentence units in each sentence set to obtain the original sentence corresponding to each sentence set.

[0030] The process of processing the original sentence to obtain multiple word vectors adopts Word2Vec.

[0031] The impact weights are:

[0032] W i,j =f(wd i,j , wc i,j )

[0033] Where: W i,j is the influence weight of the jth participle on the ith participle, wd i,j is the distance sub-weight of the j-th participle to the i-th participle, wc i,j is the part-of-speech sub-weight of the j-th participle to the ith participle.

[0034] The impact weights are specifically:

[0035]

[0036] Where: d i,j is the distance between the jth participle and the ith participle.

[0037] The distance sub-weight of the j-th participle to the ith participle is a monotonically decreasing function.

[0038] The parts of speech of participles include noun, verb, adjective, numeral and others.

[0039] A communication engineering record file information processing device comprises a memory, a processor, and a program stored in the memory, wherein the processor implements the above method when executing the program.

[0040] A storage medium stores a program, which implements the above method when executed.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. By generating multiple original sentences from the original corpus text, and then correcting the word vector of the word segmentation using the word vectors of the previous and next word segmentations, and then calculating the keywords and obtaining the final result, the contextual meaning of the word segmentation can be fully expressed, thereby improving the semantic processing results of the entire original sentence and realizing intelligent assisted review.

[0043] 2. The method of first disassembling into sentence units and then splicing them into original sentences can solve the problem of inaccurate original sentence division caused by inaccurate OCR recognition, thereby improving the accuracy of contextual meaning.

[0044] 3. By comprehensively determining the influence weight by combining the distance and part of speech of the participle, and when the distance is too large, only considering the distance, the interference of part of speech on the contextual meaning can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the main steps of the method of the present invention. DETAILED DESCRIPTION

[0046] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0047] A communication engineering record file information processing method, such as Figure 1 As shown, including:

[0048] (1) Perform OCR processing on the communication engineering record files to obtain the original corpus text;

[0049] This process can be implemented using existing OCR technology. Some record files are in doc, wps and other formats, and the original corpus text can be directly obtained.

[0050] (2) performing sentence segmentation processing based on the original corpus text to obtain multiple original sentences;

[0051] This process is one of the main improvements of this application. Specifically, multiple original sentences are obtained by sentence segmentation based on the original corpus text, including:

[0052] The original corpus text is cut into multiple sentence units according to periods, commas and semicolons;

[0053] Process each sentence unit to obtain the word vectors of all the word segments in the sentence unit;

[0054] Based on the word vectors of all the word segments in the sentence unit, the fusion vector of the sentence unit is obtained;

[0055] The similarity of the fusion vectors of the two previous and next sentence units is calculated, and the sentence units are combined based on the calculated similarity to obtain the original sentence.

[0056] In particular, in this embodiment, the similarity of the fusion vectors of the two preceding and following sentence units is calculated, and the sentence units are combined based on the calculated similarity to obtain the original sentence, including:

[0057] Step S2-1: Calculate the similarity of the fusion vectors of the two previous and next sentence units;

[0058] Step S2-2: creating a first statement set as the current statement set, selecting a first statement unit and placing it in the first statement set;

[0059] Step S2-3: Determine whether the similarity between the fusion vector of the currently selected sentence unit and the next sentence unit exceeds a pre-configured threshold. If yes, execute step S2-4; otherwise, execute step S2-6;

[0060] Step S2-4: taking the next sentence unit as the currently selected sentence unit, placing it in the current sentence set, and determining whether the number of sentence units in the current sentence set exceeds a pre-configured threshold number, if yes, executing step S2-5, otherwise, executing step S2-7;

[0061] Step S2-5: split the current sentence set using the two adjacent sentence units with the lowest fusion vector similarity in the current sentence set as the dividing line, select the latter split set as the new current sentence unit, and execute step S2-7;

[0062] Step S2-6: create a new statement set as the current statement set, place the next statement unit as the currently selected statement unit in the current statement set, and execute step S2-7;

[0063] Step S2-7: Determine whether there are any untraversed sentence units, if yes, execute step S2-3, otherwise execute step S2-8;

[0064] Step S2-8: concatenate the sentence units in each sentence set to obtain the original sentence corresponding to each sentence set.

[0065] In this way, the method of first disassembling into sentence units and then splicing into original sentences can solve the problem of inaccurate original sentence division caused by inaccurate OCR recognition, thereby improving the accuracy of contextual meaning.

[0066] (3) Processing the original sentence to obtain multiple word vectors;

[0067] Generally, this process can use tools such as Word2Vec.

[0068] (iv) generating influence weights between the participles according to the distances and parts of speech between the participles in the same original sentence;

[0069] In this application, the impact weights are:

[0070] W i,j =f(wd i,j , wc i,j )

[0071] Where: W i,j is the influence weight of the jth participle on the ith participle, wd i,j is the distance sub-weight of the j-th participle to the i-th participle, wc i,j is the part-of-speech sub-weight of the j-th participle to the ith participle.

[0072] In the present application, the influence weight consists of two parts, namely, the distance sub-weight of the j-th participle to the ith participle and the part-of-speech sub-weight of the j-th participle to the ith participle.

[0073] Specifically, the impact weights are:

[0074]

[0075] Where: d i,j is the distance between the jth participle and the ith participle.

[0076] The distance sub-weight of the j-th participle to the ith participle is a monotonically decreasing function. In this embodiment, a manual value assignment is adopted. When the distance is 1, the weight is 1, when the distance is 2, the weight is 0.8, when the distance is 3, the weight is 0.6, when the distance is 4, the weight is 0.4, when the distance is 5, the weight is 0.2, when the distance exceeds 6, the weight is 0.1, and when the distance exceeds 6, the weight is 0. Of course, in some other embodiments, the form of a reciprocal function may also be adopted.

[0077] In addition, the parts of speech of participles include nouns, verbs, adjectives, numerals, and others. The specific assignments are shown in Table 1:

[0078] Table 1

[0079]

[0080]

[0081] In Table 1, the first row represents the jth participle, and the first column represents the ith participle.

[0082] (5) updating the word vector of each word based on the influence weights between all words in the same original sentence;

[0083] The specific process includes: first normalizing the influence weight, then weighting and summing the influence weights and word vectors of other word segments in the original sentence, multiplying by a coefficient, and summing them with the original word vector. The mathematical expression is:

[0084]

[0085] Among them, α i ' is the updated word vector of the i-th word segmentation, α i is the word vector of the i-th word segmentation before and after the update, a is the coefficient, w i,j ' is the influence weight of the jth participle on the ith participle after normalization, a j is the word vector of the jth word segmentation before and after the update.

[0086] (VI) Calculate the distance between the word vectors of all updated word segments to obtain the adjacency matrix of the original sentence.

[0087] In the adjacency matrix, the element in the i-th row and j-th column is the Euclidean distance between the word vector of the j-th word segmentation after the update and the word vector of the i-th word segmentation after the update.

[0088] (VII) Inputting the adjacency matrix into the trained detection model to obtain the risk level output by the detection model;

[0089] Among them, the detection model can adopt machine learning models such as BP neural network.

[0090] (VIII) If the risk level is greater than the pre-configured threshold risk, the corresponding original sentence will be highlighted.

[0091] By generating multiple original sentences from the original corpus text, then correcting the word vector of the word segmentation using the word vectors of the previous and next word segmentations, and then calculating the keywords and obtaining the final result, the contextual meaning of the word segmentation can be fully expressed, thereby improving the semantic processing results of the entire original sentence and realizing intelligent assisted review.

[0092] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

Claims

1. A communication engineering record file information processing method, characterized in that: include: Perform OCR processing on the communication engineering record files to obtain the original corpus text; Sentence processing is performed based on the original corpus text to obtain multiple original sentences; Process the original sentence to obtain multiple word vectors; Generate the influence weights between the participles according to the distance and part of speech between the participles in the same original sentence; Update the word vector of each word based on the influence weights between all words in the same original sentence; Calculate the distance between the word vectors of all updated word segments to obtain the adjacency matrix of the original sentence; Input the adjacency matrix into the trained detection model to obtain the risk level output by the detection model; If the risk level is greater than the pre-configured threshold risk, the corresponding original statement will be highlighted; The sentence segmentation process based on the original corpus text obtains a plurality of original sentences, including: The original corpus text is cut into multiple sentence units according to periods, commas and semicolons; Process each sentence unit to obtain the word vectors of all the word segments in the sentence unit; Based on the word vectors of all the word segments in the sentence unit, the fusion vector of the sentence unit is obtained; The similarity of the fusion vectors of the two previous and next sentence units is calculated, and the sentence units are combined based on the calculated similarity to obtain the original sentence.

2. A communication engineering record file information processing method according to claim 1, characterized in that: The calculating the similarity of the fusion vectors of the two preceding and following sentence units, and combining the sentence units based on the calculated similarity to obtain the original sentence, includes: Step S2-1: Calculate the similarity of the fusion vectors of the two previous and next sentence units; Step S2-2: creating a first statement set as the current statement set, selecting a first statement unit and placing it in the first statement set; Step S2-3: Determine whether the similarity between the fusion vector of the currently selected sentence unit and the next sentence unit exceeds a pre-configured threshold. If yes, execute step S2-4; otherwise, execute step S2-6; Step S2-4: taking the latter sentence unit as the currently selected sentence unit, placing it in the current sentence set, and determining whether the number of sentence units in the current sentence set exceeds a pre-configured threshold number, if yes, executing step S2-5, otherwise, executing step S2-7; Step S2-5: split the current sentence set using the two adjacent sentence units with the lowest fusion vector similarity in the current sentence set as the dividing line, select the latter split set as the new current sentence unit, and execute step S2-7; Step S2-6: creating a new statement set as the current statement set, placing the latter statement unit as the currently selected statement unit in the current statement set, and executing step S2-7; Step S2-7: Determine whether there are any untraversed sentence units, if yes, execute step S2-3, otherwise execute step S2-8; Step S2-8: concatenate the sentence units in each sentence set to obtain the original sentence corresponding to each sentence set.

3. A communication engineering record file information processing method according to claim 1, characterized in that: The process of processing the original sentence to obtain multiple word vectors adopts Word2Vec.

4. A communication engineering record file information processing method according to claim 1, characterized in that: The impact weights are: W i,j =f(wd i,j ,wc i,j ) Where: W i,j is the influence weight of the jth participle on the ith participle, wd i,j is the distance sub-weight of the j-th participle to the i-th participle, wc i,j is the part-of-speech sub-weight of the j-th participle to the ith participle.

5. A communication engineering record file information processing method according to claim 4, characterized in that: The impact weights are specifically: Where: d i,j is the distance between the jth participle and the ith participle.

6. A communication engineering record file information processing method according to claim 4 or 5, characterized in that: The distance sub-weight of the j-th participle to the ith participle is a monotonically decreasing function.

7. A communication engineering record file information processing method according to claim 4 or 5, characterized in that: The parts of speech of participles include noun, verb, adjective, numeral and others.

8. A communication engineering record file information processing device, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

9. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 7 is implemented.

Images