Document analysis method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a document analysis method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: dividing a to-be-analyzed document into a plurality of text units according to a line division mode, and obtaining the line number of each text unit in the to-be-analyzed document; and traversing the text units in sequence according to the ranking of the line numbers, inputting every two adjacent traversed text units into a pre-trained document analysis model for semantic analysis, and outputting a semantic analysis result. And judging whether every two adjacent text units belong to the same semantic fragment or not based on the semantic analysis result. According to the scheme, whether the text units belong to the same semantic fragment or not is judged by utilizing the document analysis model in a semantic analysis mode, a basis can be provided for splicing of subsequent text units, and higher generalization performance and higher accuracy are achieved.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a document parsing method, device, electronic device and computer-readable storage medium. Background Art

[0002] At present, there are many application requirements that require the construction of semantically complete texts, for example, constructing semantically complete texts as feedback during information retrieval to provide users with accurate and complete retrieval results. Due to the layout design of some documents, the text blocks included in the document may be divided and scattered in multiple layout areas or columns. Therefore, splicing technology is needed to reconstruct these scattered text blocks into coherent and semantically complete texts.

[0003] In the prior art, a rule-based matching method or a position information detection method is generally used to determine whether the text is semantically consistent. Among them, the rule-based matching method uses pre-defined rules to determine whether two texts need to be spliced. This method relies on the design of rules. When faced with documents with diverse formats and complex contents, it often requires a lot of adjustments and optimizations, and the generalization ability of the rules is poor. The position-based detection method determines whether two texts need to be spliced ​​based on the positional relationship of the text in the document. This method performs splicing based solely on position information, and splicing errors are very likely to occur in cases such as paragraph wrapping and table splitting. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a document parsing method, device, electronic device and computer-readable storage medium to achieve document parsing with stronger generalization performance and higher accuracy.

[0005] In a first aspect, the present invention provides a document parsing method, the method comprising:

[0006] Divide the document to be parsed into multiple text units according to the line division method, and obtain the line number of each text unit in the document to be parsed;

[0007] Traversing the text units in order according to the order of line numbers, inputting every two adjacent text units traversed into a pre-trained document parsing model for semantic analysis, and outputting a semantic analysis result;

[0008] Based on the semantic analysis result, it is determined whether every two adjacent text units belong to the same semantic segment.

[0009] In an optional embodiment, the method further comprises:

[0010] If the semantic analysis result indicates that the two adjacent text units belong to the same semantic segment, the two adjacent text units are spliced ​​together, wherein all text units with continuous line numbers and spliced ​​together constitute a text slice;

[0011] If the semantic analysis result indicates that the two adjacent text units do not belong to the same semantic segment, the two adjacent text units are divided into two different text slices respectively.

[0012] In an optional implementation, the step of splicing the two adjacent text units includes:

[0013] Two adjacent text units belonging to the same semantic fragment are placed in the constructed text pool until two adjacent text units that do not belong to the same semantic fragment are traversed, all the text units in the text pool are spliced ​​in sequence according to the line numbers to form a text slice, and after the text pool is emptied, the latter text unit of the two adjacent text units that do not belong to the same semantic fragment is placed in the emptied text pool.

[0014] In an optional implementation, the step of placing two adjacent text units belonging to the same semantic segment into the constructed text pool includes:

[0015] After placing two adjacent text units belonging to the same semantic segment into a constructed text pool, the length values ​​of all text units in the text pool are detected;

[0016] If the length value exceeds a set threshold, the original text units in the text pool are spliced ​​into text slices and then the text pool is cleared, and the two adjacent text units belonging to the same semantic segment are placed in the cleared text pool.

[0017] In an optional embodiment, the method further comprises:

[0018] Add the constructed text slices to the knowledge base;

[0019] In response to the obtained consulting information, calculating the matching degree between the consulting information and each of the text slices in the knowledge base;

[0020] A text slice with the highest matching degree is obtained, and the text slice with the highest matching degree is fed back.

[0021] In an optional embodiment, the method further includes a step of pre-training the document parsing model, the step comprising:

[0022] Obtaining an original text, and dividing the original text into a plurality of sample texts based on punctuation marks in the original text, each of the sample texts being a sentence, the punctuation marks indicating that the semantics to be expressed by the corresponding sample text has been completed, and each of the sample texts including a plurality of characters;

[0023] Starting from the first character of each sample text, the sample text is divided multiple times in order to obtain multiple sample units, and in each division, a random integer is obtained within a preset range, and the same number of characters as the random integer are obtained based on the random integer to form a sample unit, wherein, for the remaining part of the sample text after the division, the number of characters remaining is less than the lower limit of the preset range, and the remaining part is taken as a sample unit alone;

[0024] Constructing multiple groups of positive samples and multiple groups of negative samples based on the multiple sample units, wherein the positive samples include two sample units belonging to the same semantic segment, the two sample units belonging to the same semantic segment are sample units belonging to the same sentence and adjacent to each other, and the negative samples include two sample units not belonging to the same semantic segment, the two sample units not belonging to the same semantic segment are sample units not belonging to the same sentence or sample units belonging to the same sentence but not adjacent to each other;

[0025] The constructed initial model is trained based on the multiple groups of positive samples and the multiple groups of negative samples to obtain a document parsing model.

[0026] In an optional implementation manner, after the step of dividing the original text into a plurality of sample texts based on punctuation marks in the original text, the method further includes:

[0027] Selecting a plurality of sample texts from the plurality of sample texts according to a preset ratio;

[0028] The punctuation marks of each selected sample text are removed to construct a sample text without punctuation marks, wherein a plurality of sample texts with punctuation marks and a plurality of sample texts without punctuation marks constitute a final sample text.

[0029] In a second aspect, the present invention provides a document parsing device, the device comprising:

[0030] A division module, used for dividing the document to be parsed into a plurality of text units according to a line division method, and obtaining the line number of each text unit in the document to be parsed;

[0031] An analysis module is used to traverse the text units in order according to the order of line numbers, input every two adjacent text units traversed into a pre-trained document parsing model for semantic analysis, and output a semantic analysis result;

[0032] A judgment module is used to judge whether every two adjacent text units belong to the same semantic segment based on the semantic analysis result.

[0033] In a third aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the document parsing method described in any one of the aforementioned implementations.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the document parsing method described in any one of the aforementioned implementations.

[0035] The beneficial effects of the embodiments of the present invention include, for example:

[0036] The present invention provides a document parsing method, device, electronic device and computer-readable storage medium, which divides a document to be parsed into multiple text units according to a line division method, and obtains the line number of each text unit in the document to be parsed. Each text unit is traversed in order according to the line number, and each two adjacent text units traversed are input into a pre-trained document parsing model for semantic analysis, and the semantic analysis result is output. Based on the semantic analysis result, it is determined whether each two adjacent text units belong to the same semantic segment. In this solution, the document parsing model is used to determine whether the text units belong to the same semantic segment through semantic analysis, thereby providing a basis for the splicing of text units, which has stronger generalization performance and higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments of the present invention are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A flowchart of a document parsing method provided by an embodiment of the present invention;

[0039] Figure 2 An example diagram of two groups of texts provided for an embodiment of the present invention;

[0040] Figure 3 A flowchart of a document parsing model training method involved in the document parsing method provided in an embodiment of the present invention;

[0041] Figure 4 An example diagram of a positive sample provided by an embodiment of the present invention;

[0042] Figure 5 An example diagram of negative samples provided by an embodiment of the present invention;

[0043] Figure 6 A flowchart of a splicing method in a document parsing method provided in an embodiment of the present invention;

[0044] Figure 7 for Figure 6 A flowchart of the sub-steps included in S31;

[0045] Figure 8 A flowchart of a method for processing a response to consultation information in a document parsing method provided in an embodiment of the present invention;

[0046] Fig. 9 A functional module block diagram of a document parsing device provided by an embodiment of the present invention;

[0047] Fig.10 Another functional module block diagram of the document parsing device provided by an embodiment of the present invention;

[0048] Fig.11 This is a structural block diagram of an electronic device provided by an embodiment of the present invention.

[0049] Icons: 10 - document parsing device; 101 - division module; 102 - analysis module; 103 - judgment module; 104 - processing module; 105 - response module; 106 - training module; 20 - electronic device; 201 - processor; 202 - memory. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0051] Figure 1 A flowchart of a document parsing method provided in an embodiment of the present invention. The document parsing method can be executed by a document parsing device. The document parsing device can be implemented in the form of hardware and / or software. The document parsing device can be configured in an electronic device. The electronic device can be a smart appliance such as a range hood, a dishwasher, an air conditioning device, or a terminal device such as a mobile phone, a tablet computer, or a computer.

[0052] Specifically, see Figure 1 The document parsing method provided in this embodiment includes the following steps:

[0053] S11, dividing the document to be parsed into multiple text units according to the line division method, and obtaining the line number of each text unit in the document to be parsed.

[0054] S12, traversing each text unit in order according to the order of line numbers, inputting every two adjacent text units traversed into a pre-trained document parsing model for semantic analysis, and outputting the semantic analysis result.

[0055] S13, judging whether every two adjacent text units belong to the same semantic segment based on the semantic analysis result.

[0056] In this embodiment, the document to be parsed may be any document that needs to be analyzed, for example, it may be an instruction manual for a kitchen appliance, an installation manual for each kitchen appliance, or other types of documents.

[0057] The format of the document to be parsed can be word, PDF, or a picture with text. For example, documents in PDF format are often very complex in layout structure, containing elements such as multi-column layouts, tables, pictures, and annotations. The text of the main text part of the document may be divided into multiple layout areas or columns, making it difficult to directly extract semantically complete fragments. Specifically, semantically complete fragments may often be divided into multiple lines of text, i.e., text lines. Therefore, for documents in PDF format, for example, there is a high demand for semantic parsing to provide a basis for splicing.

[0058] A text line can be regarded as an independent text in a document, that is, the content in a text line belongs to a semantic fragment.

[0059] Therefore, in this embodiment, the document to be parsed can be divided into multiple text units according to the line division method, and the text unit here is the text line. In addition, the line number of each text unit in the document to be parsed is obtained. Among them, the line number can be numbered with consecutive numbers, such as 1, 2, 3..., and can be numbered in order from top to bottom.

[0060] After dividing into multiple text units and determining the line number of each text unit, all text units can be stored in a list in the order of line numbers. For the content stored in the list, each text unit can be traversed in sequence in the order of line numbers, and every two adjacent text units can be input into the pre-trained document parsing model for semantic analysis.

[0061] The traversal of each two adjacent text units may include traversal in the manner of AB, BC, CD and so on, or traversal in the manner of AB, CD, EF and so on. The former traversal method can accurately determine all text units belonging to the same semantic segment in the document to be parsed. In comparison, it can avoid the situation where two text units that originally belonged to the same semantic segment are not judged as belonging to the same semantic segment because the two text units are not imported into the document parsing model for semantic analysis.

[0062] For example, for the first line of text unit 1, the second line of text unit 2, the third line of text unit 3 and the fourth line of text unit 4, the form of each adjacent two text units traversed can be text unit 1 + text unit 2, text unit 2 + text unit 3, text unit 3 + text unit 4, or it can also be text unit 1 + text unit 2, text unit 3 + text unit 4.

[0063] The document parsing model is pre-trained based on sample texts. The document parsing model can perform semantic analysis on two input text units and output semantic analysis results. The semantic analysis results can be in the form of judgment results, such as semantic consistency of the two text units or semantic inconsistency of the two text units. Alternatively, the semantic analysis results can also be in the form of probability, which represents the possibility of semantic consistency of the two text units.

[0064] Based on the semantic analysis results output by the document parsing model, it can be determined whether the two input text units belong to the same semantic segment. For example, if the semantic analysis result is a determination result indicating that the semantics of the two are consistent, it can be determined that the two text units belong to the same semantic segment. If the semantic analysis result is a determination result indicating that the semantics of the two are inconsistent, it can be determined that the two text units do not belong to the same semantic segment.

[0065] For another example, if the semantic analysis result is in the form of probability, the output probability can be compared with the preset probability. If the output probability is greater than or equal to the preset probability, it can be determined that the two text units belong to the same semantic segment. If the output probability is less than the preset probability, it can be determined that the two text units do not belong to the same semantic segment.

[0066] In this embodiment, the document parsing model can be used to analyze whether two input text units belong to a semantic segment, rather than analyzing whether the two text units are adjacent in the document. Figure 2In the two examples shown in , the semantic analysis of text unit 1 and text unit 2 in example 1 shows that the semantics of the two text units are consistent and belong to the same semantic segment. Therefore, the semantic analysis result of the document parsing model for text unit 1 and text unit 2 should indicate that the two text units belong to the same semantic segment.

[0067] From a semantic analysis, we can see that text units 3 and 4 in Example 2 are describing different things, that is, the semantics of the two text units are inconsistent and belong to different semantic segments. Therefore, the semantic analysis result of the document parsing model for text units 3 and 4 should indicate that the two text units do not belong to the same semantic segment.

[0068] The document parsing method provided in this embodiment uses a document parsing model to determine whether each two adjacent text units belong to the same semantic segment through semantic analysis, thereby providing a basis for the splicing of text units. Since this solution is implemented by deep learning, it has stronger generalization performance and can be applied to document parsing of different formats and contents. Moreover, it is implemented based on semantic analysis, which can provide more accurate basis information for the splicing of subsequent text units.

[0069] The following first introduces the process of pre-training the document parsing model. Figure 3 The document parsing model provided in this embodiment also includes the following steps of pre-training the document parsing model:

[0070] S21, obtaining an original text, and dividing the original text into a plurality of sample texts based on punctuation marks in the original text.

[0071] Each sample text is a sentence, and the punctuation marks indicate that the semantics to be expressed by the corresponding sample text has been completed. Each sample text includes multiple characters. For example, the punctuation marks may be a period, an exclamation mark, an ellipsis, etc. These punctuation marks usually indicate that the sentence structure before them is complete and the semantics have been completed.

[0072] The format of the original text obtained may be word, PDF, a picture with text, etc. By identifying punctuation marks in the original text, the original text is segmented at the position of the punctuation marks to obtain multiple sample texts, each of which ends with a punctuation mark.

[0073] Considering that in actual application scenarios, there may be some texts with missing punctuation marks, that is, the text is a text with complete semantic expression, that is, a sentence, but the punctuation marks at the end of the text are missing. In order to be applicable to this situation, in this embodiment, after the original text is divided into multiple sample texts based on the punctuation marks in the original text, the following steps may also be included:

[0074] A plurality of sample texts are selected from a plurality of sample texts according to a preset ratio, and punctuation marks of each selected sample text are removed to construct a sample text without punctuation marks, wherein a plurality of sample texts with punctuation marks and a plurality of sample texts without punctuation marks constitute a final sample text.

[0075] In this embodiment, after the original text is segmented based on punctuation marks to obtain multiple sample texts, each sample text ends with a punctuation mark. For multiple sample texts, a preset proportion of sample texts can be selected, wherein the preset proportion can be 20%, 30%, etc. For each selected sample text, the punctuation marks of the sample text are removed. That is, the sentence, exclamation mark or ellipsis mark at the end of the selected sample text is removed.

[0076] In this embodiment, through the above-mentioned augmentation processing of the sample text, a plurality of sample texts with punctuation marks and a plurality of sample texts without punctuation marks can be obtained, so that it can be applicable to the processing of texts with punctuation marks and texts without punctuation marks in actual application scenarios.

[0077] S22, starting from the first character of each sample text, sequentially divide the sample text multiple times to obtain multiple sample units.

[0078] During each division, a random integer is obtained within a preset range, and the same number of characters as the random integer are obtained based on the random integer division to form a sample unit, wherein, for the remaining part of the sample text after the division, the number of characters remaining is less than the lower limit value of the preset range, and the remaining part is taken as a separate sample unit.

[0079] In this embodiment, the preset range may be, for example, the range of [30, 40], which is not specifically limited and can be set as required. Each sample text may need to be divided multiple times. Assuming that when performing the first division, a random integer is generated from the preset range, and assuming that the generated random integer is 35, the sample unit obtained by the first division is composed of the first character to the 35th character of the sample text. Assuming that when performing the second division, the random integer generated from the preset range is 30, the sample unit obtained by the second division is composed of the 36th character to the 65th character of the sample text.

[0080] For a certain sample text, if after performing multiple divisions, the number of remaining characters (i.e., characters that have not yet been divided) is less than the lower limit of the preset range, for example, the lower limit is 30, and the number of remaining characters is 20. In this case, the remaining characters can be regarded as a sample unit.

[0081] In this embodiment, by dividing the sample units based on the generated random integers, it is possible to avoid dividing the sample units according to a uniform fixed length, which results in the trained document parsing model only being able to perform accurate semantic analysis on texts of the uniform fixed length, but having poor performance when performing semantic analysis on texts of other lengths. By obtaining sample units based on the generated random integers and then performing model training, the generalization ability of the trained model in terms of length can be improved, and flexibility is higher.

[0082] S23, constructing multiple groups of positive samples and multiple groups of negative samples based on multiple sample units.

[0083] Among them, the positive sample includes two sample units belonging to the same semantic segment, and the two sample units belonging to the same semantic segment are sample units belonging to the same sentence and adjacent to each other. The negative sample includes two sample units that do not belong to the same semantic segment, and the two sample units that do not belong to the same semantic segment are sample units that do not belong to the same sentence or sample units that belong to the same sentence but are not adjacent to each other.

[0084] In this embodiment, for sample text A, the multiple sample units finally obtained by division can be recorded as (A_1, A_2, ..., A_n), which respectively represent the first sample unit, the second sample unit, and the nth sample unit after the sample text A is divided. In addition, for sample text B, it is assumed that the multiple sample units finally obtained by division are recorded as (B_1, B_2, ..., B_m), which respectively represent the first sample unit, the second sample unit, and the mth sample unit after the sample text B is divided.

[0085] Since the function of the trained document parsing model is to perform semantic analysis on two input text units to determine whether they belong to the same semantic segment, when constructing training samples, the positive samples should be two sample units belonging to the same semantic segment, while the negative samples are sample units that do not belong to the same semantic segment.

[0086] Since two adjacent sample units in the same sentence should be consistent and coherent in semantic expression, the two adjacent sample units belonging to the same sentence can be used as positive samples. For example, taking the above example, A_1 and A_2, A_2 and A_3, A_(n-1) and A_n in sample text A can constitute positive samples. In addition, B_1 and B_2, B_2 and B_3, B_(m-1) and B_m in sample text B can constitute positive samples.

[0087] In this embodiment, each sample unit is regarded as a text line, such as Figure 4 As shown in , two groups of sample units constituting positive samples are listed, including positive sample 1 and positive sample 2.

[0088] Since two sample units in different sentences or in the same sentence but not adjacent are inconsistent or incoherent in semantic expression, they can be understood as not belonging to the same semantic segment. Therefore, negative samples can be constructed based on such sample units.

[0089] Therefore, sample units that belong to the same sentence but are not adjacent, such as A_1 and A_3, B_1 and B_3, etc., can constitute negative samples. In addition, sample units that do not belong to the same sentence, such as A_1 and B_2, A_2 and B_3, etc., can constitute negative samples.

[0090] like Figure 5 As shown in , two groups of sample units constituting negative samples are listed, including negative sample 1 and negative sample 2.

[0091] Based on the above method, multiple groups of positive samples and multiple groups of negative samples can be constructed, and each group of positive samples and negative samples includes two sample units.

[0092] S24, training the constructed initial model based on multiple groups of positive samples and multiple groups of negative samples to obtain a document parsing model.

[0093] In this embodiment, the initial model can be fastText, Bert or other semantic understanding models. During the training process, open source tools such as huggingface can be used for training.

[0094] Each group of positive samples and each group of negative samples have true labels, which are labels that characterize whether the samples are positive samples or negative samples. After each group of positive samples and each group of negative samples are input into the initial model, the initial model can learn the features of the positive samples and negative samples, and finally the initial model can output the output results of the positive samples and negative samples.

[0095] In each training iteration, the constructed loss function is used to evaluate the difference between the output results of positive samples and negative samples and their true labels. Based on the difference, the model parameters of the initial model are adjusted before the next round of iterations are performed. When the training meets the preset requirements, the training can be stopped and the document parsing model obtained by the initial model training can be obtained. The preset requirements can be that the number of iterations reaches the preset maximum number, or the iteration duration reaches the preset maximum duration, or the loss function reaches convergence and no longer changes.

[0096] The above is an implementation method of pre-training a document parsing model in the training phase. The trained document parsing model can be applied to the analysis and processing of the document to be parsed in the actual application phase. In the actual application phase, after the document to be parsed is divided into multiple text units by line division, each two adjacent text units are input into the document parsing model for semantic analysis, and based on the obtained semantic analysis results, it is determined whether each two adjacent text units belong to the same semantic segment.

[0097] On this basis, if the semantic analysis result indicates that two adjacent text units belong to the same semantic segment, the two adjacent text units are spliced ​​together, wherein all text units with continuous line numbers and spliced ​​together constitute a text slice.

[0098] If the semantic analysis result indicates that two adjacent text units do not belong to the same semantic segment, the two adjacent text units are divided into two different text slices.

[0099] In this embodiment, a text slice may be composed of one or more text units. The text units in a text slice belong to the same semantic segment, that is, the semantics of the text units in a text slice are consistent.

[0100] For two adjacent text units T_m and T_(m+1), if the semantic analysis result indicates that the two text units T_m and T_(m+1) belong to the same semantic segment, then the two text units can be spliced ​​together to form the form of T_m+T_(m+1). According to this logic, text units with consecutive line numbers and belonging to the same semantic segment can be spliced ​​together, and the spliced ​​text units can form a text slice.

[0101] However, if the semantic analysis result indicates that text units T_m and T_(m+1) do not belong to the same semantic segment, the two text units cannot be spliced ​​together but are divided into different text slices. For example, text unit T_m is divided into text slice C_w, and text unit T_(m+1) is divided into text slice C_(w+1).

[0102] In this way, multiple text units in the document to be parsed can be divided into multiple text slices, and the semantics of the text units included in each text slice are consistent and coherent. Therefore, text slices can be used as reply content in retrieval consultation scenarios, and can be used to provide more readable and practical replies.

[0103] In this embodiment, when the semantic analysis result indicates that two adjacent text units belong to the same semantic segment, the step of splicing the two adjacent text units can be implemented in the following manner. Figure 6 :

[0104] S31, placing two adjacent text units belonging to the same semantic fragment into the constructed text pool, until traversing to two adjacent text units that do not belong to the same semantic fragment, all the text units in the text pool are spliced ​​in sequence according to the line numbers to form text slices, and the text pool is cleared.

[0105] S32, placing the latter text unit of two adjacent text units that do not belong to the same semantic segment into the cleared text pool.

[0106] In this embodiment, a text pool can be initialized, which can be understood as a buffer pool, and the text units belonging to the same semantic segment can be buffered in the text pool. The text units buffered in the text pool can be saved in a list form.

[0107] Assume that for the first line of text unit 1, the second line of text unit 2, the third line of text unit 3 and the fourth line of text unit 4, text unit 1 and text unit 2 are input into the document parsing model for semantic analysis. If the semantic analysis result indicates that text unit 1 and text unit 2 belong to the same semantic fragment, indicating that text unit 1 and text unit 2 can be spliced, text unit 1 and text unit 2 can be stored in the text pool.

[0108] Text unit 2 and text unit 3 are input into the document parsing model for semantic analysis. If the semantic analysis result indicates that text unit 2 and text unit 3 belong to the same semantic segment, it indicates that text unit 2 and text unit 3 can be spliced, and text unit 2 and text unit 3 can be stored in the text pool. Since text unit 2 has been stored in the text pool before, text unit 2 does not need to be stored repeatedly, or after text unit 2 is stored in the text pool, duplicate text unit 2 in the text pool can be deduplicated.

[0109] Until text unit 3 and text unit 4 do not belong to the same semantic segment, all text units in the current text pool, including text unit 1, text unit 2 and text unit 3, can be spliced ​​together as a text slice, assuming it is text slice 1. After the text pool is emptied, text unit 4 is stored in the emptied text pool, and text unit 4 will be divided into a different text slice, assuming it is text slice 2.

[0110] In addition to text unit 4, text slice 2 may also include text units after text unit 4. Specifically, it needs to be determined based on whether the text unit after text unit 4 belongs to the same semantic segment as text unit 4. For example, if text unit 5 and text unit 4 belong to the same semantic segment, text unit 5 can be spliced ​​with text unit 4, and text unit 5 is added to text slice 2. If text unit 5 and text unit 4 do not belong to the same semantic segment, text slice 2 only includes text unit 4.

[0111] In this embodiment, a text pool is created to cache text units belonging to the same semantic segment, and when a text unit not belonging to the same semantic segment is traversed, the text pool is cleared before the text unit is stored. This ensures that text units belonging to the same semantic segment can be spliced ​​into a text slice, and avoids splicing text units not belonging to the same semantic segment into the same text slice.

[0112] See also Figure 7 In this embodiment, the step of placing two adjacent text units belonging to the same semantic segment into the constructed text pool can be implemented in the following way:

[0113] S311, placing two adjacent text units belonging to the same semantic segment into the constructed text pool and then detecting the length values ​​of all text units in the text pool.

[0114] S312, if the length value exceeds the set threshold, the original text units in the text pool are spliced ​​into text slices and then the text pool is cleared, and two adjacent text units belonging to the same semantic segment are placed in the cleared text pool.

[0115] In this embodiment, if the length of the constructed text slice is too long, it may cause defects such as unsuccinct reply and poor readability when making a text reply based on the text slice. Therefore, when constructing the text slice, a limit mechanism for the length of the text slice is added.

[0116] When determining that two adjacent text units belong to the same semantic segment, the two adjacent text units should be placed in the constructed text pool. However, if there are already text units in the text pool, after placing the two adjacent text units in the text pool, the total length value of all text units in the text pool may exceed the set threshold. This means that the length of the text slice spliced ​​based on the text units in the text pool will exceed the set threshold.

[0117] In order to avoid the length of text slices being too long, after placing text units belonging to the same semantic segment in the text pool, if the length values ​​of all text units in the text pool exceed the set threshold, it is necessary to first splice the original text units in the text pool into text slices, and then clear the text pool. Then, the two adjacent text units currently determined to belong to the same semantic segment are placed in the cleared text pool.

[0118] Through the above method, the semantics of the text units in a text slice can be guaranteed to be consistent, and the length of a single text slice can be avoided to be too long, which will cause problems such as too long response content and poor readability when the text slice is applied to retrieval and consultation scenarios.

[0119] As can be seen from the above, the spliced ​​text slices can be applied to search and consultation scenarios. Figure 8 The document parsing method provided in this embodiment may further include the following steps:

[0120] S41, adding the constructed text slices to the knowledge base.

[0121] S42, in response to the obtained consulting information, calculating the matching degree between the consulting information and each text slice in the knowledge base.

[0122] S43, obtaining the text slice with the highest matching degree, and feeding back the text slice with the highest matching degree.

[0123] In this embodiment, the text slices constructed in the above manner are stored in the knowledge base. When the user performs a search consultation, the user can enter the consultation information, which can be voice information or text information. After obtaining the user's consultation information, the consultation information is matched with each text slice in the knowledge base. The matching can be achieved based on the set rules, or based on the retrieval model obtained by pre-training based on samples.

[0124] When performing the matching between the consulting information and each text slice, each text slice may be first vectorized to be converted into a vector form. Similarly, the consulting information is converted into a vector form. Then, the matching degree between the consulting information in the vector form and each text slice is calculated.

[0125] In this way, the text slice with the highest matching degree with the consulting information can be determined, and the text slice with the highest matching degree can be fed back to the user.

[0126] For example, if the user's inquiry information is about the installation method of a kitchen appliance, the knowledge base includes text slices constructed by parsing documents related to the installation of the kitchen appliance. Based on this, after calculating the matching degree between the inquiry information and each text slice in the knowledge base, the text slice related to the installation of kitchen appliance pipes in the knowledge base can be found, that is, the text slice with the highest matching degree, and the text slice is fed back to the user.

[0127] In this way, the information that the user needs to know can be provided, and because the content of the text slice is coherent and semantically consistent, using the text slice as the reply content is more readable and practical.

[0128] The document parsing method provided in this embodiment performs semantic analysis on two adjacent text units through a text parsing model to determine whether the two belong to the same semantic segment, and then splices the text units belonging to the same semantic segment to form a text slice. The document parsing model can understand and judge the semantic relationship of the text through deep learning technology, thereby ensuring the accuracy of splicing.

[0129] By splicing scattered text units into semantically coherent and complete text slices, the application can improve the readability and practicality of the text response content returned to users when used in retrieval consultation scenarios.

[0130] Based on the same inventive concept, please refer to Fig. 9 , showing a schematic diagram of the functional modules of the document parsing device 10 provided in an embodiment of the present invention. This embodiment can divide the functional modules of the document parsing device 10 according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic, which is only a logical functional division, and there may be other division methods in actual implementation.

[0131] For example, when each functional module is divided into corresponding functional modules, Fig. 9 The document parsing device 10 shown is only a schematic diagram of the device. The document parsing device 10 may include a division module 101, an analysis module 102, and a judgment module 103. The functions of each functional module of the document parsing device 10 are described in detail below.

[0132] The division module 101 is used to divide the document to be parsed into multiple text units according to the line division method, and obtain the line number of each text unit in the document to be parsed;

[0133] Analysis module 102, used to traverse each text unit in order according to the order of line numbers, input every two adjacent text units traversed into a pre-trained document parsing model for semantic analysis, and output the semantic analysis result;

[0134] The judgment module 103 is used to judge whether every two adjacent text units belong to the same semantic segment based on the semantic analysis result.

[0135] In the solution provided in this embodiment, a document parsing model is used to determine whether text units belong to the same semantic segment through semantic analysis, thereby providing a basis for splicing text units, with stronger generalization performance and higher accuracy.

[0136] In one possible implementation, see Fig.10 The document parsing device 10 further includes a processing module 104, which can be used to:

[0137] If the semantic analysis result indicates that two adjacent text units belong to the same semantic segment, the two adjacent text units are spliced ​​together, wherein all text units with consecutive line numbers and spliced ​​together constitute a text slice;

[0138] If the semantic analysis result indicates that two adjacent text units do not belong to the same semantic segment, the two adjacent text units are divided into two different text slices.

[0139] In a possible implementation, the processing module 104 may be used to splice two adjacent text units in the following manner:

[0140] Two adjacent text units belonging to the same semantic fragment are placed in the constructed text pool until two adjacent text units that do not belong to the same semantic fragment are traversed, all the text units in the text pool are spliced ​​in sequence according to the line numbers to form a text slice, and after the text pool is emptied, the latter of the two adjacent text units that do not belong to the same semantic fragment is placed in the emptied text pool.

[0141] In a possible implementation, the processing module 104 may be used to place two adjacent text units belonging to the same semantic segment into the constructed text pool in the following manner:

[0142] After placing two adjacent text units belonging to the same semantic segment into the constructed text pool, the length values ​​of all text units in the text pool are detected;

[0143] If the length value exceeds the set threshold, the original text units in the text pool are spliced ​​into text slices and then the text pool is cleared, and two adjacent text units belonging to the same semantic segment are placed in the cleared text pool.

[0144] In a possible implementation, the document parsing device 10 further includes a response module 105, which can be used to:

[0145] Add the constructed text slices to the knowledge base;

[0146] In response to the obtained consultation information, calculating the matching degree between the consultation information and each text slice in the knowledge base;

[0147] A text slice with the highest matching degree is obtained, and the text slice with the highest matching degree is fed back.

[0148] In a possible implementation, the document parsing device 10 further includes a training module 106 for pre-training a document parsing model. The training module 106 can be used to:

[0149] Obtaining an original text, and dividing the original text into a plurality of sample texts based on punctuation marks in the original text, each sample text being a sentence, the punctuation marks representing that the semantics to be expressed by the corresponding sample text has been completed, and each sample text including a plurality of characters;

[0150] Starting from the first character of each sample text, the sample text is divided multiple times in order to obtain multiple sample units. During each division, a random integer is obtained within a preset range, and the same number of characters as the random integer are obtained based on the random integer division to form a sample unit, wherein, for the remaining part of the sample text after the division, the number of characters remaining is less than the lower limit of the preset range, and the remaining part is taken as a sample unit separately;

[0151] Construct multiple groups of positive samples and multiple groups of negative samples based on multiple sample units, wherein the positive samples include two sample units belonging to the same semantic segment, and the two sample units belonging to the same semantic segment are sample units belonging to the same sentence and adjacent to each other, and the negative samples include two sample units not belonging to the same semantic segment, and the two sample units not belonging to the same semantic segment are sample units not belonging to the same sentence or sample units belonging to the same sentence but not adjacent to each other;

[0152] The constructed initial model is trained based on multiple groups of positive samples and multiple groups of negative samples to obtain a document parsing model.

[0153] In a possible implementation, the training module 106 may also be used to:

[0154] Select multiple sample texts from multiple sample texts according to a preset ratio;

[0155] Punctuation marks of each selected sample text are removed to construct a sample text without punctuation marks, wherein a plurality of sample texts with punctuation marks and a plurality of sample texts without punctuation marks constitute a final sample text.

[0156] See also Fig.11 An embodiment of the present invention further provides an electronic device 20, which may be a controller of a kitchen appliance or a terminal device, etc. The electronic device 20 includes a memory 202 and a processor 201. The memory 202 stores a computer program that can be run on the processor 201. The above steps are implemented when the processor 201 executes the computer program.

[0157] The memory 202 and the processor 201 in the electronic device 20 communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0158] The memory 202 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory 202 may also be at least one storage device located away from the processor 201.

[0159] The above-mentioned processor 201 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0160] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the above methods is implemented.

[0161] When the embodiments of the present invention are specifically implemented, reference may be made to the above embodiments, and the corresponding technical effects are achieved.

[0162] It is understood that the embodiments described in the present invention can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions of the present invention or a combination thereof.

[0163] For software implementation, the technology of the present invention can be implemented by a unit that performs the functions of the present invention. The software code can be stored in the memory 202 and executed by the processor 201. The memory 202 can be implemented in the processor 201 or outside the processor 201.

[0164] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0165] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0166] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0167] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0169] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, disk or optical disk and other media that can store program codes.

[0170] It should be noted that, in the present invention, terms such as "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0171] The above are only specific embodiments of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined in the present invention can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in the present invention, but should conform to the widest range consistent with the principles and novel features of the present invention.

Claims

1. A document parsing method, characterized in that: The method comprises: Divide the document to be parsed into multiple text units according to the line division method, and obtain the line number of each text unit in the document to be parsed; Traversing the text units in order according to the order of line numbers, inputting every two adjacent text units traversed into a pre-trained document parsing model for semantic analysis, and outputting a semantic analysis result; Based on the semantic analysis result, it is determined whether every two adjacent text units belong to the same semantic segment.

2. The document parsing method according to claim 1, characterized in that: The method further comprises: If the semantic analysis result indicates that the two adjacent text units belong to the same semantic segment, the two adjacent text units are spliced ​​together, wherein all text units with continuous line numbers and spliced ​​together constitute a text slice; If the semantic analysis result indicates that the two adjacent text units do not belong to the same semantic segment, the two adjacent text units are divided into two different text slices respectively.

3. The document parsing method according to claim 2, characterized in that: The step of splicing the two adjacent text units includes: Two adjacent text units belonging to the same semantic fragment are placed in the constructed text pool until two adjacent text units that do not belong to the same semantic fragment are traversed, all the text units in the text pool are spliced ​​in sequence according to the line numbers to form a text slice, and after the text pool is emptied, the latter text unit of the two adjacent text units that do not belong to the same semantic fragment is placed in the emptied text pool.

4. The document parsing method according to claim 3, characterized in that: The step of placing two adjacent text units belonging to the same semantic segment into the constructed text pool includes: After placing two adjacent text units belonging to the same semantic segment into a constructed text pool, the length values ​​of all text units in the text pool are detected; If the length value exceeds a set threshold, the original text units in the text pool are spliced ​​into text slices and then the text pool is cleared, and the two adjacent text units belonging to the same semantic segment are placed in the cleared text pool.

5. The document parsing method according to claim 2, characterized in that: The method further comprises: Add the constructed text slices to the knowledge base; In response to the obtained consulting information, calculating the matching degree between the consulting information and each of the text slices in the knowledge base; A text slice with the highest matching degree is obtained, and the text slice with the highest matching degree is fed back.

6. The document parsing method according to any one of claims 1 to 5, characterized in that: The method further comprises the step of pre-training the document parsing model, which step comprises: Obtaining an original text, and dividing the original text into a plurality of sample texts based on punctuation marks in the original text, each of the sample texts being a sentence, the punctuation marks indicating that the semantics to be expressed by the corresponding sample text has been completed, and each of the sample texts including a plurality of characters; Starting from the first character of each sample text, the sample text is divided multiple times in order to obtain multiple sample units, and in each division, a random integer is obtained within a preset range, and the same number of characters as the random integer are obtained based on the random integer to form a sample unit, wherein, for the remaining part of the sample text after the division, the number of characters remaining is less than the lower limit of the preset range, and the remaining part is taken as a sample unit alone; Constructing multiple groups of positive samples and multiple groups of negative samples based on the multiple sample units, wherein the positive samples include two sample units belonging to the same semantic segment, the two sample units belonging to the same semantic segment are sample units belonging to the same sentence and adjacent to each other, and the negative samples include two sample units not belonging to the same semantic segment, the two sample units not belonging to the same semantic segment are sample units not belonging to the same sentence or sample units belonging to the same sentence but not adjacent to each other; The constructed initial model is trained based on the multiple groups of positive samples and the multiple groups of negative samples to obtain a document parsing model.

7. The document parsing method according to claim 6, characterized in that: After the step of dividing the original text into a plurality of sample texts based on punctuation marks in the original text, the method further includes: Selecting a plurality of sample texts from the plurality of sample texts according to a preset ratio; The punctuation marks of each selected sample text are removed to construct a sample text without punctuation marks, wherein a plurality of sample texts with punctuation marks and a plurality of sample texts without punctuation marks constitute a final sample text.

8. A document parsing device, characterized in that: The device comprises: A division module, used for dividing the document to be parsed into a plurality of text units according to a line division method, and obtaining the line number of each text unit in the document to be parsed; An analysis module is used to traverse the text units in order according to the order of line numbers, input every two adjacent text units traversed into a pre-trained document parsing model for semantic analysis, and output a semantic analysis result; A judgment module is used to judge whether every two adjacent text units belong to the same semantic segment based on the semantic analysis result.

9. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the document parsing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the document parsing method described in any one of claims 1 to 7 is implemented.

Images