Text block sorting method, device, electronic device and storage medium

Abstract

The present application relates to the field of artificial intelligence technology, and provides a text block sorting method, device, electronic device and storage medium. The method obtains the word frequency at the joint between each text block and other text blocks in the text block group to be sorted, and determines the text block similarity between each text block and other text blocks, and merges the word frequency at the joint and the text block similarity to obtain the coherence between each text block and other text blocks, and then determines the sorting of each text block in the text block group to be sorted based on the coherence, thereby achieving automatic sorting of text blocks with relatively low computational complexity, reducing sorting costs on the basis of ensuring sorting accuracy, increasing sorting speed and improving user experience.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text block sorting method, device, electronic device and storage medium. Background Art

[0002] In the fields of computer technology, information processing technology, and document processing technology, OCR (Optical Character Recognition) technology is an important technical means. It can convert documents in image form (such as scanned documents, photographed documents, etc.) into editable text formats, greatly improving the efficiency of document processing. However, when processing complex documents, conventional OCR technology often causes the recognized text blocks to be in a disordered order due to factors such as complex page layout and font diversity, which seriously affects the subsequent processing and use of the document.

[0003] In the related art, the following methods are usually used to sort text blocks: manually correct the sorting errors of text blocks after OCR recognition; based on simple position or format rules; based on language models to evaluate the coherence between text blocks, all possible permutations and combinations are scored by the model, and the highest-scoring permutation is selected as the most likely original text order.

[0004] These sorting methods are either time-consuming and labor-intensive, or require a large amount of computing resources, and the sorting efficiency needs to be improved urgently. Summary of the invention

[0005] In view of this, embodiments of the present application provide a text block sorting method, device, electronic device and storage medium to solve the problem of low efficiency in sorting text blocks in the prior art.

[0006] According to a first aspect of an embodiment of the present application, a text block sorting method is provided, comprising:

[0007] Get the text block group to be sorted;

[0008] Splice any two text blocks in the text block group to be sorted, and obtain the word frequency at the splicing point;

[0009] Determine the text block similarity between any two text blocks in the text block group to be sorted;

[0010] The word frequency at the joint and the similarity of the text blocks are combined to obtain the coherence of any two text blocks;

[0011] The order of each text block in the text block group to be ordered is determined based on the coherence.

[0012] A second aspect of the embodiments of the present application provides a text block sorting device, comprising:

[0013] An acquisition module is configured to acquire a group of text blocks to be sorted;

[0014] The word frequency determination module is configured to splice any two text blocks in the text block group to be sorted to obtain the word frequency at the splicing point;

[0015] A similarity determination module is configured to determine the text block similarity between any two text blocks in the text block group to be sorted;

[0016] A coherence determination module is configured to fuse the word frequency at the splicing point and the similarity of the text blocks to obtain the coherence of any two text blocks;

[0017] The sorting module is configured to determine the sorting of each text block in the text block group to be sorted based on coherence.

[0018] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0019] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0020] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain the word frequency at the joint between each text block and other text blocks in the text block group to be sorted, and determine the text block similarity between each text block and other text blocks, and fuse the word frequency at the joint and the text block similarity to obtain the coherence between each text block and other text blocks, and then determine the sorting of each text block in the text block group to be sorted based on the coherence, thereby realizing automatic sorting of text blocks with relatively low computational complexity, reducing the sorting cost on the basis of ensuring the sorting accuracy, increasing the sorting speed, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 It is a flowchart of a text block sorting method provided in an embodiment of the present application.

[0023] Figure 2It is a flow chart of a method provided in an embodiment of the present application for splicing any two text blocks in a text block group to be sorted to obtain word frequencies at the splicing point.

[0024] Figure 3 It is a flowchart of a method for determining the text block similarity between any two text blocks in a text block group to be sorted provided in an embodiment of the present application.

[0025] Figure 4 It is a flowchart of a method for determining the literal similarity between any two text blocks in a text block group to be sorted provided in an embodiment of the present application.

[0026] Figure 5 It is a flowchart of a method for determining the semantic similarity between any two text blocks in a group of text blocks to be sorted provided in an embodiment of the present application.

[0027] Figure 6 It is a flowchart of a method provided by an embodiment of the present application for fusing word frequency at a splicing point and text block similarity to obtain the coherence of any two text blocks.

[0028] Figure 7 It is a flowchart diagram of another method provided by an embodiment of the present application for fusing word frequency at a splicing point and text block similarity to obtain the coherence of any two text blocks.

[0029] Figure 8 It is a flowchart of a method for determining the order of each text block in a text block group to be sorted based on coherence provided in an embodiment of the present application.

[0030] Fig. 9 It is a flowchart of another method for determining the order of each text block in a text block group to be sorted based on coherence provided in an embodiment of the present application.

[0031] Fig.10 It is a flowchart of another text block sorting method provided in an embodiment of the present application.

[0032] Fig.11 It is a schematic diagram of a text block sorting device provided in an embodiment of the present application.

[0033] Fig.12 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0035] A text block sorting method and device according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0036] As mentioned above, the related art usually uses the following methods to sort text blocks: manually correcting the sorting errors of text blocks after OCR recognition; based on simple position or format rules; based on language models to evaluate the coherence between text blocks, all possible permutations and combinations are scored by the model, and the highest-scoring permutation is selected as the most likely original text order.

[0037] The above methods have the following defects: they are based on manual correction, which is time-consuming, labor-intensive and inefficient; they are based on simple position and format rules, lack understanding of the text content, and are difficult to accurately restore the logical structure of the original document; language model-based methods usually require a lot of computing resources and a long processing time, especially when processing large documents or large batches of documents, which limits their application in certain resource-constrained environments.

[0038] In view of this, an embodiment of the present application provides a text block sorting method, which obtains the word frequency at the joint between each text block and other text blocks in the text block group to be sorted, and determines the text block similarity between each text block and other text blocks, and merges the word frequency at the joint and the text block similarity to obtain the coherence between each text block and other text blocks, and then determines the sorting of each text block in the text block group to be sorted based on the coherence, thereby realizing automatic sorting of text blocks with relatively low computational complexity, reducing the sorting cost on the basis of ensuring sorting accuracy, increasing the sorting speed, and improving the user experience.

[0039] Figure 1 Schematic diagram of a text block sorting method provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0040] In step S101, a group of text blocks to be sorted is obtained.

[0041] In step S102, any two text blocks in the text block group to be sorted are spliced ​​together to obtain the word frequency at the splicing point.

[0042] In step S103, the text block similarity between any two text blocks in the text block group to be sorted is determined.

[0043] In step S104, the word frequency at the joint and the text block similarity are merged to obtain the coherence of any two text blocks.

[0044] In step S105, the order of each text block in the text block group to be ordered is determined based on the coherence.

[0045] In certain embodiments of the present application, the text block sorting method may be executed by a server, or by a terminal with certain computing capabilities.

[0046] In certain implementations, a group of text blocks to be sorted may be first obtained. The group of text blocks to be sorted may be a group of text blocks belonging to the same page among the text blocks obtained after the target text is recognized. For example, the target text may be a PDF (Portable Document Format) document, and the PDF document may be subjected to batch intelligent parsing using OCR and other technologies to obtain a plurality of text blocks. The obtained text blocks may be grouped according to their original page location information (page numbers) to obtain a plurality of text block groups. Furthermore, the group of text blocks that need to be sorted may be used as the above-mentioned group of text blocks to be sorted.

[0047] In some implementations, any two text blocks in the text block group to be sorted can be spliced ​​together, and the number of times the words at the splicing point appear in the target text is counted as the word frequency at the splicing point of the two text blocks. Further, the word frequency at the splicing point between each text block in the text block group to be sorted and other text blocks can be obtained.

[0048] In some implementations, the text block similarity between any two text blocks in the text block group to be sorted may be determined, and then the word frequency at the joint and the text block similarity may be merged to obtain the coherence between any two text blocks in the text block group to be sorted.

[0049] After obtaining the coherence of any two text blocks in the text block group to be sorted, the sorting of the text blocks in the text block group to be sorted can be regarded as a TSP (Traveling Salesman Problem). The TSP problem is solved based on the calculated coherence to obtain the optimal combination, and the sorting of each text block in the text block group to be sorted can be determined.

[0050] According to the technical solution provided by the embodiments of the present application, by obtaining the word frequency at the splicing position of each text block in the text block group to be sorted with other text blocks, determining the text block similarity between each text block and other text blocks, fusing the word frequency at the splicing position and the text block similarity to obtain the coherence between each text block and other text blocks, and then determining the sorting of each text block in the text block group to be sorted based on this coherence, it realizes the automatic sorting of text blocks with a relatively low computational complexity, reduces the sorting cost while ensuring the sorting accuracy, improves the sorting speed, and enhances the user experience.

[0051] Figure 2 FIG. is a flowchart of a method provided by an embodiment of the present application for splicing any two text blocks in a text block group to be sorted to obtain the word frequency at the splicing position. As Figure 2 shown, the method includes the following steps:

[0052] In step S201, each text block in the text block group to be sorted is subjected to word segmentation processing and stop word removal processing.

[0053] In step S202, the number of occurrences of each word after word segmentation in the target text is counted, and the target text is the text where the text block group to be sorted is located.

[0054] In step S203, any two text blocks in the text block group to be sorted are spliced, and the number of occurrences of the words at the splicing position in the target text is determined as the word frequency at the splicing position.

[0055] In some embodiments of the present application, when calculating the word frequency at the splicing position of each text block with other text blocks, each text block in the text block group to be sorted can be first subjected to word segmentation processing and stop word removal processing. Among them, removing stop words can be, for example, filtering out common words such as "de", "he", "zai" that have no actual semantic contribution. Further, the number of occurrences of each word after word segmentation in the target text can be counted, for example, recorded as the total word frequency. In some embodiments, the number of occurrences of each word after word segmentation in each text block can also be counted, that is, how many text blocks in the target text the word appears in, for example, recorded as the document frequency.

[0056] The total word frequency and document frequency obtained by statistics can be saved in a thesaurus. Table 1 is an exemplary thesaurus.

[0057] Table 1 Exemplary Thesaurus

[0058]

[0059] Referring to Table 1, the total number of times the word "car" appears in the target text is 805, and it appears in 215 text blocks of the target text, so the total word frequency of the word "car" is 805 and the document frequency is 215. The meanings of the total word frequency and document frequency of other words are similar and will not be repeated here.

[0060] In some implementations, any two text blocks in the group of text blocks to be sorted may be spliced ​​together, and the number of occurrences of words at the splicing point in the target text may be determined as the word frequency at the splicing point.

[0061] That is to say, a reference text block can be selected from the group of text blocks to be sorted. The reference text block can be, for example, the first text block in the group of text blocks to be sorted, or the next text block corresponding to the last text block on the previous page of the target text, or a text block selected in other ways, and there is no limitation here.

[0062] In the text block group to be sorted, the other text blocks except the reference text block are traversed, the reference text block is spliced ​​with other text blocks one by one, the context sentences at the splicing point are analyzed, and word segmentation is performed. After word segmentation, if the splicing point is located within the same vocabulary, the vocabulary is queried to obtain the total word frequency of the vocabulary in the target text, and it is used as the word frequency at the splicing point of the reference text block and other text blocks currently spliced.

[0063] On the other hand, if the splicing position after word segmentation is not located in the same vocabulary, the word frequency of the splicing position between the reference text block and other text blocks currently spliced ​​can be recorded as 0.

[0064] For example, if text block 0 in the text block group to be sorted is "A branch of a certain bank will give full play to the advantages of diversified financial services and actively support the industry development and transformation and upgrading of a certain company, mainly focusing on comprehensive credit services, digital transformation services, cash management, online banking services, retail financial services and international industry", text block 1 is "A certain company has increased its layout in the new energy industry through technology research and development, sales channel construction investment, etc., with a cumulative R&D investment of nearly 10 billion yuan in the past six years, and annual innovative R&D investment exceeds 10% of sales revenue", and text block 2 is "Provide comprehensive, high-quality and efficient financial services. The signing of this agreement also means that the cooperation between the two parties will be closer, and they will jointly promote the continuous deepening and development of a comprehensive and long-term strategic cooperative relationship."

[0065] If the text block 0 is selected as the reference text block, the text blocks 0 and 1 can be concatenated, and the context sentence at the concatenation is "Retail financial business and international business, a certain company through technology research and development". Tokenizing this sentence gives "Retail, financial, business, and, international, business, a certain, company, through, technology, research, and, development". Since the words "business" and "a certain" at the concatenation are not within the same word, it can be determined that the word frequency at the concatenation of text blocks 0 and 1 is 0.

[0066] Further, concatenating text blocks 0 and 2 gives the context sentence at the concatenation as "Retail financial business and international business, etc.". Tokenizing this sentence gives "Retail, financial, business, and, international, business, etc.". Since the word "business" at the concatenation is within the same word, the total word frequency of this word in the thesaurus can be queried. For example, querying in Table 1 determines that the total word frequency of the word "business" is 123. Therefore, it can be determined that the word frequency at the concatenation of text blocks 0 and 2 is 123.

[0067] Figure 3 It is a schematic flowchart of a method for determining the text block similarity between any two text blocks in a text block group to be sorted provided by an embodiment of the present application. As Figure 3 shown, the method includes the following steps:

[0068] In step S301, determine the literal similarity between the first target text block and the second target text block in the text block group to be sorted.

[0069] Among them, the literal similarity is calculated based on the term frequency - inverse document frequency algorithm.

[0070] In step S302, determine the semantic similarity between the first target text block and the second target text block.

[0071] Among them, the semantic similarity is calculated based on the encoding model.

[0072] In step S303, perform weighted average processing on the literal similarity and the semantic similarity to obtain the text block similarity between the first target text block and the second target text block.

[0073] Among them, the first target text block and the second target text block are any two text blocks in the text block group to be sorted.

[0074] In some embodiments of the present application, to determine the text block similarity between any two text blocks in the text block group to be sorted, for example, the first target text block and the second target text block, it can be to respectively determine the literal similarity and the semantic similarity of the first target text block and the second target text block, and then perform weighted average processing on the literal similarity and the semantic similarity to obtain the text block similarity between the first target text block and the second target text block.

[0075] In some implementations, the similarity between text blocks can be determined after the word frequency at the joint. After determining the word frequency at the joint between the text blocks, the similarity between only two text blocks whose word frequency at the joint is greater than 0 can be calculated.

[0076] That is, the text block similarity between any two text blocks in the text block group to be sorted can be determined by combining literal similarity and semantic similarity to improve the understanding of the text content, thereby accurately restoring the logical structure of the original document when sorting the text blocks.

[0077] Figure 4 1 is a flow chart of a method for determining the literal similarity of any two text blocks in a text block group to be sorted provided by an embodiment of the present application. Figure 4 As shown, the method comprises the following steps:

[0078] In step S401, the term frequency-inverse document frequency of each word in the first target text block and the second target text block are calculated respectively to obtain a first term frequency-inverse document frequency vector of the first target text block and a second term frequency-inverse document frequency vector of the second target text block.

[0079] In step S402, the cosine similarity between the first term frequency-inverse document frequency vector and the second term frequency-inverse document frequency vector is calculated to obtain the literal similarity between the first target text block and the second target text block.

[0080] In some embodiments of the present application, the TF-IDF (term frequency–inverse document frequency) algorithm can be used to calculate the literal similarity between text blocks. In one example, the term frequency-inverse document frequency of each word in the first target text block and the second target text block can be calculated respectively to obtain the first term frequency-inverse document frequency vector of the first target text block and the second term frequency-inverse document frequency vector of the second target text block. Then, the cosine similarity of the first term frequency-inverse document frequency vector and the second term frequency-inverse document frequency vector is calculated to obtain the literal similarity between the first target text block and the second target text block.

[0081] That is to say, the TF-IDF value of each word in the first target text block and the second target text block can be calculated respectively, TF-IDF=TF*IDF, where TF=the number of times the target word appears in the target text block / the total number of words in the target text block, and IDF=log(the total number of text blocks in the text block group to be sorted / (the number of text blocks including the target word in the text block group to be sorted + 1)).

[0082] For example, if the word segmentation result of a text block is "A branch of a certain bank will give full play to the advantages of diversified financial services, actively support the industry development and transformation and upgrading of a certain company, mainly focusing on comprehensive credit services, digital transformation services, cash management, online banking services, retail financial services and international business", that is, there are 38 words in text block 0, among which the target word "finance" has a frequency of 2 in text block 0. If there are 1,000 text blocks in the target text and the word "finance" exists in 85 of them, then the TF-IDF of the word "finance" is (2 / 38)*log(1000 / (85+1))=0.056.

[0083] Next, the text in the first target text block and the second target text block is regarded as a vector composed of the TF-IDF values ​​of all its words, and then the cosine similarity of the two vectors is calculated to obtain the literal similarity between the first target text block and the second target text block. If the vector composed of the TF-IDF values ​​of all the words in the first target text block is recorded as vector A, and the vector composed of the TF-IDF values ​​of all the words in the second target text block is recorded as vector B, then the cosine similarity is ,in, represents the vector dot product, represents the magnitude of vector A, Represents the magnitude of vector B.

[0084] For example, if the TF-IDF values ​​of the words in text block 0 are "retail / 0.01 finance / 0.056 business / 0.02", and the TF-IDF values ​​of the words in text block 1 are "efficient / 0.01 finance / 0.0056 service / 0.02", the word bag [retail finance business efficient service] can be obtained by segmenting text blocks 0 and 1, where the word bag is determined by taking the union of the words in text blocks 0 and 1. Then the vector formed by the TF-IDF values ​​of the words in text block 0 is [0.01 0.056 0.02 0 0], and the vector formed by the TF-IDF values ​​of the words in text block 1 is [0 0.056 0 0.01 0.02]. Therefore, the cosine similarity is .

[0085] Figure 5 is a flow chart of a method for determining the semantic similarity of any two text blocks in a text block group to be sorted provided by an embodiment of the present application. Figure 5 As shown, the method comprises the following steps:

[0086] In step S501, a first target text block and a second target text block are respectively input into a coding model to obtain a first feature vector of the first target text block and a second feature vector of the second target text block.

[0087] In step S502, the cosine similarity between the first feature vector and the second feature vector is calculated to obtain the semantic similarity between the first target text block and the second target text block.

[0088] In certain embodiments of the present application, the encoding model may also be used to extract features from text blocks in the text block group to be sorted, obtain feature vectors of each text block, and then determine the semantic similarity by calculating the cosine similarity of the feature vectors between any two text blocks.

[0089] Among them, the encoding model can be, for example, a BERT (Bidirectional Encoder Representations from Transformer) model. The Transformers library can be installed in the server or terminal that executes the text sorting method provided in the embodiment of the present application, and the BERT model and the word segmenter can be loaded. Among them, the BERT model can be a pre-trained model. The word segmenter is used to segment the text blocks in the text block group to be sorted, and the segmentation results are converted into a BERT mode and input into the BERT model. The BERT model outputs the BERT embedding vector of each text block. Then, the cosine similarity between the BERT embedding vectors of any two text blocks is calculated to obtain their semantic similarity. Among them, the method for calculating the cosine similarity is basically the same as the method for calculating the cosine similarity between the vectors composed of the TF-IDF values ​​mentioned above, which will not be repeated here.

[0090] Figure 6 1 is a flow chart of a method for fusing word frequency at a joint with text block similarity to obtain the coherence of any two text blocks provided by an embodiment of the present application. Figure 6 As shown, the method comprises the following steps:

[0091] In step S601, the word frequency at the target joint of the first target text block and the second target text block, and the similarity of the target text blocks are obtained.

[0092] In step S602, the normalized target splicing word frequency and the normalized target text block similarity are weightedly summed to obtain the coherence of the first target text block and the second target text block.

[0093] In some embodiments of the present application, the word frequency and text block similarity at the joint of the first target text block and the second target text block can be fused by weighted summation to obtain the coherence of the first target text block and the second target text block.

[0094] In one example, the word frequency of the target joint of the first target text block and the second target text block can be obtained, and the word frequency of the target joint can be normalized. On the other hand, the target text block similarity of the first target text block and the second target text block can be obtained, and the target text block similarity can be normalized. Further, the normalized target joint word frequency and the normalized target text block similarity are weighted and summed to obtain the coherence of the first target text block and the second target text block.

[0095] The weight value may be set based on empirical values ​​or in other ways, which are not limited here.

[0096] For example, if the total number of text blocks in the text block group to be sorted is N, the first text block is taken as the base text block, and the word frequencies at the junction of the base text block and other text blocks are calculated as {f2, f3, ..., f N}, and the text block similarity between the benchmark text block and other text blocks is {s2, s3, ..., s N}, then the coherence between the benchmark text block and the i-th text block can be calculated as , where a is a preset weight value, and i is a positive integer greater than 1 and less than or equal to N.

[0097] In the same way, the coherence between any two text blocks in the text block group to be sorted can be calculated.

[0098] Figure 7 FIG. 1 is a flow chart of another method provided by an embodiment of the present application for fusing word frequency at a joint with text block similarity to obtain the coherence of any two text blocks. Figure 7 As shown, the method comprises the following steps:

[0099] In step S701, a preset rule base is obtained.

[0100] The preset rule base includes at least one preset rule, and each preset rule includes a rule condition and a rule score.

[0101] In step S702, in response to determining that the word frequency and text block similarity at the joint of the first target text block and the second target text block meet the rule conditions of the target preset rule, the coherence of the first target text block and the second target text block is set to a first value, otherwise the coherence of the first target text block and the second target text block is set to a second value.

[0102] The first target text block and the second target text block are any two text blocks in the text block group to be sorted, the target preset rule is any preset rule in the preset rule base, and the first value is greater than the second value.

[0103] In some embodiments of the present application, a preset rule base can be used, and then the word frequency and text block similarity at the joint of the first target text block and the second target text block are matched with the preset rule base, and the coherence between the first target text block and the second target text block is determined to be the first value or the second value based on whether the word frequency and text block similarity at the joint meet the rule conditions of each preset rule in the preset rule base.

[0104] Among them, determining whether the word frequency and text block similarity at the joint of the first target text block and the second target text block meet the rule conditions of each preset rule in the preset rule base can be, traversing the preset rules in the preset rule base, obtaining the rule conditions of the currently traversed preset rule, judging whether the word frequency and text block similarity at the joint meet the rule conditions, and if so, setting the coherence of the first target text block and the second target text block to the first value, otherwise, setting it to the second value.

[0105] It is understandable that the rule condition can be a condition that any one of the word frequency at the splicing point and the text block similarity needs to meet, in which case it is only necessary to determine whether the corresponding item of the word frequency at the splicing point and the text block similarity meets the rule condition. Alternatively, the rule condition can also be a condition that both the word frequency at the splicing point and the text block similarity need to meet, in which case it is necessary to determine whether both the word frequency at the splicing point and the text block similarity meet the rule condition.

[0106] A method for determining the coherence between text blocks using a preset rule base may be to first compare the word frequency at the joint of a first target text block and a second target text block with a word frequency threshold in a preset rule base, and in response to determining that the word frequency at the joint of the first target text block and the second target text block is greater than a first preset threshold, set the coherence of the first target text block and the second target text block to a first value.

[0107] If it is determined that the word frequency at the joint of the first target text block and the second target text block is less than or equal to the first preset threshold and greater than the second preset threshold, the text block similarity of the first target text block and the second target text block is further compared with the similarity threshold in the preset rule base. If the text block similarity of the first target text block and the second target text block is greater than the fourth preset threshold, the coherence of the first target text block and the second target text block is set to the first value.

[0108] If it is determined that the word frequency at the joint of the first target text block and the second target text block is less than or equal to the second preset threshold and greater than the third preset threshold, the text block similarity of the first target text block and the second target text block can still be compared with the similarity threshold in the preset rule base. If the text block similarity of the first target text block and the second target text block is greater than the fifth preset threshold, the coherence of the first target text block and the second target text block is set to the first value.

[0109] Furthermore, a priority may be set for each preset rule in the preset rule base. When there are multiple coherences with the first value among the coherences between the first target text block and other text blocks, the priorities of the preset rules satisfied by the first target text block and other text blocks are further determined, and only the coherence value satisfying the preset rule with the highest priority is retained as the first value, while the values ​​of other coherences are modified to the second value.

[0110] For example, the first preset threshold, the second preset threshold, and the third preset threshold as the word frequency threshold at the splicing point can be set to 400, 100, and 10, respectively, to characterize the high frequency, medium frequency, and low frequency of the word frequency at the splicing point, the first value is set to 1, and the second value is set to 0. At the same time, the fourth preset threshold and the fifth preset threshold as the text block similarity threshold can be set to 0.2 and 0.3, respectively. Furthermore, a sixth preset threshold can be set to 0.1.

[0111] If the word frequency at the joint of the first target text block and the second target text block exceeds 400, the coherence between the first target text block and the second target text block can be directly recorded as 1.

[0112] If the word frequency at the junction of the first target text block and the second target text block is less than 400 but greater than 100, it can be further determined whether the text block similarity between the first target text block and the second target text block exceeds 0.2. If so, the coherence between the first target text block and the second target text block is recorded as 1.

[0113] If the word frequency at the joint of the first target text block and the second target text block is less than 100 but exceeds 10, it can be further determined whether the text block similarity between the first target text block and the second target text block exceeds 0.3. If so, the coherence between the first target text block and the second target text block is recorded as 1. Among them, the setting of each preset threshold can be dynamically adjusted according to the actual content difference of the target text. For example, a reasonable word frequency threshold can be set based on the word frequency distribution statistics in the target text to adapt to different document characteristics and improve the flexibility and accuracy of matching. Furthermore, a proportional factor can be set to adjust each preset threshold according to the total number of words in the target text, such as setting the preset threshold of long text to be higher to reduce mismatching, and setting the preset threshold of professional documents to be slightly lower to ensure the correct connection of professional terms.

[0114] It is understandable that it is more convenient to calculate the coherence between text blocks using a rule-based approach. Once a rule is met, there is no need to continue calculating the word frequency or text block similarity between the benchmark module and other modules, and it allows the decision on how to combine model results based on predefined logic or business rules, which can better adapt to complex decision-making scenarios. However, formulating an effective and comprehensive rule set can be very complex and time-consuming, and overly detailed rules may over-fit specific data sets and reduce generalization capabilities.

[0115] On the other hand, the weighted summation method is simple and easy to calculate the coherence between text blocks, but the amount of calculation is greater than that of the rule-based method. However, the weighted summation method is more intuitive and easy to implement. It only needs to assign a weight to the result of each strategy, and then calculate the weighted score. The weight can be adjusted according to the performance of each specific strategy or the expertise in the field, thereby optimizing the overall performance. Therefore, when using it, you can choose the appropriate method to determine the coherence between text blocks according to actual needs.

[0116] In certain implementations, the calculated coherence between the text blocks can be saved for subsequent query when determining the order of the text blocks. In one example, the coherence between the text blocks can be saved in the form of a coherence matrix. For example, if the group of text blocks to be sorted includes N text blocks, an N×N coherence matrix can be constructed, in which the element in the mth row and jth column is the coherence between the mth text block and the jth text block, and the mth text block is the previous text block when splicing, and the jth text block is the next text block when splicing. Both m and j are positive integers greater than or equal to 1 and less than or equal to N.

[0117] For example, if there are four text blocks in the text block group to be sorted, namely A, B, C and D, the coherence between text block A and text blocks B, C and D is calculated by rule-based method as 1, 0 and 0 respectively, the coherence between text block B and text blocks A, C and D is 0, 1 and 0 respectively, the coherence between text block C and text blocks A, B and D is 0, 0 and 1 respectively, and the coherence between text block D and text blocks A, B and C is 0, 0 and 1 respectively. The coherence matrix constructed for it can be:

[0118]

[0119] The optimal path can be directly determined from the coherence matrix as A→B→C→D. Therefore, the ordering of the to-be-ordered text blocks A, B, C and D can be determined as A→B→C→D.

[0120] In other embodiments, if each row of the coherence matrix formed by the coherence calculated by the rule-based method includes more than one item with a value of 1, the order of each text block cannot be directly determined according to the coherence matrix. In this case, the order of each text block can be determined by a dynamic programming algorithm or a greedy algorithm, just like the coherence matrix formed by the coherence calculated by the weighted sum method.

[0121] In one example, for the text blocks A, B, C and D to be sorted, the coherence between the text block A and the text blocks B, C and D is calculated by weighted summation, for example, to be 0.3, 0.2 and 0.5 respectively, the coherence between the text block B and the text blocks A, C and D is for example, 0.4, 0.3 and 0.3 respectively, the coherence between the text block C and the text blocks A, B and D is for example, 0.5, 0.2 and 0.3 respectively, and the coherence between the text block D and the text blocks A, B and C is for example, 0.1, 0.3 and 0.6 respectively. The coherence matrix constructed for them can be:

[0122]

[0123] In one example, a dynamic programming algorithm can be used to determine the order of each text block in the text block group to be sorted based on the calculated coherence. That is, a coherence matrix can be constructed based on the coherence between every two text blocks, and when the number of text blocks is less than or equal to the number threshold, a dynamic programming method is used to find the best text block order. As mentioned above, this method is applicable to the case where the coherence matrix composed of coherence is calculated by weighted summation, and the case where each row of the coherence matrix composed of coherence calculated by rule basis includes more than one item with a value of 1.

[0124] Figure 8FIG. 1 is a flow chart of a method for determining the order of each text block in a text block group to be ordered based on coherence provided by an embodiment of the present application. Figure 8 As shown, the method comprises the following steps:

[0125] In step S801, the initial state of the target text block is defined.

[0126] The target text block is any text block in the group of text blocks to be sorted, the initial state is recorded as a two-dimensional array, the first item of the two-dimensional array is the initial coherence of the target text block, and the second item of the two-dimensional array is the target text block.

[0127] In step S802, the path of the initial state is extended to obtain an intermediate state after the target text block and the next text block are spliced.

[0128] The first item of the intermediate state two-dimensional array is the sum of the initial coherence and the current extended coherence, and the current extended coherence is the coherence between the two text blocks spliced ​​in this path extension.

[0129] In step S803, the intermediate states are path-extended in sequence until all text blocks in the to-be-sorted text block group are traversed to obtain at least one end state.

[0130] The first item of the two-dimensional array of end states is the sum of the coherence of the next end state and the current extended coherence, and the next end state is the previous state of the end state.

[0131] In step S804, all end states obtained by taking each text block in the text block group to be sorted as the target text block are obtained, and the end state with the greatest coherence among all the end states is determined as the target end state.

[0132] In step S805, the order of each text block in the text block group to be sorted is determined according to the text concatenation path corresponding to the target end state.

[0133] As mentioned above, after obtaining the coherence of any two text blocks in the text block group to be sorted, the sorting of the text blocks in the text block group to be sorted can be regarded as a TSP problem. The TSP problem is solved based on the calculated coherence to obtain the optimal combination, and the sorting of each text block in the text block group to be sorted can be determined.

[0134] In some embodiments of the present application, a dynamic programming algorithm can be used to solve the TSP problem. In one example, when using a dynamic programming algorithm to solve the above TSP problem, the initial state of the target text block can be first defined, and then the initial state can be path-extended to obtain an intermediate state after the target text block and the next text block are spliced ​​together. The first item of the two-dimensional array of the intermediate state is the sum of the initial coherence and the current extended coherence, and the current extended coherence is the coherence between the two text blocks spliced ​​in this path extension.

[0135] Furthermore, the intermediate states can be path-extended in sequence until all text blocks in the text block group to be sorted are traversed to obtain at least one end state, wherein the first item of the two-dimensional array of the end state is the sum of the coherence of the next end state and the current extended coherence, and the next end state is the previous state of the end state.

[0136] All end states obtained by taking each text block in the text block group to be sorted as the target text block can be obtained in sequence, and the end state with the greatest coherence among all the end states is determined as the target end state. Finally, the order of each text block in the text block group to be sorted is determined according to the text splicing path corresponding to the target end state.

[0137] Still taking the above text block group to be sorted as including four text blocks A, B, C and D and the coherence between the text blocks is calculated by weighted summation, that is, the constructed coherence matrix is ​​the above matrix S2 as an example, the initial states of text blocks A, B, C and D can be defined as dp(1, A) = (0, -1), dp(2, B) = (0, -1), dp(4, C) = (0, -1), dp(8, D) = (0, -1) respectively. Among them, dp(mask, j) represents the current state, the set of text blocks that have been accessed in this state is mask, and mask is a bit mask. Further, dp(mask, j) can be represented by a two-dimensional array, one item of which represents the coherence when the last spliced ​​text block is the jth text block. The second item of the two-dimensional array represents the previous text block spliced ​​with the last text block. In the initial state, since the text blocks are not spliced, the coherence is recorded as 0, and the previous text block can be represented as the -1th text block.

[0138] Next, state transfer can be performed. For each initial state dp(mask, j), try to extend it to the next unvisited text block.

[0139] For example, we can start from text block A and expand:

[0140] dp(3,B)=(dp(1,A)[0]+s A，B ,A)=(0+0.3,A)=(0.3,A)

[0141] dp(5,C)=(dp(1,A)[0]+s A，C ,A)=(0+0.2,A)=(0.2,A)

[0142] dp(9,D)=(dp(1,A)[0]+s A，D ,A)=(0+0.5,A)=(0.5,A)

[0143] You can also expand from text block B:

[0144] dp(3,A)=(dp(2,B)[0]+s B，A ,A)=(0+0.4,A)=(0.4,A)

[0145] dp(6,C)=(dp(2,B)[0]+s B，C ,A)=(0+0.3,A)=(0.3,A)

[0146] dp(10,D)=(dp(2,B)[0]+s B，D ,A)=(0+0.3,A)=(0.3,A)

[0147] It can also be expanded starting from text block C:

[0148] dp(5,A)=(dp(4,C)[0]+s C，A ,A)=(0+0.5,A)=(0.5,A)

[0149] dp(6,B)=(dp(4,C)[0]+s C，B ,A)=(0+0.2,A)=(0.2,A)

[0150] dp(12,D)=(dp(4,C)[0]+s C，D ,A)=(0+0.3,A)=(0.3,A)

[0151] You can also expand from text block D:

[0152] dp(9,A)=(dp(8,D)[0]+s D，A ,A)=(0+0.1,A)=(0.1,A)

[0153] dp(10,B)=(dp(8,D)[0]+s D，B ,A)=(0+0.3,A)=(0.3,A)

[0154] dp(12,C)=(dp(8,D)[0]+s D，C,A)=(0+0.6,A)=(0.6,A)

[0155] Among them, dp(1,A)[0] represents the first item of the two-dimensional array of state dp(1,A), s A，B Indicates the coherence between text blocks AB when text block B is attached to text block A. This coherence can be obtained from the coherence matrix described above. The meanings of dp(2,B)[0], dp(4,C)[0], and dp(8,D)[0] are similar to those of dp(1,A)[0]. A，C 、s A，D 、s B，A 、s B，C 、s B，D 、s C，A 、s C，B 、s C，D 、s D，A 、s D，B and D，C The meaning of A，B Similar, no further description is given here.

[0156] Next, we can further expand the path. We can start from text block A or from dp(3, B):

[0157] dp(7,C)=(dp(3,B)[0]+s B，C ,B)=(0.3+0.3,B)=(0.6,B)

[0158] dp(11,D)=(dp(3,B)[0]+s B，D ,B)=(0.3+0.3,B)=(0.6,B)

[0159] We can also expand from dp(5,C):

[0160] dp(7,B)=(dp(5,C)[0]+s C，B ,C)=(0.2+0.2,C)=(0.4,C)

[0161] dp(13,D)=(dp(5,C)[0]+s C，D ,C)=(0.2+0.3,C)=(0.5,C)

[0162] You can also expand from dp(9,D):

[0163] dp(11,B)=(dp(9,D)[0]+s D，B ,D)=(0.5+0.3,D)=(0.8,D)

[0164] dp(13,C)=(dp(9,D)[0]+s D，C ,D)=(0.5+0.6,D)=(1.1,D)

[0165] Expand the paths one by one until all possible paths have been tried, including:

[0166] Starting from dp(7,C) and extending:

[0167] dp(15,D)=(dp(7,C)[0]+s C，D ,C)=(0.6+0.3,C)=(0.9,C)

[0168] Starting from dp(11,D) and extending:

[0169] dp(15,C)=(dp(11,D)[0]+s D，C ,D)=(0.6+0.6,D)=(1.2,D)

[0170] Starting from dp(7,B) and extending:

[0171] dp(15,D)=(dp(7,B)[0]+s B，D ,B)=(0.4+0.3,B)=(0.7,B)

[0172] Starting from dp(13,D) and extending:

[0173] dp(15,B)=(dp(13,D)[0]+s D，B ,D)=(0.5+0.3,D)=(0.8,D)

[0174] Starting from dp(11,B) and extending:

[0175] dp(15,C)=(dp(11,B)[0]+s B，D ,B)=(0.8+0.3,B)=(1.1,B)

[0176] Starting from dp(13,C) and extending:

[0177] dp(15,B)=(dp(13,C)[0]+s C，B ,C)=(1.1+0.2,C)=(1.3,C)

[0178] Among them, the meanings of dp(3,B)[0], dp(5,C)[0], dp(9,D)[0], dp(7,C)[0], dp(11,D)[0], dp(7,C)[0], dp(13,D)[0], dp(11,B)[0], and dp(13,C)[0] are similar to those of dp(1,A)[0]. B，C 、s B，D 、s C，B 、s C，D 、s D，B 、s D，C 、s C，D 、s D，C 、s B，D and D，B The meaning of A，B Similar, no further description is given here.

[0179] Similarly, when all text blocks have been visited, the mask is 15 (that is, 1111 in binary), and all possible end states can be obtained, including dp (15, A), dp (15, B), dp (15, C), and dp (15, D). It can be understood that each of dp (15, A), dp (15, B), dp (15, C), and dp (15, D) can include one or more states. For example, in the above example, starting from the state dp (1, A), multiple dp (15, B), multiple dp (15, C), and multiple dp (15, D) can be obtained. Each dp (15, A), dp (15, B), dp (15, C), or dp (15, D) can represent an end state.

[0180] Finally, determine the maximum value among all the end states, that is, determine max(dp(15,A), dp(15,B), dp(15,C), dp(15,D)). In other words, determine the state corresponding to the maximum value of the first item in the two-dimensional array of each dp(15,A), dp(15,B), dp(15,C), and dp(15,D) as the target end state. Starting from the end node corresponding to the target end state, that is, the last text block, backtrack to find the previous text block that leads to the maximum score in each expansion step until returning to the initial state, and the order of each text block can be determined.

[0181] For example, assuming that the maximum coherence is 1.3 and the corresponding state is dp(15, B) = (1.3, C), then backtrack from state dp(15, B) to find the optimal text block path: 1101.

[0182] The first backtracking is: the current state is dp(15, B) = (1.3, C), the previous node is C, and mask = 13 (binary 1101);

[0183] The second backtracking is: the current state is dp(13, C) = (1.1, D), the previous node is D, mask = 9 (binary 1001)

[0184] The third backtracking is: the current state is dp(9, D) = (0.5, A), the previous node is A, mask = 1 (binary 0001)

[0185] In this way, we can backtrack and obtain the path A→D→C→B, that is, the order of the corresponding text blocks is A→D→C→B.

[0186] In another example, a greedy algorithm can be used to determine the order of each text block in the text block group to be sorted based on the calculated coherence. That is, a coherence matrix can be constructed based on the coherence between every two text blocks, and when the number of text blocks is greater than a quantity threshold, a greedy algorithm is used to find the best text block order.

[0187] As mentioned above, this method is also applicable to the case where the coherence matrix composed of coherence calculated by weighted summation and the case where each row of the coherence matrix composed of coherence calculated by rule basis includes more than one item with value 1. Fig. 9 FIG. 1 is a flow chart of another method for determining the order of each text block in a text block group to be ordered based on coherence provided by an embodiment of the present application. Fig. 9 As shown, the method comprises the following steps:

[0188] In step S901, an initial text block is determined.

[0189] The initial text block is the text block in the text block group to be sorted that has the greatest coherence with the last text block on the previous page of the text, or the text block with the smallest sum of coherence with other text blocks in the text block group to be sorted.

[0190] In step S902, it is determined that among the text blocks in the text block group to be sorted except the initial text block, the text block with the greatest coherence with the initial text block is the first concatenated text block.

[0191] In step S903, the text block with the greatest coherence with the k-1th spliced ​​text block among the other text blocks in the text block group to be sorted except the initial text block and the spliced ​​text blocks is determined in sequence as the kth spliced ​​text block, until k equals N-1.

[0192] Wherein, N is the total number of text blocks in the text block group to be sorted, and k is a positive integer greater than 1 and less than or equal to N-1.

[0193] In step S904, the concatenation order from the initial text block to the kth concatenated text block is determined to be the order of the text blocks in the text block group to be sorted.

[0194] In some embodiments of the present application, a greedy algorithm may also be used to solve the above TSP problem. In one example, when using a greedy algorithm to solve the above TSP problem, the initial text block may be first determined, and then the text block with the greatest coherence with the initial text block among the other text blocks in the text block group to be sorted except the initial text block is determined as the first spliced ​​text block. Next, the text block with the greatest coherence with the k-1th spliced ​​text block among the other text blocks in the text block group to be sorted except the initial text block and the spliced ​​text blocks is determined in sequence as the kth spliced ​​text block, until k equals N-1. Finally, the splicing order from the initial text block to the kth spliced ​​text block is determined as the sorting of the text blocks in the text block group to be sorted.

[0195] Still taking the above text block group to be sorted as including four text blocks A, B, C and D and the coherence between the text blocks is calculated by weighted summation, that is, the constructed coherence matrix is ​​the above matrix S2 as an example, if text block A is selected as the initial text block, the current path can be recorded as [A], and the visited text block can be recorded as {A}. Next, iterative selection is performed, and the next unvisited text block with the highest coherence with the current text block is selected at each step. For example, the next unvisited text block with the highest coherence with text block A is selected first. By querying the above coherence matrix, it can be known that A→B: coherence = 0.3; A→C: coherence = 0.2; A→D: coherence = 0.5. Therefore, text block D is selected as the next text block. At this time, the current path is updated to [A, D], and the visited text block is updated to {A, D}.

[0196] Then continue to select the next unvisited text block with the highest coherence with text block D. Continuing to query the above coherence matrix, we know that D→B: coherence = 0.3; D→C: coherence = 0.6. Therefore, select text block C as the next text block, and update the current path to [A, D, C], and update the visited text blocks to {A, D, C}.

[0197] Then select the next unvisited text block with the highest coherence with text block C. Continuing to query the above coherence matrix, we know that C→B: coherence = 0.2. Therefore, select text block B as the next text block, and update the current path to [A, D, C, B], and update the visited text blocks to {A, D, C, B}.

[0198] The iteration is terminated when all text blocks have been visited. At this time, the order of the text blocks is A→D→C→B.

[0199] Fig.10FIG. 1 is a flow chart of another text block sorting method provided in an embodiment of the present application. Fig.10 As shown, when sorting text blocks, data preparation can be performed first, and then word segmentation and word frequency statistics can be performed on the prepared data. For the data after word segmentation and word frequency statistics, on the one hand, the word frequency at the text block splicing point can be calculated, and on the other hand, the text block literal similarity calculation and text block semantic similarity calculation can be performed in sequence, and the text block similarity calculation can be performed based on the calculation results of the literal similarity and semantic similarity. The text block coherence calculation can be performed using the calculation results of the splicing point word frequency and the text block similarity calculation, and finally the text blocks can be sorted based on the calculated coherence.

[0200] The technical solution provided in the embodiment of the present application provides a basis for calculating the similarity of text blocks by performing word segmentation on the text blocks, calculating word frequency and document frequency, filtering out commonly used meaningless words, and retaining valuable vocabulary data. At the same time, the method combines literal similarity and semantic similarity to comprehensively evaluate the similarity between text blocks from different dimensions. Among them, literal similarity is good at capturing the importance and frequency of occurrence of keywords in the text, while semantic similarity based on the pre-trained language model can deeply understand the contextual relationship and semantics between words.

[0201] Furthermore, the method calculates the similarity between text blocks by combining literal similarity and semantic similarity, and calculates the coherence of text blocks by combining word frequency analysis at the splicing points, taking into account both semantic coherence and literal coherence (word frequency at the splicing points).

[0202] Furthermore, this method uses two strategies, weighted fusion and rule-based fusion, to integrate word frequency and similarity data to obtain the coherence of text blocks, thereby optimizing the order of text blocks. At the same time, a coherence matrix is ​​constructed based on the coherence between text blocks, and the final text block order is obtained by combining dynamic programming / greedy thinking.

[0203] By adopting the technical solution of the embodiment of the present application, through a multi-dimensional text analysis method, word segmentation, word frequency calculation, and text block similarity analysis are comprehensively used to optimize the order of text blocks, so that the reorganized text is more coherent in semantics and logic. The method is flexible and adaptable, suitable for a variety of document types and complex typesetting formats, and improves the automation level and efficiency of information processing. At the same time, the method strengthens the understanding of document content, reduces manual intervention, and reduces the error rate, which is particularly suitable for scenarios where a large number of documents are processed.

[0204] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0205] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0206] Fig.11 Schematic diagram of a text block sorting device provided in an embodiment of the present application. Fig.11 As shown, the device comprises:

[0207] The acquisition module 1101 is configured to acquire a group of text blocks to be sorted.

[0208] The word frequency determination module 1102 is configured to splice any two text blocks in the text block group to be sorted to obtain the word frequency at the splicing point.

[0209] The similarity determination module 1103 is configured to determine the text block similarity between any two text blocks in the text block group to be sorted.

[0210] The coherence determination module 1104 is configured to merge the word frequency at the joint and the text block similarity to obtain the coherence of any two text blocks.

[0211] The sorting module 1105 is configured to determine the sorting of each text block in the text block group to be sorted based on the coherence.

[0212] According to the technical solution provided in the embodiment of the present application, by obtaining the word frequency at the joint between each text block and other text blocks in the text block group to be sorted, and determining the text block similarity between each text block and other text blocks, the word frequency at the joint and the text block similarity are merged to obtain the coherence between each text block and other text blocks, and then the sorting of each text block in the text block group to be sorted is determined based on the coherence, thereby achieving automatic sorting of text blocks with relatively low computational complexity, reducing the sorting cost on the basis of ensuring the sorting accuracy, increasing the sorting speed, and improving the user experience.

[0213] In some implementations, any two text blocks in the text block group to be sorted are spliced ​​together to obtain the word frequency at the splicing point, including: performing word segmentation processing and removing stop words on each text block in the text block group to be sorted; counting the number of occurrences of each word after word segmentation in the target text, the target text being the text where the text block group to be sorted is located; any two text blocks in the text block group to be sorted are spliced ​​together, and determining the number of occurrences of the words at the splicing point in the target text as the word frequency at the splicing point.

[0214] In some embodiments, determining the text block similarity of any two text blocks in the text block group to be sorted includes: determining the literal similarity of a first target text block and a second target text block in the text block group to be sorted, the literal similarity being calculated based on a word frequency-inverse document frequency algorithm; determining the semantic similarity of the first target text block and the second target text block, the semantic similarity being calculated based on a coding model; performing weighted averaging processing on the literal similarity and the semantic similarity to obtain the text block similarity of the first target text block and the second target text block; wherein the first target text block and the second target text block are any two text blocks in the text block group to be sorted.

[0215] In some implementations, the word frequency at the joint and the text block similarity are merged to obtain the coherence of any two text blocks, including: obtaining the target joint word frequency of the first target text block and the second target text block, and the target text block similarity; taking the weighted sum of the normalized target joint word frequency and the normalized target text block similarity to obtain the coherence of the first target text block and the second target text block; wherein the first target text block and the second target text block are any two text blocks in the text block group to be sorted.

[0216] In some implementations, the word frequency at the joint and the text block similarity are merged to obtain the coherence of any two text blocks, including: obtaining a preset rule base, the preset rule base includes at least one preset rule, and each preset rule includes a rule condition and a rule score; in response to determining that the word frequency at the joint and the text block similarity of the first target text block and the second target text block meet the rule condition of the target preset rule, setting the coherence of the first target text block and the second target text block to a first value, otherwise setting the coherence of the first target text block and the second target text block to a second value; wherein the first target text block and the second target text block are any two text blocks in the text block group to be sorted, the target preset rule is any one of the preset rules in the preset rule base, and the first value is greater than the second value.

[0217] In some implementations, determining the order of each text block in the text block group to be sorted based on coherence includes: constructing a coherence matrix based on the coherence between every two text blocks, and when the number of text blocks is less than or equal to a quantity threshold, using a dynamic programming method to find the optimal text block order.

[0218] In some implementations, determining the order of each text block in the text block group to be ordered based on coherence includes: constructing a coherence matrix based on the coherence between every two text blocks, and when the number of text blocks is greater than a quantity threshold, using a greedy algorithm to find the best text block order.

[0219] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0220] Fig.12 Schematic diagram of an electronic device provided in an embodiment of the present application. Fig.12 As shown, the electronic device 12 of this embodiment includes: a processor 1201, a memory 1202, and a computer program 1203 stored in the memory 1202 and executable on the processor 1201. When the processor 1201 executes the computer program 1203, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 1201 executes the computer program 1203, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0221] The electronic device 12 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 12 may include, but is not limited to, a processor 1201 and a memory 1202. Those skilled in the art will appreciate that Fig.12 The electronic device 12 is merely an example and does not limit the electronic device 12 , and may include more or fewer components than those shown in the figure, or different components.

[0222] The processor 1201 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0223] The memory 1202 may be an internal storage unit of the electronic device 12, for example, a hard disk or memory of the electronic device 12. The memory 1202 may also be an external storage device of the electronic device 12, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 12. The memory 1202 may also include both an internal storage unit of the electronic device 12 and an external storage device. The memory 1202 is used to store computer programs and other programs and data required by the electronic device.

[0224] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0225] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0226] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A text block sorting method, characterized in that: include: Get the text block group to be sorted; Splicing any two text blocks in the text block group to be sorted to obtain the word frequency at the splicing point, wherein the word frequency at the splicing point is the number of times the words at the splicing point of the any two text blocks appear in the target text, and the target text is the text where the text block group to be sorted is located; Determining the text block similarity between any two text blocks in the group of text blocks to be sorted; The word frequency at the joint and the similarity of the text blocks are merged to obtain the coherence of the two arbitrary text blocks; Determining the order of each text block in the group of text blocks to be ordered based on the coherence; The step of determining the order of each text block in the group of text blocks to be ordered based on the coherence comprises: A coherence matrix is ​​constructed based on the coherence between every two text blocks, and when the number of the text blocks is less than or equal to a number threshold, a dynamic programming method is used to find an optimal text block sequence.

2. The method according to claim 1, characterized in that The step of splicing any two text blocks in the to-be-sorted text block group to obtain the word frequency at the splicing position includes: Performing word segmentation and stop word removal processing on each text block in the group of text blocks to be sorted; Count the number of occurrences of each word after word segmentation in the target text; Any two text blocks in the to-be-sorted text block group are spliced ​​together, and the number of occurrences of words at the splicing point in the target text is determined as the word frequency at the splicing point.

3. The method according to claim 1, characterized in that The determining of the text block similarity between any two text blocks in the text block group to be sorted includes: Determining the literal similarity between the first target text block and the second target text block in the group of text blocks to be sorted, wherein the literal similarity is calculated based on a term frequency-inverse document frequency algorithm; Determining semantic similarity between the first target text block and the second target text block, wherein the semantic similarity is calculated based on a coding model; Performing weighted average processing on the literal similarity and the semantic similarity to obtain text block similarity between the first target text block and the second target text block; The first target text block and the second target text block are any two text blocks in the group of text blocks to be sorted.

4. The method according to claim 1, characterized in that The step of fusing the word frequency at the joint and the similarity of the text blocks to obtain the coherence of the two arbitrary text blocks includes: Obtaining the word frequency at the target joint of the first target text block and the second target text block, and the similarity of the target text blocks; Taking a weighted sum of the normalized word frequency at the target joint and the normalized similarity of the target text block, to obtain the coherence of the first target text block and the second target text block; The first target text block and the second target text block are any two text blocks in the group of text blocks to be sorted.

5. The method according to claim 1, characterized in that The step of fusing the word frequency at the joint and the similarity of the text blocks to obtain the coherence of the two arbitrary text blocks includes: Acquire a preset rule base, wherein the preset rule base includes at least one preset rule, and each preset rule includes a rule condition and a rule score; In response to determining that the word frequency and text block similarity at the joint of the first target text block and the second target text block meet the rule condition of the target preset rule, the coherence of the first target text block and the second target text block is set to a first value, otherwise the coherence of the first target text block and the second target text block is set to a second value; Among them, the first target text block and the second target text block are any two text blocks in the text block group to be sorted, the target preset rule is any preset rule in the preset rule base, and the first value is greater than the second value.

6. The method according to claim 1, characterized in that The step of determining the order of each text block in the group of text blocks to be ordered based on the coherence comprises: A coherence matrix is ​​constructed based on the coherence between every two text blocks, and when the number of the text blocks is greater than a number threshold, a greedy algorithm is used to find the best text block sequence.

7. A text block sorting device, characterized in that: include: An acquisition module is configured to acquire a group of text blocks to be sorted; A word frequency determination module is configured to splice any two text blocks in the to-be-sorted text block group to obtain a word frequency at the splicing point, wherein the word frequency at the splicing point is the number of times a word at the splicing point of the any two text blocks appears in a target text, wherein the target text is the text in which the to-be-sorted text block group is located; A similarity determination module is configured to determine the text block similarity between any two text blocks in the text block group to be sorted; A coherence determination module is configured to merge the word frequencies at the joints and the similarities of the text blocks to obtain the coherence of the two arbitrary text blocks; A sorting module, configured to determine the sorting of each text block in the group of text blocks to be sorted based on the coherence; The step of determining the order of each text block in the group of text blocks to be ordered based on the coherence comprises: A coherence matrix is ​​constructed based on the coherence between every two text blocks, and when the number of the text blocks is less than or equal to a number threshold, a dynamic programming method is used to find an optimal text block sequence.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images