Text processing method and device, model training method and device, equipment and storage medium

Abstract

The application provides a text processing method and device, a model training method and device, and a storage medium, and relates to the technical field of neural networks. The text processing model is trained by using a training sample text added with a separation mark. Since the training sample text is labeled with label information and position information of the separation mark, the label information of the separation mark indicates whether the text at the position of the separation mark needs to be merged, and the label information of the separation mark is generated according to the real semantics of the text at the position of the separation mark in the training sample text, and the accuracy of the label is high. Therefore, based on the label information and position information of the separation mark labeled by the training sample text, the text processing model obtained by training can be used for accurate merging processing of a target processing text. The training sample text can be obtained by concatenating multiple lines of text, so that the text processing model obtained by training can be applied to the merging processing of multiple lines of text, and the efficiency of the merging processing of multiple lines of text is improved.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and more specifically, to a text processing method, a model training method, an apparatus, a device, and a storage medium. Background Technology

[0002] In the process of text processing, we often encounter the problem of merging complex tables, especially for text that spans multiple pages and text that does not extend beyond the page. It is not possible to simply determine the merging relationship of the text from information such as the spacing between lines and indentation.

[0003] Existing technologies typically treat whether two lines of text should be merged as a binary classification task. For multiple lines of text, each line is compared pairwise to determine if it should be merged, thus obtaining the final merge result. Therefore, this method is inefficient for text merging. Summary of the Invention

[0004] The purpose of this application is to address the shortcomings of the prior art by providing a text processing method, model training method, apparatus, device, and storage medium to solve the problem of low efficiency in text merging processing in the prior art.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0006] In a first aspect, embodiments of this application provide a text processing method, including:

[0007] Read the text from at least one cell in the file to be processed;

[0008] Add delimiters to the text in at least one cell to obtain the target processed text;

[0009] The target text is input into a pre-trained text processing model. The model identifies whether the text segmented by each delimiter in the target text needs to be merged. Based on the identification results, the target text is merged to obtain at least one target text. The text processing model is trained using training sample text with annotation information. The annotation information includes the label information of the delimiters added to the training sample text and the position of the delimiters. The label information is generated based on the true semantics of the text at the position of the delimiters in the training sample text.

[0010] Optionally, adding delimiters to the text of the at least one cell to obtain the target processed text includes:

[0011] Add separators between the text in adjacent cells to obtain the target text for processing.

[0012] Optionally, adding delimiters to the text of the at least one cell to obtain the target processed text includes:

[0013] Add separators between the text in adjacent cells, and then add separators between the text within each cell to obtain the target text.

[0014] Optionally, adding separators between the text within each cell includes:

[0015] Insert separator marks at at least one random position within the text of each cell to obtain the target processed text.

[0016] Optionally, adding separators between the text within each cell includes:

[0017] Perform word segmentation on the text within the cell to obtain the word segmentation result;

[0018] Based on the word segmentation results, at least one complete word is determined in the text within the cell;

[0019] Determine at least one target word from the at least one complete word;

[0020] Add separator markers within each of the target words.

[0021] Secondly, embodiments of this application provide a text processing model training method, including:

[0022] Multiple initial sample texts are collected and preprocessed to obtain a first training text set. The first training text set includes multiple first training sample texts, each of which has annotation information. The annotation information includes: label information of the delimiter added to the first training sample text and the position of the delimiter. The label information is generated based on the real semantics of the text at the position of the delimiter in the first training sample text.

[0023] The first sample text set is used to train and obtain a text processing model.

[0024] Optionally, the step of collecting multiple first initial sample texts and preprocessing the first initial sample texts to obtain a first sample training text set includes:

[0025] Extract multiple first initial sample texts from at least one sample file having a preset format, each first initial sample text including the text of at least one cell in the sample file;

[0026] Denoising is performed on each initial sample text, and non-text characters are removed from each initial sample text to obtain the first preprocessed sample text corresponding to each initial sample text.

[0027] Add delimiters to the text of at least one cell in the first preprocessed sample text to obtain the first training sample text;

[0028] The first training text set is obtained based on each first training sample text.

[0029] Optionally, the text processing model is trained using the first sample training text set, including:

[0030] Obtain the second sample training text set corresponding to the target domain. The annotation information of each second training sample text in the second sample training text set is annotated by the user.

[0031] The text processing model is trained using the first training sample text set and the second training sample text set.

[0032] Optionally, the step of extracting multiple first initial sample texts from at least one sample file having a preset format includes:

[0033] Extract a whole column of cell text sequentially from a wired table in at least one sample file with a preset format, and concatenate the texts sequentially to form a first initial sample text.

[0034] Optionally, the step of denoising each first initial sample text and deleting non-text characters from each first initial sample text to obtain the first preprocessed sample text corresponding to each first initial sample text includes:

[0035] The first initial sample text is converted from full-width to half-width characters, and non-text characters in the first initial sample text are deleted to obtain the first pre-processed sample text corresponding to the first initial sample text. The non-text characters include: preset delimiters, spaces, hypertext markup language tags, and Chinese garbled characters.

[0036] Optionally, adding delimiters to the text of at least one cell in the first preprocessed sample text to obtain the first training sample text includes:

[0037] If the character length of the first preprocessed sample text after inserting delimiters meets the preset length, or the number of inserted delimiters meets the preset number, then the text after the preset length is deleted from the first preprocessed sample text to obtain the first training sample text.

[0038] Thirdly, embodiments of this application also provide a text processing apparatus, including: a reading module, a marking module, and a processing module;

[0039] The reading module is used to read the text of at least one cell in the file to be processed;

[0040] The marking module is used to add separator marks to the text of at least one cell to obtain the target processed text;

[0041] The processing module is used to input the target text into a pre-trained text processing model, identify whether the text segmented by each delimiter in the target text needs to be merged, and perform merging processing on the target text according to the identification result to obtain at least one target text; the text processing model is trained using training sample text with annotation information, the annotation information including the labels of the delimiters added to the training sample text and the position of the delimiters.

[0042] Optionally, the marking module is specifically used to add separator marks between the text of adjacent cells to obtain the target processed text.

[0043] Optionally, the marking module is specifically used to add separator marks between the text in adjacent cells and to add separator marks between the text in each cell to obtain the target processed text.

[0044] Optionally, the marking module is specifically used to insert a separator mark at at least one random position in the text within each cell to obtain the target processed text.

[0045] Optionally, the tagging module is specifically used to perform word segmentation on the text within the cell to obtain the word segmentation result;

[0046] Based on the word segmentation results, at least one complete word is determined in the text within the cell;

[0047] Determine at least one target word from the at least one complete word;

[0048] Add separator markers within each of the target words.

[0049] Fourthly, embodiments of this application also provide a text processing model training device, including: a data acquisition module and a training module;

[0050] The acquisition module is used to acquire multiple first initial sample texts and preprocess the first initial sample texts to obtain a first sample training text set. The first sample training text set includes multiple first training sample texts, and each first training sample text has annotation information. The annotation information includes: label information of the separator markers added to the first training sample texts and the position of the separator markers. The label information is generated based on the real semantics of the text at the position of the separator markers in the training sample texts.

[0051] The training module is used to train and obtain a text processing model using the first sample training text set.

[0052] Optionally, the acquisition module is specifically used for

[0053] Extract multiple first initial sample texts from at least one sample file having a preset format, each first initial sample text including the text of at least one cell in the sample file;

[0054] Denoising is performed on each initial sample text, and non-text characters are removed from each initial sample text to obtain the first preprocessed sample text corresponding to each initial sample text.

[0055] Add delimiters to the text of at least one cell in the first preprocessed sample text to obtain the first training sample text;

[0056] The first training text set is obtained based on each first training sample text.

[0057] Optionally, the training module is specifically used for

[0058] Obtain the second sample training text set corresponding to the target domain. The annotation information of each second training sample text in the second sample training text set is annotated by the user.

[0059] The text processing model is trained using the first training sample text set and the second training sample text set.

[0060] Optionally, the acquisition module is specifically used for

[0061] Extract a whole column of cell text sequentially from a wired table in at least one sample file with a preset format, and concatenate the texts sequentially to form a first initial sample text.

[0062] Optionally, the acquisition module is specifically used for

[0063] The first initial sample text is converted from full-width to half-width characters, and non-text characters in the first initial sample text are deleted to obtain the first pre-processed sample text corresponding to the first initial sample text. The non-text characters include: preset delimiters, spaces, hypertext markup language tags, and Chinese garbled characters.

[0064] Optionally, the acquisition module is specifically used for

[0065] If the character length of the first preprocessed sample text after inserting delimiters meets the preset length, or the number of inserted delimiters meets the preset number, then the text after the preset length is deleted from the first preprocessed sample text to obtain the first training sample text.

[0066] Fifthly, embodiments of this application provide an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the methods provided in the first or second aspect.

[0067] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the methods provided in the first or second aspect.

[0068] The beneficial effects of this application are:

[0069] This application provides a text processing method, model training method, apparatus, device, and storage medium. A text processing model is obtained by training sample text with added delimiters. Since the training sample text is labeled with delimiter tag information and position information, the delimiter tag information indicates whether the text at the delimiter's position needs to be merged. This delimiter tag information is generated based on the true semantics of the text at the delimiter's position in the training sample text, resulting in high accuracy. Therefore, based on the delimiter tag information and position information of the training sample text, the trained text processing model can be used to accurately merge target text. The training sample text can be obtained by concatenating multiple lines of text, thus the trained text processing model is applicable to merging multi-line text, improving the efficiency of multi-line text merging.

[0070] In addition, manually labeled samples are used for the target domain to obtain the corresponding sample training text set, which expands the sample training text set under the automatic labeling method. While maintaining the universality of the samples and improving the accuracy of specific scenarios, it also saves labeling costs and further improves the generalization of the trained text processing model. Attached Figure Description

[0071] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 Flowchart of the text processing method provided in the embodiments of this application Figure 1 ;

[0073] Figure 2 Flowchart of the text processing method provided in the embodiments of this application Figure 2 ;

[0074] Figure 3 A flowchart illustrating the text processing model training method provided in this application embodiment. Figure 1 ;

[0075] Figure 4 A flowchart illustrating the text processing model training method provided in this application embodiment. Figure 2 ;

[0076] Figure 5 A flowchart illustrating the text processing model training method provided in this application embodiment. Figure 3 ;

[0077] Figure 6 A network diagram illustrating a text processing model provided in an embodiment of this application;

[0078] Figure 7 A schematic diagram of a text processing device provided in an embodiment of this application;

[0079] Figure 8 A schematic diagram of a text processing model training device provided in an embodiment of this application;

[0080] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0082] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0083] To enable those skilled in the art to utilize the content of this application, and in conjunction with the specific application scenario of "merging table text in PDF files," the following implementation methods are provided. Those skilled in the art can apply the general principles defined herein to other embodiments and application scenarios without departing from the spirit and scope of this application.

[0084] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0085] PDF (Portable Document Format) is widely used for storing and transmitting various files, and it is often necessary to extract information from PDF documents. However, extracting information from PDFs often involves complex table merging issues that require semantic analysis, such as infinite tables and multi-page tables. Infinite tables often contain multiple lines of text, which cannot be easily determined from information such as line spacing and indentation, but are easy to determine semantically. In contrast, whether a multi-page table needs to be merged can be determined by the semantic relationships between the cell text.

[0086] Based on this, when using semantic judgment to solve the table merging problem, a text processing model can be trained by inserting delimiters into the obtained sample text. Whether the text at the position where the delimiter is inserted needs to be merged is determined by the actual semantic information of the sample text. Therefore, the text processing model trained with the sample text with the delimiter inserted can be used to make an accurate judgment on whether any multiple lines of text extracted from the table need to be merged.

[0087] Existing methods for handling text merging problems using trained models typically treat merging pairs of text within a multi-line text as a binary classification task, requiring multiple pairwise comparisons for multi-line text. For example, assuming four lines of text, pairwise comparisons between lines 1 and 2, 2 and 3, and 3 and 4 are needed, with lines 2 and 3 each input into the model twice. This significantly increases the prediction time for models sensitive to sequence length. Furthermore, text merging requires large amounts of low-cost, automatically labeled corpora, and methods for generating corpora for pairwise text merging are unsuitable for multi-text merging.

[0088] Based on this, this application proposes a method for merging multi-line text and a text processing model training method. By improving the training samples, multi-line text is concatenated to obtain training samples, which are then automatically labeled to train a text processing model suitable for merging multi-line text. Compared to binary classification, the number of times training samples are input into the model is reduced, and the total length of the training samples is smaller, resulting in higher training efficiency and improved efficiency when the model infers the merging results.

[0089] The relevant steps of this method will be explained below through specific embodiments.

[0090] Figure 1 Flowchart of the text processing method provided in the embodiments of this application Figure 1 The execution subject of this method can be a computer device or a server. For example... Figure 1 As shown, the method may include:

[0091] S101. Read the text of at least one cell in the file to be processed.

[0092] This method is applicable to files of any format, not limited to PDF files, HTML (HyperText Markup Language) files, and other rich text formats. When the file to be processed contains multiple lines of text, errors such as accidental line breaks during the writing process may cause text that originally belonged to the same paragraph to be mistakenly divided into multiple paragraphs, resulting in unsmooth text reading and errors in text information extraction. Therefore, text merging can be used to merge the text that needs to be merged, while keeping the text that does not need to be merged independent, so as to finally obtain the complete text.

[0093] In this embodiment, the file to be processed is a PDF file, and the PDF file contains table text to illustrate this method. In actual applications, the file to be processed can be other files, and the file may not contain table text.

[0094] The text of at least one cell in the file to be processed can be read through document autonomous scanning or document image recognition. At least one cell may include adjacent cells or non-adjacent cells.

[0095] S102. Add delimiters to the text in at least one cell to obtain the target text for processing.

[0096] Adding a separator to the text in at least one cell can include adding a separator to the text between two cells or adding a separator to the text within a single cell.

[0097] In some embodiments, a separator mark may be added to each cell of text read until all the text in all cells to be read has been read, at which point the process ends and the target text is obtained.

[0098] In other embodiments, the text of all cells to be read can be read first and the text of each cell can be concatenated together in the order of reading. Then, a separator mark can be added to the concatenated text to obtain the target text for processing.

[0099] S103. Input the target text into the pre-trained text processing model, identify whether the text segmented by each delimiter in the target text needs to be merged, and merge the target text according to the identification result to obtain at least one target text; the text processing model is trained using training sample text with annotation information, the annotation information includes the label information of the delimiter added to the training sample text, and the position of the delimiter, the label information is generated based on the real semantics of the text at the position of the delimiter in the training sample text.

[0100] Optionally, a pre-trained text processing model can be used to identify the input target text, and the target text can be merged by identifying the text merging method indicated by each delimiter added in the target text.

[0101] The text merging method indicated by the separator mark can include: text to be merged or text not to be merged. When the separator mark indicates that the text needs to be merged, the text at the position where the separator mark is added is merged. When the separator mark indicates that the text does not need to be merged, the text at the position where the separator mark is added is not merged. In the case of no merging, the text before the separator mark is treated as one text, and the text after the separator mark is treated as another text. Thus, at least one target text is obtained after merging the target text.

[0102] Optionally, segmentation tags in the training sample text can be added manually based on the semantic information of the text, or they can be automatically added based on the actual semantic information after semantic recognition of the training sample text. Since different people may have biases in their semantic analysis of the same text, when adding segments manually, the final result can be obtained by combining the segmentation tag addition results from multiple people for the same training sample text.

[0103] Since the segmentation markers in the training sample text are added based on the semantic information of the text, and the semantic information of the text can clearly determine whether the text needs to be segmented, the accuracy of the segmentation markers added to the training sample text is high, resulting in high accuracy of the label information of the segmentation markers.

[0104] Since the text processing model is trained on training sample text with added delimiters, and the training sample text is also labeled with the label information and position information of the delimiters, the label information of the delimiters can indicate whether the text at the position of the delimiter needs to be merged. The label information of the delimiters is generated based on the true semantics of the text at the position of the delimiters in the training sample text, and its label accuracy is high. Therefore, based on the label information and position information of the delimiters in the training sample text, the trained text processing model can be used to perform accurate merging processing on the target text.

[0105] In summary, the text processing method provided in this embodiment obtains a text processing model by training training sample text with added delimiters. Since the training sample text is labeled with delimiter tag information and position information, the delimiter tag information indicates whether the text at the delimiter's location needs to be merged. This delimiter tag information is generated based on the true semantics of the text at the delimiter's location in the training sample text, resulting in high accuracy. Therefore, based on the delimiter tag information and position information of the training sample text, the trained text processing model can be used to perform precise merging processing on the target text. Furthermore, the training sample text can be obtained by concatenating multiple lines of text, thus the trained text processing model is applicable to merging multi-line text, improving the efficiency of multi-line text merging processing.

[0106] Optionally, in step S102, adding separator marks to the text of at least one cell to obtain the target processed text may include: adding separator marks between the text of each adjacent cell to obtain the target processed text.

[0107] In one feasible approach, separators can be added only between the text in adjacent cells, while no separators are added between the text within a single cell. Here, adding separators between the text in adjacent cells can also be understood as adding separators between the text in independent cells.

[0108] For example, if we extract the text from three cells: Cell 1 - The weather is nice today; Cell 2 - Where are you going to play?; Cell 3 - Let's go together. Then, after adding separators between the text in each adjacent cell, the resulting target text could be: The weather is nice today / Where are you going to play? / Let's go together.

[0109] It should be noted that in this embodiment, " / " is used as the separator, but in actual applications, it can also be [sep] to indicate the end of a sentence, or any other arbitrary symbol.

[0110] Optionally, in step S102, adding separator marks to the text of at least one cell to obtain the target processed text may include: adding separator marks between the text of each adjacent cell and adding separator marks between the text within each cell to obtain the target processed text.

[0111] In another embodiment, separators can be added between the text in adjacent cells as well as between the text within each cell.

[0112] Optionally, in the above steps, adding a separation marker between the texts in each cell may include: inserting a separation marker at at least one random position in the text in each cell to obtain a target processed text.

[0113] Among them, there are two ways to add a separation marker between the texts in each cell. In one way, a separation marker can be randomly added between the texts in the cell.

[0114] For example: The texts of 3 cells are extracted respectively: Cell 1 - The weather is nice today; Cell 2 - Where are you going to play; Cell 3 - Let's go together. Then, the target processed text obtained after adding a separation marker between the texts of each adjacent cell and inserting a separation marker at at least one random position in the text of each cell can be: today / weather is nice / where are you going / to play / let's go / together.

[0115] Figure 2 Flow diagram of the text processing method provided by the embodiments of this application Figure 2 ; Optionally, in the above steps, adding a separation marker between the texts in each cell may include:

[0116] S201. Perform word segmentation processing on the text in the cell to obtain a word segmentation processing result.

[0117] In addition to the way of randomly adding a separation marker, it can also be to perform word segmentation processing on the text in a single cell to obtain a word segmentation result. Here, the word segmentation processing can be to extract the complete words in the text.

[0118] S202. Determine at least one complete word in the text in the cell according to the word segmentation processing result.

[0119] Through the above word segmentation processing, the text in the cell can be divided into multiple words, some words may be a single character, and some may be a complete word. In this embodiment, the complete words are determined.

[0120] Continuing with the above example, assume that the text in Cell No. 1 is subjected to word segmentation processing, and the obtained word segmentation processing result may include: today; weather; nice. Among them, the three words obtained by word segmentation all belong to complete words.

[0121] Assume that the text in Cell No. 2 is subjected to word segmentation processing, and the obtained word segmentation processing result may include: you are going to; where; play; ah. Then, the complete word among them is where.

[0122] S203. Determine at least one target word from at least one complete word.

[0123] Optionally, one or more of the complete words obtained from word segmentation can be selected as target words.

[0124] S204. Add separator marks within each target word.

[0125] Add separator marks within the identified target words. Separator marks can be added between target words. If the target word is "where", then after adding separator marks, it will be "where / in".

[0126] Of course, the above are just some optional ways to insert separators; in actual applications, there may be other ways to add them.

[0127] In one implementation, when there are multiple target texts, in order to avoid confusion when different target texts are input into the text processing model, special markers can be inserted at the beginning and end of the target texts to distinguish them.

[0128] For example, if the target text is: Today / The weather is nice / Where are you going / Play / Let's go together ...

[0129] The above describes the relevant steps in applying the text processing model. Next, we will explain the training method of the text processing model.

[0130] The above embodiments illustrate the application process of the text processing model, applying it to a text merging scenario to accurately merge cell text in a file. The training process of the text processing model will be explained next through specific embodiments.

[0131] Figure 3 A flowchart illustrating the text processing model training method provided in this application embodiment. Figure 1 Optionally, this method may also include:

[0132] S301. Collect multiple first initial sample texts and preprocess the first initial sample texts to obtain a first sample training text set. The first sample training text set includes multiple first training sample texts. Each first training sample text has annotation information, including: label information of the delimiter added to the first training sample text and the position of the delimiter. The label information is generated based on the real semantics of the text at the position of the delimiter in the first training sample text.

[0133] Similar to the application process of text processing models, the first initial sample text collected during the training process of text processing models can also consist of multiple lines of text, so that the trained text processing model can accurately merge multiple lines of text.

[0134] In some embodiments, the acquired first initial sample text may be preprocessed to obtain the first sample training text. Preprocessing may include noise reduction to make the format of the first initial sample text more standardized, and may also include special marker removal to avoid interfering with subsequently added delimiters.

[0135] Optionally, the first sample training text can be automatically labeled. The labeling process is similar to the application process. It can be that separator marks are added to the first sample training text, so that the first sample training text has labeled information. The model can be trained based on the labeled information to determine whether the texts are merged.

[0136] The annotation information includes: the label of the delimiter added to the first training sample text, and the position of the delimiter. The label of the delimiter includes 0 and 1, where 0 indicates that the text at the position of the delimiter does not need to be merged, and 1 indicates that the text at the position of the delimiter needs to be merged. The position of the delimiter refers to the character position of the delimiter in the first training sample text.

[0137] For example, if the first sample training text is: "Today / The weather is nice / Where are you going / Playing?", then the position of the first separator marker is 3, the position of the second separator marker is 8, and the position of the third separator marker is 14.

[0138] S302. Use the first sample training text set to train and obtain the text processing model.

[0139] Based on the first sample training text with the aforementioned labeled information, a text processing model can be trained. It is worth noting that this refers to a single first sample training text. However, training the text processing model requires acquiring a large number of first sample training texts to form a first sample training text set, which is then used to train the text processing model.

[0140] Figure 4 A flowchart illustrating the text processing model training method provided in this application embodiment. Figure 2 Optionally, in step S301, collecting multiple first initial sample texts and preprocessing the first initial sample texts to obtain a first sample training text set may include:

[0141] S401. Extract multiple first initial sample texts from at least one sample file with a preset format, each first initial sample text including the text of at least one cell in the sample file.

[0142] The default format here can be PDF, and the PDF file must contain tables. However, in practice, it's not limited to PDF. Multiple initial sample texts can be extracted from different PDF sample files.

[0143] Optionally, the extracted first initial sample text may be text containing at least one cell in a sample file; that is, the first initial sample text may be composed of text extracted from at least one cell in a sample file.

[0144] S402. Denoise each first initial sample text and delete non-text characters in each first initial sample text to obtain the first preprocessed sample text corresponding to each first initial sample text.

[0145] The purpose of the noise reduction process here is to make the initial sample text format more standardized, which is easier for the model to learn. The purpose of removing non-text characters is to remove the interference of special characters on the semantics of the text; the specific processing methods can be found below.

[0146] S403. Add a separator mark to the text of at least one cell in the first preprocessed sample text to obtain the first training sample text.

[0147] Optionally, the method used to add delimiters to at least one cell of the preprocessed sample text is the same as that used in step S102 above to add delimiters to at least one cell of the text to be processed. This will not be elaborated upon here.

[0148] S404. Based on each first training sample text, obtain the first sample training text set.

[0149] The above describes the method for obtaining a first training sample text, and the multiple first training sample texts obtained will constitute the first sample training text set.

[0150] Figure 5 A flowchart illustrating the text processing model training method provided in this application embodiment. Figure 3 Optionally, in step S302, training the text processing model using the first sample training text set may include:

[0151] S501. Obtain the second sample training text set corresponding to the target domain. The annotation information of each second training sample text in the second sample training text set is annotated by the user.

[0152] In some embodiments, for merging texts in certain specific domains (target domains), since training sample texts are relatively scarce, a second sample training text set can be extracted from the specific domain. Unlike the first sample training text set, the annotation information of each second training sample text in the second sample training text set is manually annotated instead of using the automatic annotation method described above. The manual annotation method is used to improve the accuracy of the annotation of samples in the specific domain.

[0153] S502. Using the first training sample text set and the second training sample text set, train and obtain the text processing model.

[0154] Optionally, the first training text set and the second training text set can both be used as training samples to train the text processing model.

[0155] Optionally, in step S401, extracting multiple first initial sample texts from at least one sample file with a preset format may include: extracting a whole column of cell text from a wired table in at least one sample file with a preset format in a column-wise order, and concatenating each text in sequence as a first initial sample text.

[0156] Assuming the table in the sample file has three rows and four columns, then for the first column, we can extract the cell text sequentially starting from the first row of cells, and concatenate the extracted text to form a first initial sample text. Similarly, the second, third, and fourth columns can each be extracted to form a first initial sample text.

[0157] Of course, you can also extract a whole row of cell text from a wired table row by row; there are no restrictions on the specific extraction method.

[0158] Optionally, in step S402, denoising each first initial sample text and deleting non-text characters in each first initial sample text to obtain the first preprocessed sample text corresponding to each first initial sample text may include: converting the first initial sample text from full-width to half-width characters and deleting non-text characters in the first initial sample text to obtain the first preprocessed sample text corresponding to the first initial sample text. Non-text characters include: preset separators, spaces, hypertext markup language tags, and Chinese garbled characters.

[0159] It's worth noting that full-width characters refer to characters that occupy two character positions in the text, while half-width characters refer to characters that occupy one character position. Characters occupying two character positions in the initial sample text can be converted to characters occupying one character position, ensuring that each character in the initial sample text occupies one character position, thus facilitating the determination of the delimiter's position within the text.

[0160] In addition, non-text characters in the first initial sample text are deleted. In this embodiment, the non-text characters listed may include preset separators, spaces, Hypertext Markup Language tags, and Chinese garbled characters. The preset separator can be " / ". Since the preset separator is the same as the separator added to the training sample text, it is a type of non-text character that must be deleted. Other characters, including spaces, Hypertext Markup Language tags, and Chinese garbled characters, are not required to be deleted and can be left as is.

[0161] Optionally, in step S403 above, adding separator markers to the text of at least one cell in the first preprocessed sample text to obtain the first training sample text may include: sequentially traversing the text of at least one cell and adding separator markers between the text of adjacent cells.

[0162] Adding separator markers to the text of at least one cell in the first preprocessed sample text to obtain the first training sample text may further include: adding separator markers between the text of adjacent cells; and adding separator markers at at least one random position between the text in each cell.

[0163] Adding separator markers to the text of at least one cell in the first preprocessed sample text to obtain the first training sample text may further include: adding separator markers between the text of adjacent cells; performing word segmentation on the text in each cell to obtain the word segmentation result; determining at least one complete word in the text in each cell based on the word segmentation result; and adding separator markers to the at least one complete word.

[0164] The method for adding delimiters to the training sample text is the same as the method for adding delimiters to the cell text extracted from the file to be processed, as described above.

[0165] Optionally, in step S403, adding delimiters to the text of at least one cell in the first preprocessed sample text to obtain the first training sample text may include: if the character length of the first preprocessed sample text after the insertion of delimiters meets a preset length, or the number of inserted delimiters meets a preset number, then deleting the text after the preset length from the first preprocessed sample text to obtain the first training sample text.

[0166] In some embodiments, constraints are also imposed on the first training sample text to prevent it from becoming too long and exceeding the range that the model can process.

[0167] In one approach, it can be determined whether the length of the text after adding the separator marks has reached a preset length. If it has, the text after the preset length is discarded. The preset length could be, for example, 512.

[0168] Alternatively, you can determine if the number of currently added separators has reached a preset number. If it has, discard the text following the last separator. The preset number could be, for example, 10.

[0169] For example, suppose the first preprocessed sample text is: "The weather is nice today / Where are you going to play / Let's go together", with a total length of 21, while the preset length is 10. Then, if the text after the 11th character is deleted, the first training sample text will be "The weather is nice today / Where are you going?".

[0170] Alternatively, if the first preprocessed sample text is: "The weather is nice today / Where are you going to play / Let's go together", with 3 separators and 2 preset separators, then the first training sample text would be: "The weather is nice today / Where are you going to play / ".

[0171] The training of the text processing model is explained below:

[0172] Figure 6 This is a network diagram illustrating a text processing model provided in an embodiment of this application. Figure 6As shown, the input to the initial text processing model is the first training sample text in the first training sample text set, assuming [cls] includes other comprehensive gains / reclassification of losses / gains / measured at fair value and their changes in [sep]. Simultaneously, the labels of each delimiter are input, assuming [1,0,1], and the positions of the delimiters in the first training sample text [7,16,13]. [cls] includes other comprehensive gains / reclassification of losses / gains / measured at fair value and their changes in [sep], corresponding to input1 (first layer), [7,16,23] to input2 (second layer), and [1,0,1] to input3 (third layer). input1 inputs a 6-layer BERT (Bidirectional Encoder Representations from Transformer) network, taking the prediction results p1, p2, p3 of each delimiter predicted by the output layer of the last layer. Here, p1 refers to the probability that the text indicated by the separator at position 7, predicted by the model, does not need to be merged; p2 refers to the probability that the text indicated by the separator at position 16, predicted by the model, does not need to be merged; and p3 refers to the probability that the text indicated by the separator at position 23, predicted by the model, does not need to be merged. Based on p1, p2, p3, and the true labels of the separators [1,0,1], the cross-entropy loss is calculated and backpropagated to continuously correct the network parameters, and finally the text processing model is trained.

[0173] In summary, the text processing model training method provided in this embodiment obtains the text processing model by training it with training sample text marked with delimiters. Since the training sample text is annotated with delimiter label information and position information, the delimiter label information indicates whether the text at the position of the delimiter needs to be merged. This label information is generated based on the true semantics of the text at the delimiter position in the training sample text, resulting in high label accuracy. Therefore, based on the delimiter label information and position information annotated in the training sample text, the trained text processing model can be used to perform precise merging processing on the target text. Furthermore, the training sample text can be obtained by concatenating multiple lines of text, thus the trained text processing model is applicable to merging multi-line text, improving the efficiency of multi-line text merging processing.

[0174] In addition, manually labeled samples are used for the target domain to obtain the corresponding sample training text set, which expands the sample training text set under the automatic labeling method. While maintaining the universality of the samples and improving the accuracy of specific scenarios, it also saves labeling costs and further improves the generalization of the trained text processing model.

[0175] The following describes the apparatus, device, and storage medium used to execute the text processing method and text processing model training method provided in this application. The specific implementation process and technical effects are described above and will not be repeated below.

[0176] Figure 7 This is a schematic diagram of a text processing device provided in an embodiment of this application. The functions implemented by this text processing device correspond to the steps of text processing described above. This device can be understood as the aforementioned server, or the server's processor, or as a component that implements the functions of this application under the control of the server, independent of the aforementioned server or processor. Figure 7 As shown, the device may include: a reading module 710, a marking module 720, and a processing module 730;

[0177] The reading module 710 is used to read the text of at least one cell in the file to be processed;

[0178] The marker module 720 is used to add separator markers to the text of at least one cell to obtain the target processed text;

[0179] The processing module 730 is used to input the target text into a pre-trained text processing model, identify whether the text segmented by each delimiter in the target text needs to be merged, and merge the target text according to the identification result to obtain at least one target text; the text processing model is trained using training sample text with annotation information, including the labels of the delimiters added to the training sample text and the position of the delimiters.

[0180] Optionally, the marker module 720 is specifically used to add separator markers between the text of adjacent cells to obtain the target processed text.

[0181] Optionally, the marking module 720 is specifically used to add separator marks between the text in adjacent cells and to add separator marks between the text in each cell to obtain the target processed text.

[0182] Optionally, the marking module 720 is specifically used to insert a separator mark at at least one random position in the text within each cell to obtain the target processed text.

[0183] Optionally, the tagging module 720 is specifically used to perform word segmentation on the text within the cell to obtain the word segmentation result;

[0184] Based on the word segmentation results, determine at least one complete word in the text within the cell;

[0185] Identify at least one target word from at least one complete word;

[0186] Add separators within each target word.

[0187] Figure 8 This is a schematic diagram of a text processing model training device provided in an embodiment of this application. The functions implemented by this text processing model training device correspond to the steps executed by the above-described text processing model training method. Figure 8 As shown, the device may include: a data acquisition module 810 and a training module 820;

[0188] The acquisition module 810 is used to acquire multiple first initial sample texts and preprocess the first initial sample texts to obtain a first sample training text set. The first sample training text set includes multiple first training sample texts, and each first training sample text has annotation information, including: the label of the separator added to the first training sample text and the position of the separator.

[0189] Training module 820 is used to train the text processing model using the first sample training text set.

[0190] Optionally, the acquisition module 810 is specifically used for

[0191] Extract multiple first initial sample texts from at least one sample file with a preset format, each first initial sample text including the text of at least one cell in the sample file;

[0192] Denoising is performed on each initial sample text, and non-text characters are removed from each initial sample text to obtain the first preprocessed sample text corresponding to each initial sample text.

[0193] Add delimiters to the text of at least one cell in the first preprocessed sample text to obtain the first training sample text;

[0194] The first training text set is obtained based on each first training sample text.

[0195] Optionally, the training module 820 is specifically used for

[0196] Obtain the second sample training text set corresponding to the target domain. The annotation information of each second training sample text in the second sample training text set is annotated by the user.

[0197] The text processing model is trained using the first training sample text set and the second training sample text set.

[0198] Optionally, the acquisition module 810 is specifically used for

[0199] Extract a whole column of cell text sequentially from a wired table in at least one sample file with a preset format, and concatenate the texts sequentially to form a first initial sample text.

[0200] Optionally, the acquisition module 810 is specifically used for

[0201] The first initial sample text is converted from full-width to half-width characters, and non-text characters in the first initial sample text are deleted to obtain the first pre-processed sample text corresponding to the first initial sample text. Non-text characters include: preset delimiters, spaces, hypertext markup language tags, and Chinese garbled characters.

[0202] Optionally, the acquisition module 810 is specifically used for

[0203] If the character length of the first preprocessed sample text after inserting delimiters meets the preset length, or the number of inserted delimiters meets the preset number, then the text after the preset length is deleted from the first preprocessed sample text to obtain the first training sample text.

[0204] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0205] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0206] The modules described above can be connected or communicate with each other via wired or wireless connections. Wired connections can include metal cables, optical fibers, hybrid cables, or any combination thereof. Wireless connections can include connections via LAN, WAN, Bluetooth, ZigBee, or NFC, or any combination thereof. Two or more modules can be combined into a single module, and any module can be divided into two or more units. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here.

[0207] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The device can be integrated into a terminal device or a chip of a terminal device. The terminal can be a computing device with data processing capabilities.

[0208] The device includes: a processor 801 and a memory 802.

[0209] The memory 802 is used to store programs, and the processor 801 calls the programs stored in the memory 802 to execute the above method embodiments. The specific implementation and technical effects are similar, and will not be described again here.

[0210] The memory 802 stores program code, which, when executed by the processor 801, causes the processor 801 to perform various steps in the methods according to various exemplary embodiments of this application described in the "Exemplary Methods" section above.

[0211] The processor 801 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0212] Memory 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 802 may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0213] Optionally, this application also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, performs the above-described method embodiments.

[0214] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0215] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0216] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0217] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A text processing method, characterized in that, include: Read the text from at least one cell in the file to be processed; Add delimiters to the text in at least one cell to obtain the target processed text; The target text is input into a pre-trained text processing model. The model identifies whether the text segmented by each delimiter in the target text needs to be merged, and merges the target text based on the identification results to obtain at least one target text. The text processing model is trained using training sample text with annotation information, including the label information of the delimiters added to the training sample text and the position of the delimiters. The label information of the delimiters indicates whether the text at the position of the delimiter needs to be merged. The label information is generated based on the true semantics of the text at the position of the delimiters in the training sample text. The training steps of the text processing model are as follows: The training sample text with annotation information is input into an initial text processing model, and the initial text processing model... Based on the training sample text and the positions of the delimiters in the training sample text, prediction results for each delimiter are obtained, and the prediction results are used to indicate the probability that the text at the position of the delimiter does not need to be merged; based on the prediction results of each delimiter and the label information of each delimiter, the loss information of the initial text processing model is calculated, and the network parameters of the initial text processing model are iteratively corrected to obtain the text processing model; wherein, identifying whether the text segmented by each delimiter in the target processing text needs to be merged includes: identifying the text merging method indicated by each delimiter in the target processing text, and determining whether the text segmented by each delimiter needs to be merged based on the text merging method; the text merging method indicated by the delimiter includes: text needs to be merged, text does not need to be merged.

2. The method according to claim 1, characterized in that, Adding delimiters to the text of at least one cell to obtain the target processed text includes: Add separators between the text in adjacent cells to obtain the target text for processing.

3. The method according to claim 1, characterized in that, Adding delimiters to the text of at least one cell to obtain the target processed text includes: Add separators between the text in adjacent cells, and then add separators between the text within each cell to obtain the target text.

4. The method according to claim 3, characterized in that, Adding separators between text within each cell includes: Insert separator marks at at least one random position within the text of each cell to obtain the target processed text.

5. The method according to claim 3, characterized in that, Adding separators between text within each cell includes: Perform word segmentation on the text within the cell to obtain the word segmentation result; Based on the word segmentation results, at least one complete word is determined in the text within the cell; Determine at least one target word from the at least one complete word; Add separator markers within each of the target words.

6. A method for training a text processing model, characterized in that, The method includes: Multiple initial sample texts are collected and preprocessed to obtain a first training text set. The first training text set includes multiple first training sample texts, each with annotation information. The annotation information includes: label information of the separator markers added to the first training sample texts, and the position of the separator markers. The label information of the separator markers is used to indicate whether the text at the position of the separator marker needs to be merged. The label information is generated based on the true semantics of the text at the position of the separator markers in the first training sample texts. Using the first sample training text set, a text processing model is trained and obtained. The training steps of the text processing model are as follows: input each first training sample text with annotation information into the initial text processing model; the initial text processing model predicts the prediction results of each separator in each first training sample text based on the position of each separator in each first training sample text and the position of the separator in each first training sample text. The prediction results are used to indicate the probability that the text at the position of the separator does not need to be merged; calculate the loss information of the initial text processing model based on the prediction results of each separator in each first training sample text and the label information of each separator; iteratively correct the network parameters of the initial text processing model to obtain the text processing model.

7. The method according to claim 6, characterized in that, The process involves collecting multiple initial sample texts and preprocessing them to obtain a first sample training text set, including: Extract multiple first initial sample texts from at least one sample file having a preset format, each first initial sample text including the text of at least one cell in the sample file; Denoising is performed on each initial sample text, and non-text characters are removed from each initial sample text to obtain the first preprocessed sample text corresponding to each initial sample text. Add delimiters to the text of at least one cell in the first preprocessed sample text to obtain the first training sample text; The first training text set is obtained based on each first training sample text.

8. The method according to claim 6, characterized in that, Using the first sample training text set, the text processing model is trained and obtained, including: Obtain the second sample training text set corresponding to the target domain. The annotation information of each second training sample text in the second sample training text set is annotated by the user. The text processing model is trained using the first sample training text set and the second sample training text set.

9. The method according to claim 7, characterized in that, The step of extracting multiple first initial sample texts from at least one sample file having a preset format includes: Extract a whole column of cell text sequentially from a wired table in at least one sample file with a preset format, and concatenate the texts sequentially to form a first initial sample text.

10. The method according to claim 7, characterized in that, The step of denoising each first initial sample text and deleting non-text characters from each first initial sample text to obtain the first preprocessed sample text corresponding to each first initial sample text includes: The first initial sample text is converted from full-width to half-width characters, and non-text characters in the first initial sample text are deleted to obtain the first pre-processed sample text corresponding to the first initial sample text. The non-text characters include: preset delimiters, spaces, hypertext markup language tags, and Chinese garbled characters.

11. The method according to claim 7, characterized in that, The step of adding delimiters to the text of at least one cell in the first preprocessed sample text to obtain the first training sample text includes: If the character length of the first preprocessed sample text after inserting delimiters meets the preset length, or the number of inserted delimiters meets the preset number, then the text after the preset length is deleted from the first preprocessed sample text to obtain the first training sample text.

12. A text processing device, characterized in that, include: Reading module, marking module, processing module; The reading module is used to read the text of at least one cell in the file to be processed; The marking module is used to add separator marks to the text of at least one cell to obtain the target processed text; The processing module is used to input the target text into a pre-trained text processing model, identify whether the text segmented by each delimiter in the target text needs to be merged, and perform merging processing on the target text according to the identification result to obtain at least one target text; the text processing model is trained using training sample text with annotation information, the annotation information including the label information of the delimiters added to the training sample text and the position of the delimiters, the label information of the delimiters is used to indicate whether the text at the position of the delimiter needs to be merged; the label information is generated based on the real semantics of the text at the position of the delimiters in the training sample text; the training steps of the text processing model are as follows: input the training sample text with annotation information into the initial text processing model, and the initial text... The processing model predicts the prediction results of each delimiter based on the training sample text and the positions of the delimiters in the training sample text. The prediction results indicate the probability that the text at the position of the delimiter does not need to be merged. Based on the prediction results of each delimiter and the label information of each delimiter, the loss information of the initial text processing model is calculated, and the network parameters of the initial text processing model are iteratively corrected to obtain the text processing model. Among them, identifying whether the text segmented by each delimiter in the target processing text needs to be merged includes: identifying the text merging method indicated by each delimiter in the target processing text, and determining whether the text segmented by each delimiter needs to be merged based on the text merging method. The text merging method indicated by the delimiter includes: the text needs to be merged and the text does not need to be merged.

13. A text processing model training device, characterized in that, The device includes: a data acquisition module and a training module; The acquisition module is used to acquire multiple first initial sample texts and preprocess the first initial sample texts to obtain a first sample training text set. The first sample training text set includes multiple first training sample texts, each of which has annotation information. The annotation information includes: label information of the separator markers added to the first training sample texts, and the position of the separator markers. The label information of the separator markers is used to indicate whether the text at the position of the separator marker needs to be merged. The label information is generated based on the true semantics of the text at the position of the separator markers in the training sample texts. The training module is used to train a text processing model using the first sample training text set. The training steps of the text processing model are as follows: inputting each first training sample text with annotation information into the initial text processing model, the initial text processing model predicts the prediction results of each separator in each first training sample text based on the position of each separator in the first training sample text and the position of the separator in each first training sample text. The prediction results are used to indicate the probability that the text at the position of the separator does not need to be merged. Based on the prediction results of each separator in each first training sample text and the label information of each separator, the loss information of the initial text processing model is calculated, and the network parameters of the initial text processing model are iteratively corrected to obtain the text processing model.

14. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus. The processor executes the program instructions to perform the steps of the text processing method as described in any one of claims 1 to 5 or the steps of the text processing model training method as described in any one of claims 6 to 11.

15. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, performs the steps of the text processing method as described in any one of claims 1 to 5 or the steps of the text processing model training method as described in any one of claims 6 to 11.

Images