Text processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention discloses a text processing method and device, equipment, a storage medium and a program product, and aims to efficiently and accurately complete text format conversion, easily extract text information and enhance text accessibility. The method comprises the steps that a to-be-processed text with the format type being a first format and a format mapping rule are obtained, and the format mapping rule is obtained by processing a text sample of the first format and a text sample of a second format through a pre-training language model, the format mapping rule is used for representing a mapping relationship between a format grammar of the first format and a format grammar of the second format, and the format read-write difficulty of the first format is greater than that of the second format; extracting text content features of the to-be-processed text; extracting text structure features of the to-be-processed text based on the to-be-processed text and the text content features; and processing the format mapping rule, the text content feature and the text structure feature based on a pre-training language model to generate a target text of which the format type is a second format.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a text processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] Text format conversion refers to changing text from one format to another. Different text formats have different characteristics. Through format conversion, text can be converted into different format types to meet the needs of different application scenarios. For example, text in a format that supports reading can be converted into a format that supports editing.

[0003] In texts with complex structures containing various text elements such as headings and tables, while the text format may support certain functionalities, the complexity of the structure means that simply using text conversion tools that match the text format in existing solutions is insufficient for content extraction and reconstruction. This leads to the inability of current methods to accurately and efficiently convert complex text into other easily editable and maintainable formats, resulting in difficulties in reading and understanding the text and significantly reducing the user experience. Therefore, there is an urgent need to propose an efficient conversion method to transform text from complex formats into easily editable and maintainable formats. Summary of the Invention

[0004] This application provides a method, apparatus, device, storage medium, and program product for text processing, which can efficiently and accurately complete text format conversion, making the converted text easy to edit and maintain, and easy for users to extract text information, thereby enhancing the accessibility of the text.

[0005] In a first aspect, embodiments of this application provide a text processing method. The method includes: acquiring a text to be processed in a first format and a format mapping rule, wherein the format mapping rule is obtained by processing text samples in the first format and text samples in the second format using a pre-trained language model, and the format mapping rule is used to characterize the mapping relationship between the format syntax of the first format and the format syntax of the second format, wherein the format reading and writing difficulty of the first format is greater than that of the second format; extracting text content features of the text to be processed; extracting text structure features of the text to be processed based on the text to be processed and the text content features; and processing the format mapping rule, the text content features, and the text structure features based on the pre-trained language model to generate target text in the second format.

[0006] Secondly, embodiments of this application provide a text processing apparatus. The text processing apparatus includes an acquisition unit, an extraction unit, and a processing unit.

[0007] The acquisition unit is used to acquire the text to be processed in the first format and the format mapping rule. The format mapping rule is obtained by the pre-trained language model processing the text samples in the first format and the text samples in the second format. The format mapping rule is used to represent the mapping relationship between the format syntax of the first format and the format syntax of the second format. The format reading and writing difficulty of the first format is greater than that of the second format.

[0008] The extraction unit is used to extract the text content features of the text to be processed;

[0009] The extraction unit is used to extract the text structure features of the text to be processed based on the text to be processed and the text content features;

[0010] The processing unit is used to process format mapping rules, text content features and text structure features based on a pre-trained language model to generate target text of the second format type.

[0011] In another possible implementation of the embodiments of this application, the processing unit is specifically used for:

[0012] By using a pre-trained language model, text content features are mapped based on format mapping rules to obtain target content that conforms to the second format.

[0013] By using a pre-trained language model, structural mapping is performed on the text structural features based on format mapping rules to obtain the target structure that conforms to the second format.

[0014] Based on the target content and target structure, generate target text in the second format.

[0015] In another possible implementation of an embodiment of this application, the text structure features include text title features and text paragraph features; the processing unit is specifically used for:

[0016] The target title is determined based on text title features and format mapping rules;

[0017] Target paragraphs are determined based on text paragraph features and format mapping rules;

[0018] Based on the target title and target paragraph, a target structure conforming to the second format is obtained.

[0019] In another possible implementation of an embodiment of this application, the processing unit is specifically used for:

[0020] Determine the title content and title level in the text to be processed based on text title features;

[0021] Based on the heading level and format mapping rules, determine the heading level symbol corresponding to the second format of the heading level;

[0022] The target title is determined based on the title content and title level symbols.

[0023] In another possible implementation of an embodiment of this application, the processing unit is specifically used for:

[0024] Identify the paragraph marks of the text paragraphs in the text to be processed based on the characteristics of the text paragraphs;

[0025] Based on the format mapping rules, the paragraph marks of the text paragraphs are changed to the target paragraph marks corresponding to the second format in order to determine the target paragraph.

[0026] In another possible implementation of an embodiment of this application, the text structure features further include text list features and text table features; the processing unit is specifically further used for:

[0027] The target list is determined based on text list features and format mapping rules;

[0028] The target table is determined based on text table features and format mapping rules;

[0029] By using a pre-trained language model, a target structure conforming to the second format is obtained based on the target title, target paragraph, target list, and target table.

[0030] In another possible implementation of an embodiment of this application, the processing unit is specifically used for:

[0031] The list symbols of the text list in the text to be processed are determined based on the features of the text list;

[0032] Based on the format mapping rules, the list symbols of the text list are changed to the target list symbols of the second format to obtain the target list.

[0033] In another possible implementation of an embodiment of this application, the processing unit is specifically used for:

[0034] Determine the table symbols of the text tables in the text to be processed based on the characteristics of the text tables;

[0035] Based on the format mapping rules, the table symbols of the text table are changed to the target table symbols of the second format to obtain the target table.

[0036] In another possible implementation of an embodiment of this application, the processing unit is specifically used for:

[0037] Identify text content from the text to be processed, including text font and multiple text blocks;

[0038] Text font features are obtained by performing text feature recognition processing on the text font.

[0039] Extract the coordinate information of each text block. The coordinate information of each text block is used to represent the position of the corresponding text block in the text to be processed.

[0040] The text blocks are sorted based on their coordinate information to obtain text location features;

[0041] Based on text font features and text position features, the text content features of the text to be processed are obtained.

[0042] In another possible implementation of an embodiment of this application, the processing unit is specifically used for:

[0043] The text title features are determined based on the text font features and the font threshold of the preset title level;

[0044] Paragraph identifiers between text blocks are determined based on text location features to obtain text paragraph features;

[0045] The list identifiers in the text to be processed are identified to obtain the text list features;

[0046] The grid line symbols in the text to be processed are identified to obtain the text table features;

[0047] Based on one or more of the following features: text title features, text paragraph features, text list features, and text table features, the text structure features of the text to be processed are obtained.

[0048] In another possible implementation of an embodiment of this application, the processing unit is further configured to:

[0049] After processing the format mapping rules, text content features, and text structure features based on the pre-trained language model to generate target text of the second format type, the target text is then subjected to second format syntax checking to obtain target format information that does not conform to the second format syntax in the target text.

[0050] The target format information is processed based on the format syntax of the second format to obtain the optimized target text.

[0051] In another possible implementation of the embodiments of this application, the target format information includes one or more of blank lines, indentation, non-compliant characters, and error symbols.

[0052] In another possible implementation of an embodiment of this application, the processing unit is further configured to:

[0053] After processing the target format information based on the format syntax of the second format to obtain the optimized target text, the optimized target text is then checked for text language based on the language detection algorithm to obtain the text language of the optimized target text.

[0054] Based on a preset symbol lookup table and text language, the first symbol in the optimized target text is corrected to obtain target text that conforms to the text language. The first symbol is a text symbol in the optimized target text that does not conform to the text language.

[0055] A third aspect of this application provides a text processing device, including: a memory, an input / output interface, and a processor. The memory stores program instructions. The processor executes the program instructions in the memory to perform the text processing method corresponding to the embodiments of the first aspect described above.

[0056] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method corresponding to the embodiments of the first aspect described above.

[0057] The fifth aspect of this application provides a computer program product containing instructions that, when run on a computer or processor, causes the computer or processor to execute the method described above for performing the implementation method of the first aspect.

[0058] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0059] In this embodiment, the format mapping rule is obtained by processing text samples of the first format and the second format using a pre-trained language model. Furthermore, this format mapping rule can be used to characterize the mapping relationship between the format syntax of the first format and the format syntax of the second format. The first format is more difficult to read and write than the second format. In this application, the text to be processed (format type 1) and the format mapping rule are first obtained. After obtaining the text to be processed, the text content features are extracted. Similarly, after extracting the text content features, the text content features and the text to be processed are considered comprehensively, and then the text structure features of the text to be processed are extracted based on the text to be processed and the text content features. Then, the format mapping rule, text content features, and text structure features are processed based on the pre-trained language model to generate target text of the second format. Through the above methods, this application utilizes a pre-trained language model and format mapping rules to efficiently and accurately convert the text to be processed from a first format, which is more difficult to read and write, to a second format, which is easier to read and write. This not only makes the generated target text easier to edit and maintain and facilitates rapid updates of the text content, but also makes it easier for users to understand the target text, which is easier to read and write, thereby enhancing the accessibility of the text and improving the user's experience in extracting text information. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This application provides a schematic diagram of an application scenario.

[0062] Figure 2 This illustration shows an application scenario where the text processing method provided in this application can be applied.

[0063] Figure 3 A flowchart of a text processing method provided in this application is shown;

[0064] Figure 4 A flowchart illustrating the format mapping rules provided in this application is shown;

[0065] Figure 5 A schematic diagram of the process structure for extracting text content features provided in this application is shown;

[0066] Figure 6This invention provides a schematic diagram of the structure for extracting text structural features.

[0067] Figure 7 This application provides a schematic diagram of the code for extracting text structural features.

[0068] Figure 8 A schematic diagram of the process for generating target text provided in this application is shown;

[0069] Figure 9 This application provides a schematic diagram of the code for generating target text.

[0070] Figure 10 This document illustrates a schematic diagram of the framework process for converting text from PDF format to Markdown format provided in this application.

[0071] Figure 11 A schematic diagram of the functional module structure of the text processing device provided in the embodiments of this application is shown;

[0072] Figure 12 A schematic diagram of the hardware structure of the text processing device provided in the embodiments of this application is shown. Detailed Implementation

[0073] This application provides a method, apparatus, device, storage medium, and program product for text processing, which can efficiently and accurately complete text format conversion, making the converted text easy to edit and maintain, and easy for users to extract text information, thereby enhancing the accessibility of the text.

[0074] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0075] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0076] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that implementations of the application described herein can be implemented, for example, in sequences other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0077] In today's rapidly developing information technology landscape, text processing has become an essential part of daily life and work, and a key force driving innovation across various industries. Different texts may originate from different data sources, resulting in differences in format, encoding, and typesetting standards. For example, given a text whose format only supports function A, if a user wants to extract more relevant text information, using only function A may not meet this requirement. In such cases, the user might want to convert the text from its current format (which only supports function A) to another format that supports other functions to satisfy their needs.

[0078] For example, Figure 1 A schematic diagram illustrating an application scenario provided in this application is shown. For example... Figure 1 As shown, when a user obtains text XXX, they want to edit it to supplement the definition of a certain word P within the text XXX, such as "text format". However, due to the format limitations of text XXX, its text format only supports reading and not editing. Therefore, the user wants to convert the text format of text XXX from a reading-enabled format to an editing-enabled format to generate a new text YYY, where the text format of text YYY supports editing. After generating the new text YYY, the user can then edit the word P within text YYY to input the definition of "text format".

[0079] Converting text from one format to another allows for the transformation of text into different format types to support various text functionalities and meet diverse scenario requirements. However, current technologies typically employ text conversion tools that sample and match text to achieve this conversion. But for complex texts with multiple distinct text elements, current conversion tools are not conducive to content extraction and reconstruction, often resulting in poor conversion efficiency and accuracy. Furthermore, they can create difficulties in extracting text information for reading and comprehension, significantly diminishing the user experience in obtaining text information.

[0080] To address the aforementioned technical problems, this application provides a text processing method. This text processing method can be applied to scenarios such as text format conversion. For example, text format conversion can be applied to document editing, text sharing, content consumption and reception scenarios, etc., and is not specifically limited in this application. In the text processing method of this application, by comprehensively considering both the text content and text structure of the text to be processed, and utilizing a pre-trained language model and format mapping rules, the text to be processed is converted from a first format with higher reading and writing difficulty to a second format with lower reading and writing difficulty. This not only improves the efficiency and accuracy of text format conversion, making the generated target text easier to edit and maintain, and facilitating rapid updates of text content, but also makes it easier for users to understand the target text with lower reading and writing difficulty, enhancing text accessibility and improving the user experience in extracting text information.

[0081] For example, the text processing method provided in this application can be applied to one or more of the following application scenarios, as follows:

[0082] (1) Document editing and sharing scenarios

[0083] In today's era of increasingly frequent digital office work and information exchange, enabling documents to be edited and accessed on different operating systems and devices may require converting the document's text format to a format supported by those operating systems and devices to ensure cross-platform compatibility and fidelity to the original format. By applying the text processing method provided in this application to document editing and sharing scenarios, it comprehensively considers the text content and structure, utilizing a pre-trained language model and format mapping rules to complete the format conversion from a first format (more difficult to read and write) to a second format (easier to read and write). This allows the converted text to be smoothly shared across different operating systems and devices, while also being easy to edit, improving document editing and sharing efficiency and enhancing text accessibility.

[0084] (2) Data Analysis Scenarios

[0085] In data analysis, it may be necessary to convert data from different sources into easily editable and maintainable text to facilitate data extraction and subsequent data mining. By applying the text processing method provided in this application to data analysis scenarios, the text content and structure can be comprehensively considered. A pre-trained language model and format mapping rules can be used to convert the text from a first format (which is more difficult to read and write) to a second format (which is easier to read and write). This makes it much easier for analysts to understand the converted text, thereby facilitating the extraction of valuable data and improving the efficiency of data analysis and mining.

[0086] (3) Content consumption and reception scenarios

[0087] In content consumption and reception scenarios, content consumers and receivers may need to convert text content such as articles and blogs published by content distributors from one format to another to suit their content information retrieval needs. By applying the text processing method provided in this application to content consumption and reception scenarios, the text content and structure can be comprehensively considered. A pre-trained language model and format mapping rules can be used to complete the format conversion from a first format (which is more difficult to read and write) to a second format (which is easier to read and write). This makes the converted, less difficult-to-read text easier for content consumers and receivers to understand, enhancing text accessibility and facilitating the extraction of valuable textual information.

[0088] It should be noted that, in addition to the document editing and sharing, data analysis, and content consumption and reception scenarios mentioned above, the text processing method of this application can also be applied to other text format conversion scenarios in practical applications. For example, it can also be applied to multimedia content production scenarios, programming and software development scenarios, etc., without specific limitations in this application.

[0089] For example, the text processing method provided in this application can be applied to Figure 2 The implementation environment shown uses a text processing device 100 to perform operations. Figure 2 The illustrated implementation environment includes a text processing device 100 and, for example, a user terminal 101. The text processing device 100 and the user terminal 101 can be directly or indirectly connected via wired or wireless communication, etc., without specific limitations in this application. Optionally, the implementation environment may also include a database 103, etc. The text processing device 100 can also be connected to the database 103 via a network.

[0090] When users try to process text in the first format, they often find it difficult to extract the textual information due to the complexity of reading and writing this format. For example... Figure 2As shown, in step S1, the user can send the text to be processed in the first format to the text processing device 100 via the user terminal 101. For example, after receiving the user's input of the text to be processed, the user terminal 101 carries the text in the first format in the processing request and then sends the processing request to the text processing device 100, thereby obtaining the text in the first format carried in the processing request. Furthermore, in step S2, the text processing device 100 also needs to obtain the format mapping rule. Through this format mapping rule, the mapping relationship between the format syntax of the first format and the format syntax of the second format can be understood. For example, the text processing device 100 can obtain the format mapping rule from a storage medium such as the database 103; this application does not limit this.

[0091] In step S3, after obtaining the text to be processed, the text processing device 100 extracts the text content features of the text to be processed. In step S4, after extracting the text content features, the text processing device 100 also extracts the text structure features of the text to be processed based on the text to be processed and the text content features. In step S5, the text processing device 100 further processes the format mapping rules, text content features, and text structure features based on a pre-trained language model to generate target text of the second format type.

[0092] Optionally, in Figure 2 The implementation environment shown may also include step S6. That is, in step S6, the text processing device 100 may also send the target text of the second format type to the user terminal 101, and then the user terminal 101 may display the target text through a client, engine or other visual interface in step S7, so as to provide the user with a view and enable subsequent text processing operations such as editing the target text.

[0093] This application addresses the connection and difference between the first format and the second format, namely, that the readability and writeability of the first format must be greater than that of the second format. This application does not specify which first format to use or which second format to convert to.

[0094] The text processing device 100 mentioned in this application can be a processing device with data processing capabilities, such as a terminal device or a server. The terminal device can include, but is not limited to, smartphones, desktop computers, laptops, tablets, smart speakers, in-vehicle devices, smartwatches, wearable smart devices, smart voice interaction devices, smart home appliances, and aircraft. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This application does not specifically limit the scope of the application.

[0095] The database 103 mentioned in this application can be simply viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, shared by multiple users, with minimal redundancy, and independent of application programs. A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or Extensible Markup Language (XML); or according to the type of computer they support, such as server clusters or mobile phones; or according to the query language used, such as Structured Query Language (SQL) or XQuery; or according to performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, simultaneously supporting multiple query languages. The database 103 in this application can be used to store pre-trained language models, etc. Optionally, it can also be used to store text to be processed, etc.

[0096] The pre-trained language models mentioned in this application can also be deployed in the text processing device 100 or in the database 103, without any specific limitation. In addition, the pre-trained language models mentioned may include, but are not limited to, generative pre-trained transformer (GPT) models based on the transformer structure.

[0097] For example, the text processing method provided in this application, in addition to being applied to the above... Figure 2 Beyond the application scenarios mentioned, in practical applications, it can also be applied to fields such as artificial intelligence (AI), blockchain, and cloud technology, which are not limited in this application.

[0098] The following describes a text processing method provided by an embodiment of this application with reference to the accompanying drawings. Figure 3 A flowchart illustrating a text processing method provided in an embodiment of this application is shown, which is described only as an example using a text processing device as the execution subject. Figure 3 As shown, the text processing method may include the following steps:

[0099] 301. Obtain the text to be processed and the format mapping rules of the first format. The format mapping rules are obtained by the pre-trained language model processing the text samples of the first format and the text samples of the second format. The format mapping rules are used to represent the mapping relationship between the format syntax of the first format and the format syntax of the second format. The format reading and writing difficulty of the first format is greater than that of the second format.

[0100] In one or more embodiments, the format type of the text to be processed is a first format. In other words, the first format reflects the text format of the text to be processed. The described text format can be understood as a specification or standard used to represent text content. In this application, the text processing device needs to obtain the text to be processed in the first format in order to perform text format conversion on the text to be processed.

[0101] Optionally, the first format of this application includes, but is not limited to, portable document format (PDF), Word, hypertext markup language (HTML), and other text formats.

[0102] As an illustrative example, the way a text processing device acquires the text to be processed can vary depending on the source of the text. For example, it can be acquired in the following way:

[0103] In Method 1, the user can use their user terminal to generate text to be processed in real time in a first format. When the user needs to convert the format, the user terminal sends the real-time generated text to the text processing device via the network, allowing the text processing device to obtain the text in the first format. Alternatively, in Method 2, the text to be processed can be generated in advance and stored in a database or other storage medium. When format conversion is needed, the text processing device can directly retrieve the corresponding text from the database or other storage medium based on the text identifier in the format conversion request. Or, in Method 3, after generating the text to be processed, it can be uploaded to a blockchain network. In this way, the text processing device can obtain the text from the blockchain network.

[0104] It should be noted that in practical applications, other methods of obtaining the text to be processed may also be included, and this application does not limit them.

[0105] In this application, the text processing device also needs to obtain format mapping rules. These format mapping rules can be obtained by a pre-trained language model processing text samples of the first format and text samples of the second format. The text sample of the first format can be understood as text of the first format type; the text sample of the second format can be understood as text of the second format type.

[0106] The second format mentioned in this application includes, but is not limited to, Word and lightweight markup language (Markdown). Markdown is a lightweight markup language that allows for the easy creation of formatted text and is commonly used in writing, blogging, and other text-related scenarios. It achieves text layout and styling through simple formatting syntax.

[0107] This application addresses the relationship and difference between the first and second formats, namely: the readability and writeability of text conforming to the first format is greater than that of text conforming to the second format. In other words, in this application, text in the first format is more difficult to read and write than text in the second format.

[0108] It should be noted that this application does not specifically limit the first format or the second format from which the text will be converted. For example, this application can convert text from PDF to Word format; or, it can convert text from PDF to Markdown format; or, it can convert text from Word to Markdown format; or, it can convert text from HTML to Markdown format, etc. This application will only describe the first format as PDF and the second format as Markdown as examples, and no specific limitations are made in practice.

[0109] As an illustrative example, the format mapping rules mentioned above can be specifically described as follows: Figure 4 Use the described process structure diagram to understand. For example... Figure 4 As shown, after acquiring a text sample in a first format, the text processing device uses a pre-trained language model to parse the format syntax of the first format text sample, thereby obtaining the format syntax of the first format. Similarly, after acquiring a text sample in a second format, the text processing device also uses a pre-trained language model to parse the format syntax of the second format text sample, thereby obtaining the format syntax of the second format. In this way, the text processing device uses the pre-trained language model to perform a grammatical mapping between the format syntax of the first format and the format syntax of the second format, thereby obtaining the format mapping rule. For example, if the format syntax of the first format includes a first heading and the format syntax of the second format includes a second heading, the pre-trained language model can establish a mapping relationship between the first heading and the second heading. For example, if the first format is PDF format and the second format is Markdown format, if the level of the first heading is the third level heading in the PDF format, it is represented as "third-level heading", while the third level heading of the second heading is represented as "###". In this case, a mapping relationship is established between "third-level heading" and "###".

[0110] In other words, this format mapping rule reflects the mapping relationship between the format syntax of the first format and the format syntax of the second format. Alternatively, this format mapping rule can also be understood as the format syntax transformation from the first format to the second format.

[0111] It should be noted that the formatting syntax for the first format and the formatting syntax for the second format may include other text elements such as tables, paragraphs, and lists in addition to the text element of headings. This application does not make specific limitations.

[0112] 302. Extract the text content features of the text to be processed.

[0113] In one or more embodiments, after obtaining the text to be processed, the text processing device also needs to extract the text content features of the text to be processed. For example, the text processing device can call a text parsing model corresponding to the first format and use the text to be processed as input to the text parsing model. In this way, the text processing device extracts features from the text to be processed through the text parsing model to obtain the text content features of the text to be processed.

[0114] For example, taking PDF as the first format, its text parsing model can include, but is not limited to, parsing libraries such as Python's PyPDF2, PDFMiner, or PyMuPDF (also known as Fitz), etc., and this application does not specify the specific parsing model.

[0115] As an illustrative description, Figure 5 This diagram illustrates the flowchart for extracting text content features provided in this application. Figure 5 As shown, after obtaining the text to be processed, the text processing device can identify the text content from the text to be processed.

[0116] For example, a text processing device can directly identify the plain text content in the text to be processed using a text parsing model. Alternatively, it can use the same model to identify embedded charts or images within the text. After identifying the embedded charts or images, it can extract the descriptive text content located within the detection box, such as the chart or image title or description. For images where descriptive text cannot be directly extracted from the detection box, the text processing device can also utilize optical character recognition (OCR) technology, such as the Tesseract model, to identify the descriptive text content in the image. It should be noted that the text content (i.e., plain text content and descriptive text) in this application includes text information such as font and multiple text blocks. The font includes one or more of the following: font size, font style, font weight, and font name. A text block can include, but is not limited to, a sentence, a paragraph, or a chapter.

[0117] After recognizing the text content, the text processing device uses the text parsing model to perform text feature recognition on the text font, thereby obtaining the text font features. It should be noted that the text font features provide information about the font type of the text.

[0118] Besides extracting text font features, it's also necessary to understand the position of text blocks. To this end, the text processing device needs to extract the coordinate information of each text block using the text parsing model. It should be noted that the coordinate information of each text block reflects its position within the text to be processed. After extracting the coordinate information of each text block, the text processing device then uses the text parsing model to sort the multiple text blocks according to their coordinate information, thereby obtaining text position features. It should be noted that these text position features reflect the text order of each text block within the text to be processed. Determining the position of text blocks through these text position features ensures that the reading order of the text conforms to natural reading habits.

[0119] In this way, after extracting the text font features and text position features, the text processing device obtains the text content features of the text to be processed based on these features. These text font features and text position features assist in text recognition and extraction, improve the accuracy of text recognition, and enhance the accuracy of text content feature extraction.

[0120] 303. Extract text structure features of the text to be processed based on the text to be processed and text content features.

[0121] In one or more embodiments, since text structure is usually associated with text content, after performing step 301 to obtain the text to be processed and performing step 302 to extract the text content features, the text processing device also takes into account the text to be processed and the text content features to extract the text structure features of the text to be processed.

[0122] It should be noted that the text structure of the text to be processed includes, but is not limited to, headings, paragraphs, lists, and tables, etc., and is not specifically limited in this application. Taking a text structure including headings, paragraphs, lists, and tables as an example, the corresponding determined text structure may include one or more of the following features: text heading features, text paragraph features, text list features, and text table features.

[0123] As an illustrative description, Figure 6 A schematic diagram illustrating the structure for extracting text structural features provided in an embodiment of this application is shown. For example... Figure 6As shown, taking a text structure including headings, paragraphs, lists, and tables as an example, after obtaining the text to be processed and its content features, the text processing device can determine the text heading features based on the text font features and the font thresholds for preset heading levels. For example, the text processing device obtains the font size feature from the text font features. Further, the text processing device compares the font size feature with the font thresholds for preset heading levels. If the font size feature falls within the range corresponding to a certain preset heading level's font threshold, the corresponding text heading feature can be determined. For example, if the font thresholds for each preset heading level are: a heading font size > 30, it is recorded as a level 1 heading; 20 ≤ font size < 30, it is recorded as a level 2 heading; 16 ≤ font size < 20, it is recorded as a level 3 heading. Assuming the extracted font size feature is 28, by comparison, it can be seen that the font size feature falls within the threshold range corresponding to a level 2 heading. At this point, the text heading feature of the text to be processed can be determined to be a level 2 heading. Optionally, the text heading features include the heading content and the heading level.

[0124] from Figure 6 Furthermore, after extracting text position features from text content features, the text processing device also determines paragraph identifiers between text blocks based on the text position features, thereby obtaining text paragraph features based on these paragraph identifiers. For example, taking PDF format as an example, its paragraph identifiers can be represented by the spacing and indentation between text blocks, thus text paragraph features can include features such as the spacing and indentation between text blocks.

[0125] In addition, such as Figure 6 As shown, after obtaining the text to be processed, the text processing device also identifies list identifiers in the text to obtain text list features. For example, taking PDF format as the first example, list identifiers may include bullet points, numbering, or indentation. Therefore, by identifying features such as bullet points, numbering, or indentation, the list type and list level can be identified to obtain text list features. In other words, text list features include list type and list level.

[0126] In addition, after obtaining the text to be processed, the text processing device also needs to recognize the gridline symbols in the text to obtain the text table features. For example, taking PDF format as an example, table symbols may include gridlines and other symbols. Therefore, by recognizing the gridline symbols, the table structure can be identified, thus obtaining the text table features. Alternatively, in practical applications, the text processing device can also recognize regular text blocks to identify the table structure and thus obtain the text table features. In another optional example, after recognizing the table structure, the content of each cell in the table structure can be parsed, and the structural relationships can be maintained so that it can be correctly represented in the subsequent target text.

[0127] from Figure 6 It can also be seen that after extracting the text title features, text paragraph features, text table features, and text list features, the text processing device can use one or more of these text title features, text paragraph features, text table features, and text list features to determine the text structure features of the text to be processed.

[0128] It should be noted that the above is only an example of a text structure including headings, paragraphs, tables, and lists. In practical applications, other structural elements may also be included, and this application does not impose any specific limitations.

[0129] For example, Figure 7 A schematic diagram of the code for extracting text structural features provided in this application is shown. For example... Figure 7 As shown, taking a PDF file as the text to be processed and the PyMuPDF parsing library as the text parsing model, the PyMuPDF parsing library is imported by executing the code "import fitz", and the extract_text_from_pdf(...) function is called, accepting the path of the PDF file as a parameter. Then, the code "document=fitz.open(pdf_path)" is executed to open the PDF file. After opening the PDF file, the code "print(f``Page{page_number+1}:\n{text}\n``) is executed to open the text content of the current page in the PDF file. Then, the code corresponding to the analyze_page_structure(...) function ".....if line.isupper()or``CHAPTER``in line:print(f``Title:{line}``)...." is executed to identify text structure features.

[0130] 304. Based on the pre-trained language model, the format mapping rules, text content features and text structure features are processed to generate target text with the second format type.

[0131] In one or more embodiments, after extracting the text content features and text structure features of the text to be processed, the text processing device also uses the text content features and text structure features as input to a pre-trained language model, and, under the guidance of format mapping rules, performs mapping processing on the text content features and text structure features through the pre-trained language model to generate target text with a second format type. That is, the target text has a second format, which satisfies the format syntax of the second format.

[0132] This application identifies the text structure and content of text that is difficult to read and write. Based on this, and leveraging a pre-trained language model and format mapping rules, it efficiently and accurately converts the text from a first format (difficult to read and write) to a second format (easier to read and write). This not only makes the generated target text easier to edit and maintain, facilitating rapid updates, but also makes it easier for users to access information from the easier-to-read target text, enhancing text accessibility.

[0133] In some other alternative embodiments, in the above... Figure 3 Based on the described embodiments, the generation process for generating the target text in step 304 above can be referred to the following: Figure 8 The flowchart shown is for your understanding, as detailed below:

[0134] S3041. By using a pre-trained language model, the text content features are mapped based on format mapping rules to obtain target content that conforms to the second format.

[0135] In one or more embodiments, after extracting text content features, the text processing device uses these features as input to a pre-trained language model. Then, according to format mapping rules, the pre-trained language model maps the text content features to obtain target content conforming to a second format. As an illustrative description, the text processing device maps text content features to fields corresponding to the format syntax of the second format according to the format mapping rules. For example, the text processing device can map text font features to font fields corresponding to the format syntax of the second format according to the format mapping rules; for example, the font size and style of the text content are mapped to fields corresponding to the format syntax of the second format. Similarly, the text processing device can map text position features to position fields corresponding to the format syntax of the second format according to the format mapping rules to ensure that text blocks are displayed in the target content in the correct order.

[0136] S3042. Using a pre-trained language model, structural mapping is performed on the text structural features based on format mapping rules to obtain the target structure that conforms to the second format.

[0137] In one or more embodiments, after obtaining the text structural features, the text processing device also uses these text structural features as input to a pre-trained language model. Thus, according to the format mapping rules, the pre-trained language model performs structural mapping on the text structural features, thereby obtaining a target structure conforming to the second format.

[0138] In some alternative examples, the text structure may include a text heading and a text paragraph. Furthermore, from the foregoing... Figure 6 It is known that text structure features include text title features and text paragraph features. In step S3042 above, the process for determining the target structure that conforms to the second format can be implemented as follows:

[0139] The target title is determined based on text title features and formatting mapping rules. As an illustrative description, in determining the target title, the text processing device first determines the title content and title level in the text to be processed based on text title features. After obtaining the title content and title level, the title level and formatting mapping rules are used to determine the title level symbol in the second format corresponding to the title level. Thus, the text processing device then determines the target title based on the title content and title level symbol.

[0140] For example, taking PDF as the first format and Markdown as the second format, if the heading level of the PDF text to be processed is determined to be "Level 3 Heading," then according to the format mapping rules, the corresponding number of heading level symbols, i.e., "###", can be determined in the Markdown format. Further, based on the style corresponding to the heading level symbol "###", the font weight and other parameters of the heading content are adjusted to obtain the corresponding target heading.

[0141] In addition, the text processing device needs to determine the target paragraph based on text paragraph features and format mapping rules. As an illustrative example, the text processing device first determines the paragraph marks of the text paragraphs in the text to be processed based on the text paragraph features. After obtaining the paragraph marks, the text processing device then changes the paragraph marks of the text paragraphs to the target paragraph marks corresponding to the second format based on the format mapping rules, thereby determining the target paragraph.

[0142] For example, taking PDF as the first format and Markdown as the second, if the paragraph mark for the text to be processed in the PDF format is determined to be "indented," then according to the format mapping rules, the paragraph mark "indented" can be adjusted to the corresponding target paragraph mark in the Markdown format, namely "blank line." In other words, you only need to keep the text content in each paragraph of the text to be processed and change the "indented" between each paragraph to "blank line" to obtain the target paragraph.

[0143] In this way, after obtaining the target title and target paragraph, the text processing device then uses the target title and target paragraph to obtain a target structure that conforms to the second format. By presenting a target structure that conforms to the second format from the target title and target paragraph in the above manner, it not only helps to enhance the reader's understanding of the text information and improve reading efficiency, but also enhances the readability and attractiveness of the text.

[0144] In other alternative examples, besides text headings and paragraphs being important elements of the text structure, tables and lists within the text are also crucial components of the text structure. Therefore, when extracting, for example, the aforementioned... Figure 6 In the context of the text list and text table features mentioned above, the text processing device can further consider both text table and text list features to determine the target structure, in addition to using the target heading and target paragraph to determine the target structure. Specifically, this can be achieved by sampling the following methods:

[0145] The target list is determined based on text list features and format mapping rules. As an illustrative description, the text processing device first determines the list notation of the text list in the text to be processed based on the text list features. Then, based on the format mapping rules, the text processing device changes the list notation of the text list to the target list notation of a second format to obtain the target list.

[0146] For example, taking PDF as the first format and Markdown as the second format, if the list symbol of the text list in the PDF format is determined to be "bullet ●" in an unordered list, then according to the format mapping rules, "bullet ●" can be changed to the corresponding target list symbol in the Markdown format, such as "-", "*", "+", etc., unordered list symbols. Alternatively, for symbols in an ordered list, they can also be changed to symbols such as "number plus a period (e.g., 1.)". This application does not impose any specific limitations.

[0147] Similarly, the target table is determined based on text table features and format mapping rules. As an illustrative description, the text processing device determines the table symbols of the text tables in the text to be processed based on the text table features. Further, the text processing device changes the table symbols of the text tables to the target table symbols of a second format based on the format mapping rules to obtain the target table.

[0148] For example, taking a PDF format as the first example and a Markdown format as the second example, if it is determined that the text tables in the PDF format use gridlines as table symbols, then according to the format mapping rules, the gridlines can be changed to the corresponding target list symbols in the Markdown format, such as using the separator "|". For example, it can be changed to symbols such as "|column1|column2|column3|" or "|-----|-----|-----|". This application does not impose any specific limitations.

[0149] Thus, based on the aforementioned determination of the target title and target paragraph, the text processing device comprehensively considers the target table and target list to determine the target structure. In other words, the text processing device needs to use a pre-trained language model to obtain a target structure conforming to the second format, based on the target title, target paragraph, target list, and target table.

[0150] By comprehensively considering the target headings, paragraphs, tables, and lists, a highly efficient, clear, and easy-to-understand target structure conforming to the second format is created. This target structure not only helps to clearly present textual information but also enhances the reader's comprehension, improving reading efficiency and information absorption efficiency.

[0151] It should be noted that, regarding the execution order between the above steps S3041 and S3042, this application is not limited to executing step S3041 first or step S3042 first.

[0152] S3043. Based on the target content and target structure, generate target text in the second format.

[0153] In one or more embodiments, after determining the target content in step S3041 and determining the target structure in step S3042, the text processing device can generate target text of a second format based on the target content and target structure. By using the target content and target structure obtained after format syntax conversion to generate the corresponding second-format target text, target text with lower reading and writing difficulty can more easily convey textual information and is easier for users to understand, thus enhancing the communicative power and comprehensibility of the text and improving its editability and maintainability.

[0154] It should be noted that the second format and other content mentioned here can be understood by referring to the content described in step 301 above, and will not be repeated here.

[0155] For example, Figure 9 A schematic diagram of the code for generating the target text provided in this application is shown. For example... Figure 9 As shown, taking PDF as the first format and Markdown as the second format as an example, the target content is generated by executing the code "def generate_markdown(text_blocks)". Additionally, the code "markdown_content+=f``#{content}\n\n`` is executed to convert the headings in the text to be processed into target headings conforming to Markdown format. Similarly, the code "markdown_content+=f``{content}\n\n`` is executed to convert the paragraphs in the text to be processed into target paragraphs conforming to Markdown format. Furthermore, the code "markdown_content+=f``-{content}\n\n`` is executed to convert the headings in the text to be processed into unordered lists conforming to Markdown format. After determining the target table using the same logic, the code "return markdown_content" is executed to obtain the target text conforming to Markdown format.

[0156] In some alternative examples, based on one or more of the foregoing embodiments, after generating the target text in step 304, the text processing device may further optimize the target text. The specific optimization method can be understood as follows:

[0157] After generating the target text, the text processing device first performs a second-format syntax check on the target text to obtain target formatting information that does not conform to the second-format syntax. As an illustrative description, target formatting information includes one or more of the following: blank lines, indentation, non-compliant characters, and incorrect symbols.

[0158] Furthermore, the text processing device processes the target format information based on the format syntax of the second format to obtain the optimized target text.

[0159] For example, taking the second format as Markdown, under different input sources or during text recognition, a dash might be incorrectly recognized as a long dash (—) or a short dash (-). In this case, the generated target text will also display as a long dash (—) or a short dash (-), which does not conform to the Markdown format syntax. Therefore, when a text processing device recognizes an incorrect symbol, it can replace it with a symbol conforming to the Markdown format syntax, such as replacing both long dashes (—) and short dashes (-) with double hyphens (--).

[0160] For example, extra spaces or other incorrect symbols should not exist before or after common punctuation marks (such as commas, periods, etc.), inside parentheses, or at the end of each line of text. Therefore, when a text processing device detects an incorrect symbol, it can perform corrective operations such as deletion.

[0161] For example, in the second format, Markdown, besides the text beginning and ending for each list item, there should be a blank line before and after each list item. If the text processing device does not recognize blank lines or other symbols before and after the list item, it can add blank lines or similar processing to the list item.

[0162] It should be noted that in practical applications, other target format information that does not conform to the second format syntax is also included, but this application does not specify the specifics.

[0163] By using the above methods, and by identifying target format information in the target text that does not conform to the second format syntax, and by performing optimization processing such as correcting the target format information, the text quality and readability of the generated optimized target text are significantly improved.

[0164] In some alternative embodiments, when different languages ​​are used to express the content of the target text, the characters of the content may take different forms due to the different grammar of the languages, resulting in an inaccurate representation of the text's meaning. For example, the grammatical structures of English and French differ; in French, adjectives are usually placed after nouns, while in English they are placed before nouns. Therefore, if English grammatical structures are used to express text that should be expressed in French, the text will be inaccurate and fail to convey its true meaning.

[0165] Therefore, in this application, after optimizing the target text to obtain the optimized target text, the text processing device can also detect whether the symbols in the optimized target text satisfy the text language set by the target text. Specifically, this can be achieved in the following way:

[0166] After obtaining the optimized target text, the text processing device needs to perform a text language check on the optimized target text based on a language detection algorithm to obtain the text language of the optimized target text. For example, the text language includes, but is not limited to, English, French, Chinese, etc., and this application does not make specific limitations. In addition, the described language detection algorithm includes, but is not limited to, natural language processing (NLP) models.

[0167] Thus, after obtaining the text language of the optimized target text, the text processing device can use, for example, a multilingual pre-trained model and, based on a pre-defined symbol lookup table and the text language, correct the first symbol in the optimized target text to obtain target text that conforms to the text language. It should be noted that the first symbol refers to the text symbol in the optimized target text that does not conform to the text language. Furthermore, the described multilingual pre-trained model includes, but is not limited to, models based on the Transformer architecture such as multilingual bidirectional encoder representations from transformers (mBERT) or XLM-R (XLM-RoBERTa).

[0168] For example, if the optimized target text is detected to be in French, and the first symbol in the target text is identified as an English quotation mark (such as ""), then the English quotation mark needs to be corrected to a French quotation mark (i.e., 《 ") using a preset symbol lookup table between English and French. This will ultimately generate a target text that conforms to both the second format grammar and the French language.

[0169] Modifying and optimizing text symbols in the target text that do not conform to the text language in the above manner can improve text accuracy, avoid readers' misunderstanding of the text, enhance text standardization and readability, and improve the overall quality of the text.

[0170] Optionally, taking PDF format as the first format and Markdown format as the second format as an example, Figure 10 This document illustrates a framework process for converting text from PDF to Markdown format, as provided in this application.

[0171] like Figure 10As shown, after obtaining the text to be processed in PDF format (hereinafter referred to as the PDF document), content parsing and preprocessing can be performed on the PDF document, namely, identifying text content features and text structure features. For example, text content features can be identified by analyzing descriptive text in multi-column layout text, charts, and images within the PDF document. Additionally, text structure features can be obtained by identifying text titles, paragraphs, lists, and tables. It should be noted that the extraction process for text content features and text structure features can be referred to the aforementioned methods. Figure 3 The content in the text will be understood in detail here, and will not be elaborated upon further.

[0172] After extracting the text content and structure features of the PDF document, a pre-trained language model, such as the GPT model, combined with format mapping rules, is used to process these features and generate the target text in Markdown format. It should be noted that the processing of text content and structure features here is explained in the preceding text. Figure 3 The content in the text will be understood in detail here, and will not be elaborated upon further.

[0173] After generating the Markdown target text, you can further optimize it and perform multilingual processing on the optimized Markdown target text to obtain a Markdown document that conforms to the specified language. It should be noted that how to optimize the Markdown target text and perform multilingual processing and other text language checking operations can be found in the previous section. Figure 3 The content of other optional embodiments can be understood from them, and will not be elaborated here.

[0174] Through the methods described above, Markdown, a lightweight markup language, automatically converts PDF documents into Markdown-compliant documents. This not only reduces the need for manual editing and formatting but also eliminates the need for professional PDF editing software, saving costs. Furthermore, the ease of modification and maintenance of Markdown allows for rapid updates of text content. Its ease of access by screen readers improves the efficiency of text access for users with visual impairments, and it also achieves cross-platform compatibility. In addition, Markdown documents are based on plain text, facilitating format standardization, which not only enables rapid information retrieval but also ensures consistency between different documents. It also makes collaborative document editing easier, promoting collaboration.

[0175] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. It is understood that to achieve the above functions, corresponding hardware structures and / or software modules are included to execute each function. Those skilled in the art should readily recognize that, based on the modules and algorithm steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0176] This application embodiment can divide the device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0177] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0178] The text processing apparatus in the embodiments of this application will now be described in detail. Figure 11 This is a schematic diagram of the functional module structure of the text processing device provided in the embodiments of this application. Figure 11 As shown, the text processing device may include an acquisition unit 1101, an extraction unit 1102, and a processing unit 1103.

[0179] The acquisition unit 1101 is used to acquire the text to be processed with the first format and the format mapping rule. The format mapping rule is obtained by the pre-trained language model processing the text samples of the first format and the text samples of the second format. The format mapping rule is used to represent the mapping relationship between the format syntax of the first format and the format syntax of the second format. The format reading and writing difficulty of the first format is greater than that of the second format.

[0180] Extraction unit 1102 is used to extract text content features of the text to be processed;

[0181] Extraction unit 1102 is used to extract text structure features of the text to be processed based on the text to be processed and text content features;

[0182] The processing unit 1103 is used to process the format mapping rules, text content features and text structure features based on the pre-trained language model to generate target text of the second format type.

[0183] In another possible implementation of this application embodiment, the processing unit 1103 is specifically used for:

[0184] By using a pre-trained language model, text content features are mapped based on format mapping rules to obtain target content that conforms to the second format.

[0185] By using a pre-trained language model, structural mapping is performed on the text structural features based on format mapping rules to obtain the target structure that conforms to the second format.

[0186] Based on the target content and target structure, generate target text in the second format.

[0187] In another possible implementation of this application embodiment, the text structure features include text title features and text paragraph features; the processing unit 1103 is specifically used for:

[0188] The target title is determined based on text title features and format mapping rules;

[0189] Target paragraphs are determined based on text paragraph features and format mapping rules;

[0190] Based on the target title and target paragraph, a target structure conforming to the second format is obtained.

[0191] In another possible implementation of an embodiment of this application, the processing unit 1103 is specifically used for:

[0192] Determine the title content and title level in the text to be processed based on text title features;

[0193] Based on the heading level and format mapping rules, determine the heading level symbol corresponding to the second format of the heading level;

[0194] The target title is determined based on the title content and title level symbols.

[0195] In another possible implementation of an embodiment of this application, the processing unit 1103 is specifically used for:

[0196] Identify the paragraph marks of the text paragraphs in the text to be processed based on the characteristics of the text paragraphs;

[0197] Based on the format mapping rules, the paragraph marks of the text paragraphs are changed to the target paragraph marks corresponding to the second format in order to determine the target paragraph.

[0198] In another possible implementation of this application embodiment, the text structure features further include text list features and text table features; the processing unit 1103 is specifically used for:

[0199] The target list is determined based on text list features and format mapping rules;

[0200] The target table is determined based on text table features and format mapping rules;

[0201] By using a pre-trained language model, a target structure conforming to the second format is obtained based on the target title, target paragraph, target list, and target table.

[0202] In another possible implementation of an embodiment of this application, the processing unit 1103 is specifically used for:

[0203] The list symbols of the text list in the text to be processed are determined based on the features of the text list;

[0204] Based on the format mapping rules, the list symbols of the text list are changed to the target list symbols of the second format to obtain the target list.

[0205] In another possible implementation of an embodiment of this application, the processing unit 1103 is specifically used for:

[0206] Determine the table symbols of the text tables in the text to be processed based on the characteristics of the text tables;

[0207] Based on the format mapping rules, the table symbols of the text table are changed to the target table symbols of the second format to obtain the target table.

[0208] In another possible implementation of an embodiment of this application, the processing unit 1103 is specifically used for:

[0209] Identify text content from the text to be processed, including text font and multiple text blocks;

[0210] Text font features are obtained by performing text feature recognition processing on the text font.

[0211] Extract the coordinate information of each text block. The coordinate information of each text block is used to represent the position of the corresponding text block in the text to be processed.

[0212] The text blocks are sorted based on their coordinate information to obtain text location features;

[0213] Based on text font features and text position features, the text content features of the text to be processed are obtained.

[0214] In another possible implementation of an embodiment of this application, the processing unit 1103 is specifically used for:

[0215] The text title features are determined based on the text font features and the font threshold of the preset title level;

[0216] Paragraph identifiers between text blocks are determined based on text location features to obtain text paragraph features;

[0217] The list identifiers in the text to be processed are identified to obtain the text list features;

[0218] The grid line symbols in the text to be processed are identified to obtain the text table features;

[0219] Based on one or more of the following features: text title features, text paragraph features, text list features, and text table features, the text structure features of the text to be processed are obtained.

[0220] In another possible implementation of an embodiment of this application, the processing unit 1103 is further configured to:

[0221] After processing the format mapping rules, text content features, and text structure features based on the pre-trained language model to generate target text of the second format type, the target text is then subjected to second format syntax checking to obtain target format information that does not conform to the second format syntax in the target text.

[0222] The target format information is processed based on the format syntax of the second format to obtain the optimized target text.

[0223] In another possible implementation of the embodiments of this application, the target format information includes one or more of blank lines, indentation, non-compliant characters, and error symbols.

[0224] In another possible implementation of an embodiment of this application, the processing unit 1103 is further configured to:

[0225] After processing the target format information based on the format syntax of the second format to obtain the optimized target text, the optimized target text is then checked for text language based on the language detection algorithm to obtain the text language of the optimized target text.

[0226] Based on a preset symbol lookup table and text language, the first symbol in the optimized target text is corrected to obtain target text that conforms to the text language. The first symbol is a text symbol in the optimized target text that does not conform to the text language.

[0227] The text processing device in the embodiments of this application has been described above from the perspective of modular functional entities. The text processing equipment in the embodiments of this application will be described below from the perspective of hardware processing. Figure 12 This is a schematic diagram of the hardware structure of a text processing device provided in an embodiment of this application. The text processing device can vary considerably due to differences in configuration or performance, and may include, but is not limited to, [other types of devices]. Figure 11 The text processing device shown in the figure, etc.

[0228] like Figure 12 As shown, the text processing device may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and a memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. The memory 332 and storage media 330 may be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the text processing device. Furthermore, the CPU 322 may be configured to communicate with the storage media 330 and execute a series of instruction operations in the storage media 330 on the text processing device 300. Exemplarily, the CPU 322 is used to execute the application program 342 stored in the storage media 330, thereby implementing the text processing method provided in the above embodiments of this application.

[0229] The text processing device 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0230] For example, Figure 12 The central processing unit 322 can invoke computer execution instructions stored in memory 332 to cause the text processing device to perform actions such as... Figures 3 to 10 The method in the corresponding method embodiment.

[0231] Specifically, Figure 11 The functions / implementation processes of the extraction unit 1102 and the processing unit 1003 can be obtained through... Figure 12 The central processing unit 322 in the memory calls computer execution instructions stored in the memory 332 to achieve this. Figure 11 The function / implementation process of the acquisition unit 1101 can be achieved through... Figure 12 The input / output interface 358 is used to implement this.

[0232] The steps performed by the text processing device in the above embodiments can be based on this Figure 12 The text processing device structure shown is illustrated.

[0233] This application also provides a computer-readable storage medium storing a computer program or instructions thereon, which, when executed by a processor, implements the steps of the methods described in the foregoing embodiments.

[0234] This application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the methods described in the foregoing embodiments.

[0235] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0236] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0237] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0238] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0239] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0240] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0241] A computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, they generate, in whole or in part, the processes or functions according to embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., SSDs), etc.

[0242] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A text processing method, characterized in that, include: The text to be processed is obtained in a first format and a format mapping rule. The format mapping rule is obtained by a pre-trained language model processing text samples in the first format and text samples in the second format. The format mapping rule is used to characterize the mapping relationship between the format syntax of the first format and the format syntax of the second format. The format reading and writing difficulty of the first format is greater than that of the second format. Extract the text content features of the text to be processed; Based on the text to be processed and the text content features, extract the text structure features of the text to be processed; Based on the pre-trained language model, the format mapping rules, the text content features, and the text structure features are processed to generate target text with the second format type.

2. The method according to claim 1, characterized in that, Based on the pre-trained language model, the format mapping rules, the text content features, and the text structure features are processed to generate target text of the second format type, including: Using the pre-trained language model, the text content features are mapped based on the format mapping rules to obtain target content that conforms to the second format. Using the pre-trained language model, the text structure features are structurally mapped based on the format mapping rules to obtain a target structure that conforms to the second format. Based on the target content and the target structure, target text of the second format type is generated.

3. The method according to claim 2, characterized in that, The text structure features include text title features and text paragraph features; through the pre-trained language model, the text structure features are structurally mapped based on the format mapping rules to obtain a target structure that conforms to the second format, including: The target title is determined based on the text title features and the format mapping rules; The target paragraph is determined based on the text paragraph features and the format mapping rules; Based on the target title and the target paragraph, a target structure conforming to the second format is obtained.

4. The method according to claim 3, characterized in that, Determining the target title based on the text title features and the format mapping rules includes: The title content and title level in the text to be processed are determined based on the text title features. Based on the heading level and the format mapping rule, determine the heading level symbol of the second format corresponding to the heading level; The target title is determined based on the title content and the title level symbols.

5. The method according to claim 3, characterized in that, Determining the target paragraph based on the text paragraph features and the format mapping rules includes: The paragraph symbols of the text paragraphs in the text to be processed are determined based on the characteristics of the text paragraphs. Based on the format mapping rules, the paragraph symbols of the text paragraphs are changed to the target paragraph symbols corresponding to the second format to determine the target paragraph.

6. The method according to claim 3, characterized in that, The text structure features also include text list features and text table features; The method further includes: The target list is determined based on the text list features and the format mapping rules; The target table is determined based on the text table features and the format mapping rules; Based on the target title and the target paragraph, a target structure conforming to the second format is obtained, including: Using the pre-trained language model, a target structure conforming to the second format is obtained based on the target title, the target paragraph, the target list, and the target table.

7. The method according to claim 6, characterized in that, Determining the target list based on the text list features and the format mapping rules includes: The list symbols of the text list in the text to be processed are determined based on the features of the text list; Based on the format mapping rules, the list symbols of the text list are changed to the target list symbols of the second format to obtain the target list.

8. The method according to claim 6, characterized in that, Determining the target table based on the text table features and the format mapping rules includes: The table symbols of the text tables in the text to be processed are determined based on the characteristics of the text tables. Based on the format mapping rules, the table symbols of the text table are changed to the target table symbols of the second format to obtain the target table.

9. The method according to any one of claims 1 to 8, characterized in that, Extracting the text content features of the text to be processed includes: Identify text content from the text to be processed, the text content including text font and multiple text blocks; The text font is subjected to text feature recognition processing to obtain text font features; The coordinate information of each text block is extracted, and the coordinate information of each text block is used to characterize the position of the corresponding text block in the text to be processed; Based on the coordinate information of all the text blocks, the multiple text blocks are sorted to obtain text position features; Based on the text font features and the text position features, the text content features of the text to be processed are obtained.

10. The method according to claim 9, characterized in that, Based on the text to be processed and the text content features, extract the text structure features of the text to be processed, including: Based on the text font features and the font threshold of the preset title level, the text title features are determined; Based on the text location features, paragraph identifier symbols between the text blocks are determined to obtain text paragraph features; The list identifier symbols in the text to be processed are identified to obtain the text list features; The grid line symbols in the text to be processed are identified to obtain text table features; Based on one or more of the text title features, text paragraph features, text list features, and text table features, the text structure features of the text to be processed are obtained.

11. The method according to any one of claims 1 to 10, characterized in that, After processing the format mapping rules, text content features, and text structure features based on the pre-trained language model to generate target text of the second format type, the method further includes: The target text is subjected to the second format syntax check to obtain target format information that does not conform to the second format syntax in the target text; The target format information is processed based on the format syntax of the second format to obtain the optimized target text.

12. The method according to claim 11, characterized in that, The target format information includes one or more of the following: blank lines, indentation, non-compliant characters, and error symbols.

13. The method according to claim 11 or 12, characterized in that, After processing the target format information based on the format syntax of the second format to obtain the optimized target text, the method further includes: The optimized target text is checked for text language based on a language detection algorithm to obtain the text language of the optimized target text. Based on a preset symbol lookup table and the text language, the first symbol in the optimized target text is corrected to obtain target text that conforms to the text language. The first symbol is a text symbol in the optimized target text that does not conform to the text language.

14. A text processing device, characterized in that, include: The acquisition unit is used to acquire the text to be processed in the first format and the format mapping rule. The format mapping rule is obtained by processing the text samples in the first format and the text samples in the second format by a pre-trained language model. The format mapping rule is used to characterize the mapping relationship between the format syntax of the first format and the format syntax of the second format. The format reading and writing difficulty of the first format is greater than that of the second format. An extraction unit is used to extract the text content features of the text to be processed; The extraction unit is used to extract the text structure features of the text to be processed based on the text to be processed and the text content features; The processing unit is used to process the format mapping rules, the text content features, and the text structure features based on the pre-trained language model to generate target text with the second format type.

15. A text processing device, characterized in that, include: Input / output interface, processor, and memory, wherein the memory stores program instructions; The processor is configured to execute program instructions stored in the memory to perform the method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 13.

17. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 13.

Images