Text file processing method, system, computing device, and storage medium
By identifying the language of text files and calling deep learning models and prompt word template libraries for text segmentation and logical verification, the scalability and semantic understanding problems of multilingual and multi-format text file processing in existing technologies are solved, achieving efficient and accurate text information extraction and output.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHONGKE YUNGU TECH
- Filing Date
- 2026-03-31
- Publication Date
- 2026-06-09
Smart Images
Figure CN122174823A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a text file processing method, system, computing device, and storage medium. Background Technology
[0002] Currently, existing technologies face numerous challenges in processing multilingual and multi-format text files. First, existing technologies typically rely on preset rules and template matching, such as regular expressions, to parse text files. This results in problems like hard-coding, high coupling, and poor scalability, and is largely ineffective for free-format, non-standardized text files. Second, existing technologies lack multilingual support for text files, making it difficult to meet internationalization requirements. Summary of the Invention
[0003] The purpose of this application is to provide a text file processing method, system, computing device, and storage medium. By parsing and identifying the language of the text file and calling the corresponding deep learning model for processing, it realizes automated processing of multilingual and multi-format text files, improving the accuracy of information extraction and processing efficiency of text files.
[0004] To achieve the above objectives: In a first aspect, embodiments of this application provide a text file processing method, comprising: receiving and parsing a text file to obtain first text information; identifying at least two first languages contained in the first text information; routing the text region containing the first languages in the first text information to a prompt word template library and a first deep learning model corresponding to the first languages to obtain second text information using a second language, the second text information including multiple field information; performing cross-field logical consistency verification on the multiple field information in the second text information; and outputting the second text information if the verification passes.
[0005] In one embodiment, receiving and parsing a text file to obtain first text information includes: receiving the text file; verifying the received text file, the verification including verifying the file's format, size, and integrity; intercepting the file if it fails verification; matching the corresponding parsing engine according to the type of the text file; and routing the text file to the parsing engine to parse the text file to obtain the first text information.
[0006] In one embodiment, identifying at least two first languages contained in the first text information includes: inputting the first text information into a second deep learning model, wherein the number of parameters of the second deep learning model is smaller than the number of parameters of the first deep learning model; and identifying the language type and number of languages contained in the first text information based on the second deep learning model to determine at least two first languages.
[0007] In one embodiment, the step of routing the text region containing the first language in the first text information to a prompt word template library and a first deep learning model corresponding to the first language to obtain second text information in the second language, the second text information including multiple field information, includes: performing text segmentation on the first text information based on the first language to obtain at least two text regions; routing the at least two text regions to a prompt word template library corresponding to the first language to match prompt instructions, the prompt instructions being used to guide the first deep learning model to perform semantic understanding and field extraction on the at least one text segmentation region, different first languages corresponding to different first deep learning models; determining the corresponding first deep learning model based on the first language, inputting the at least two text regions and the prompt instructions into the first deep learning model; performing semantic understanding and field extraction on the at least two text regions based on the first deep learning model to obtain second text information in the second language, the second text information including multiple field information; wherein, the second language of the second text information output by any of the first deep learning models is the same.
[0008] In one embodiment, the cross-field logical consistency check of multiple fields in the second text information includes: extracting at least two time information from the multiple fields, wherein the at least two time information comes from the same text region in the second text; constructing a timeline between the at least two time information; and analyzing whether there is a time logical conflict in the events corresponding to the time information based on the timeline.
[0009] In one embodiment, the cross-field logical consistency check of multiple fields in the second text information further includes: extracting at least two related fields from the multiple fields, wherein the at least two related fields come from different text regions in the second text and the different text regions have a preset association relationship; determining the semantic matching degree of the at least two related fields based on preset semantic association rules; and determining that the semantic association of the at least two related fields is inconsistent if the semantic matching degree is lower than a preset threshold.
[0010] In one embodiment, the text file processing method further includes: real-time monitoring of performance indicators during the processing of the text file, wherein the performance indicators include at least one of response time, error rate, and concurrency; and automatically sending an alarm message when any of the performance indicators exceeds a preset threshold.
[0011] Secondly, embodiments of this application provide a file processing system, including: The receiving and parsing module is used to receive and parse the text file to obtain the first text information; A language recognition module is used to identify at least two first languages contained in the first text information; The routing processing module is used to route the text region containing the first language in the first text information to the prompt word template library and the first deep learning model corresponding to the first language, so as to obtain the second text information using the second language, wherein the second text information includes multiple field information. The verification module is used to perform logical consistency verification on multiple fields of information in the second text information. The output module is used to output the second text information when the verification module passes the verification.
[0012] Thirdly, embodiments of this application provide a computing device, including: a processor and a memory storing a computer program, wherein when the processor runs the computer program, the above-described text file processing method is implemented.
[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described text file processing method.
[0014] The text file processing method, system, computing device, and storage medium provided in this application receive and parse a text file to obtain first text information; identify at least two first languages contained in the first text information; route the text region containing the first language in the first text information to a prompt word template library and a first deep learning model corresponding to the first language to obtain second text information in the second language, the second text information including multiple field information; perform cross-field logical consistency verification on the multiple field information in the second text; if the verification passes, output the second text information. Thus, this application, by parsing and identifying the language of the text file and calling the corresponding deep learning model for processing, combined with the guidance of prompt word templates, achieves automatic parsing and structuring of multilingual text, and by performing logical consistency verification on the processed text information, significantly improves the efficiency and accuracy of text file processing. Attached Figure Description
[0015] Figure 1 This is a schematic diagram of the architecture of the text file processing method provided in the embodiments of this application.
[0016] Figure 2 This is a flowchart illustrating the text file processing method provided in an embodiment of this application.
[0017] Figure 3 This is a schematic diagram illustrating the specific process of the text file processing method provided in the embodiments of this application.
[0018] Figure 4 This is a schematic diagram of the structure of a text file processing system provided in an embodiment of this application.
[0019] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. Detailed Implementation
[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the scope of the application.
[0021] Existing technologies face challenges in processing multilingual and multi-format text files, especially resumes, in the following specific aspects: First, it has a weak ability to process unstructured content. It cannot effectively handle unstructured content such as handwritten notes, mixed text and images, and long paragraphs of text descriptions in PDF scans. For example, it cannot extract core skills from paragraphs describing "project experience" in text, such as "using Python to implement data crawling," and can only mechanically extract keywords, failing to understand the semantic relationships in the text, thus making it impossible to accurately extract key information from complex descriptions.
[0022] Second, it has poor adaptability and resistance to interference. Minor changes in text format (such as font size, layout errors, or variations in field names) or changes in field position (such as moving the "Education" field from the left to the right) can lead to parsing failures or missed detections. For example, if the "Education" field is placed on the right side of the page, but the rule is preset to extract from the left, it will be missed. This is because existing technologies use hard-coded and highly coupled parsing methods, lacking the ability to adapt to changes in text format.
[0023] Third, semantic understanding is lacking, leading to incomplete information extraction. Existing technologies cannot identify implicit information, abbreviations, or deeper meanings within complex contexts in textual information. For example, they cannot identify "project scale" and "management experience" from "5 years of experience in the internet industry, leading 3 projects worth tens of millions," nor can they interpret "large company" as a leading enterprise or "responsible for user growth" as operational capabilities, resulting in incomplete information extraction.
[0024] Fourth, due to its hard-coded and highly coupled nature, existing technologies are difficult to extend to support new text formats or more languages, making it difficult to meet the international needs of globalized businesses for cross-language text management, and thus have limitations in cross-language and complex format text parsing.
[0025] To address the aforementioned challenges, this application provides a text file processing method. Figure 1 This is a schematic diagram illustrating the architecture of a text file processing method provided in an embodiment of this application. The text file processing method of this application embodiment can be applied to, for example... Figure 1 The AIGC platform (hereinafter referred to as the AIGC platform) is shown. Specifically, it deploys a GPU / CPU server cluster at the underlying level, employing distributed storage, high-availability networks, security protection (firewalls / encryption), and a monitoring and alarm system to support the text file processing workflow. The text file processing workflow includes modules such as interface attachment import, OCR (Optical Character Recognition) attachment parsing, large model processing, and text data output. The interface attachment import module supports HTTP / SDK file upload interfaces to receive text files in various formats, such as PDFs, images, and scanned documents. The OCR attachment parsing module performs text recognition and extraction on the received text files. The recognized content is then cleaned, denoised, and formatted by the data management layer to form structured intermediate results, which are then stored. The data management layer also provides log auditing functionality for easy traceability and querying. Next, the large model processing module processes the recognized content, performing semantic understanding through model scheduling, inference acceleration, and context management to obtain the processed text information. During the large model processing module, the model management layer provides support services, including but not limited to model loading, version control, parameter tuning, inference acceleration, performance monitoring, and resource scheduling. Finally, the processing results are output in plain text, JSON, or XML formats through the text data output module. The output is then sent to downstream systems via API callbacks, output distribution, database writing, or file export through the output distribution layer. The entire process is equipped with log auditing and performance monitoring mechanisms, forming an end-to-end automated text file processing system.
[0026] See Figure 2This application provides a text file processing method, which can be implemented using software and / or hardware, such as a computer or server. In this embodiment, the execution entity of the text file processing method is a server (such as a cloud server). The text file processing method provided in this embodiment includes: Step S101: Receive and parse the text file to obtain the first text information.
[0027] Optionally, an HTTP API interface for receiving text files can be provided. Users can log in to the client and directly call the HTTP API interface via a common network protocol to upload and transmit text files. The second approach is to provide a software development kit (SDK), which encapsulates the HTTP API interface and is designed for specific programming languages such as Python and Java. Users can call simplified functions or methods in the SDK to indirectly call the underlying HTTP API interface, thereby simplifying the integration process, reducing development complexity, improving access efficiency, and adapting to diverse access scenarios.
[0028] Optionally, to parse text files, different parsing engines or parsing methods can be used for different types of text files to extract the first text information in the text file as input data for the deep learning model.
[0029] In one embodiment, receiving and parsing a text file to obtain first text information includes: receiving the text file; verifying the received text file, including verifying the file's format, size, and integrity; intercepting the file if it fails verification; matching the corresponding parsing engine according to the text file's type; and routing the text file to the parsing engine to parse the text file and obtain the first text information.
[0030] Optionally, after receiving a text file uploaded by a user through a preset interface, a file verification process is initiated to ensure the quality and security of the input data. The verification process may include: format verification, checking whether the file extension and content type are whitelisted (e.g., only common document formats such as PDF, JPG, and PNG are allowed; executable files such as .exe are rejected); file size verification, checking whether the file size is within a set threshold (e.g., limiting to 10MB to prevent excessive memory usage by oversized files); and integrity verification, confirming that the file has not been corrupted or tampered with during transmission by verifying the file hash value or parsing the file structure. If a text file fails any of these verifications, an automatic interception will be triggered, terminating subsequent processing of the text file and returning a specific error message to the user. For example, when a user attempts to upload a 15MB TIFF image, the file size verification will detect that it exceeds the limit, and the format verification will identify that it is not a supported common format, thus intercepting the file and returning the message "File too large and format not supported." In this way, filtering out invalid or malicious files reduces the load on subsequent processing modules and avoids system crashes or resource waste caused by processing illegal files. Furthermore, timely feedback guides users to submit compliant text files, improving the security and robustness of file data input and ensuring data quality and processing efficiency in the text file processing process.
[0031] Optionally, an upload task queue can be constructed to queue and schedule multiple received text files.
[0032] When multiple users submit text files simultaneously, a first-in-first-out (FIFO) task queue can be constructed for buffering and queuing. Specifically, through a configured thread pool or distributed worker nodes, multiple parallel processing tasks can be retrieved from the task queue in batches, such as processing 10 text files simultaneously, to achieve concurrent uploads of multiple files. For example, during peak periods, if more than 5,000 text file upload requests are received instantaneously, all upload requests can be queued and wait, while 50 concurrent worker threads are started. Each thread retrieves 10 text files from the queue for parallel processing, thereby alleviating the file upload pressure.
[0033] Optionally, the corresponding parsing engine is matched according to the type of the text file; the text file is routed to the parsing engine to parse the text file and obtain the first text information.
[0034] The AIGC platform integrates OCR parsing engines from multiple vendors, such as Baidu OCR and Tencent OCR. For different types of text files, including PDFs, images, and scanned documents, it automatically routes the data to the appropriate OCR tool using an algorithmic routing strategy. For example, for text files containing complex tables, it calls a high-precision table recognition engine. The parsed text information is then denoised, removing garbled characters and duplicate content that could lead to errors, and undergoing format conversion, such as mapping unstructured OCR results to structured data and converting tables to JSON. Finally, the first text information is output.
[0035] Step S102: Identify at least two first languages contained in the first text information.
[0036] Optionally, before outputting the first text information in the first language as the second text information in the second language, it is necessary to identify the type of the first language. Specifically, the first text information can be input into a lightweight deep learning-based language identification model to identify the multiple first languages contained in the first text information. Understandably, the first language refers to the various natural language types or dialect variations used in the text file. For example, when the second language is Chinese, the non-Chinese languages contained in the first text information are considered the first language.
[0037] In one embodiment, identifying at least two first languages contained in the first text information includes: inputting the first text information into a second deep learning model, wherein the number of parameters of the second deep learning model is smaller than the number of parameters of the first deep learning model; and identifying the language type and number of languages contained in the first text information based on the second deep learning model to determine at least two first languages.
[0038] Optionally, the first deep learning model is used to output second text information in a second language from the input first text information in a first language. The second deep learning model can be a lightweight routing model for language identification. The number of parameters in the second deep learning model is smaller than that in the first deep learning model; for example, the number of parameters in the second deep learning model is approximately 20M. The second deep learning model can receive the first text information as input and output the type of the first language and a confidence score. Furthermore, the second deep learning model can also distinguish variants of the same language in different regions. For example, it can distinguish between the Brazilian variant of Portuguese (pt-BR) and the Portuguese variant (pt-PT). For instance, when a resume is input into the second deep learning model, it identifies the language as "Spanish-Latin American variant" (code es-LA) and gives a confidence score of 0.97. By employing a lightweight language identification model, rapid analysis of the first text information can be achieved.
[0039] Step S103: The text region containing the first language in the first text information is routed to the prompt word template library and the first deep learning model corresponding to the first language to obtain the second text information using the second language. The second text information includes multiple field information.
[0040] Optionally, the text regions in the first text information containing the first language are routed to the corresponding prompt word template library and the first deep learning model based on the type of the first language. The prompt word template library contains a dedicated instruction set for the first language, used to guide the first deep learning model in performing terminology conversion, cultural hints, and field extraction on the first text information. The first deep learning model performs semantic parsing and structured field extraction on the first text information according to the mapping instructions output by the prompt word template library, outputting second text information expressed in the second language and in a structured form. The structured form can be JSON, XML, etc. The second text information can contain multiple fields, such as personal background, educational experience, and work skills.
[0041] In one embodiment, the text region containing the first language in the first text information is routed to the prompt word template library and the first deep learning model corresponding to the first language to obtain second text information using the second language. The second text information includes multiple fields, including: Based on the first language, the first text information is segmented to obtain at least two text regions; At least two text regions are routed to a prompt word template library corresponding to the first language to match prompt instructions. The prompt instructions are used to guide the first deep learning model to perform semantic understanding and field extraction on at least one text segmentation region. Different first languages correspond to different first deep learning models. Based on the first language, determine the corresponding first deep learning model, and input at least two text regions and prompt instructions into the first deep learning model; Based on the first deep learning model, semantic understanding and field extraction are performed on at least two text regions to obtain second text information in the second language. The second text information includes multiple field information. In this case, the second language of the second text information output by any of the first deep learning models is the same.
[0042] Optionally, when determining the distribution area of the first language in the first text information, if multiple first languages are identified in the first text information, the first text information is segmented according to the type of the first language to form multiple text regions containing the same first language. If multiple languages are identified in the same paragraph or sentence of the first text information, it can be regarded as a mixed-language text region to avoid errors in semantic understanding caused by overly fragmented text segmentation.
[0043] Next, the text region is routed to the corresponding prompt template library based on the first language type. The prompt template library contains a pre-set Prompt Template Set for each first language, including industry sub-templates, job sub-templates, and cultural variant sub-templates. The industry sub-templates have pre-defined mapping rules for "term anchors." For example, in the Japanese prompt template library, the input Japanese phrase "プロジェクトマネージャ" can be mapped to the standard tag "IT_Project_Manager," instead of being mistakenly split into the two irrelevant words "プロジェクト+マネージャ," thus improving the accuracy of understanding industry terminology.
[0044] The job templates can be pre-defined with "standardized job title mapping rules," which uniformly map diverse job descriptions expressing the same or similar functions in different languages to preset standard job category tags. For example, in the English template, expressions such as "Software Development Engineer," "Backend Developer," and "Programmer" can be uniformly mapped to the standard tag "Software_Engineer." This mapping overcomes language differences, accurately understands the actual functional category of candidates, avoids misjudgments or classification confusion caused by superficial differences in vocabulary, and thus improves the accuracy of talent-job matching.
[0045] To understand habitual expressions across different cultural backgrounds, "cultural cues" can be pre-set in the cultural variant sub-templates. For example, the Japanese template would emphasize the weight of keywords such as "mid-career recruitment" and "seniority ranking." "Mid-career recruitment" refers to hiring professionals, while "seniority ranking" refers to the connection between seniority and promotion. By pre-setting these keywords and increasing their weight in the Japanese template, the first deep learning model can identify and accurately understand these concepts, correctly mapping them to standardized field labels, effectively avoiding parsing errors or information loss caused by cultural differences.
[0046] Prompt instructions matching the text region are obtained from a prompt word template library to guide the first deep learning model in subsequent semantic understanding and field extraction processing of the text region. Optionally, the corresponding first deep learning model can be called according to the type of the first language, and the text region and the matching prompt instructions can be input together, so that the first deep learning model can perform semantic understanding under the guidance of the prompt instructions, and extract and standardize field information from the text region. Optionally, the corresponding first deep learning model can be called according to the type of the first language and the domain to which the text region belongs. The first deep learning model is trained for specific languages and industries to achieve accurate two-dimensional parsing combining language and industry. For example, for the same Japanese resume, the work experience section can be routed to the "IT-ja" sub-model, and the education background section can be routed to the "Edu-ja" sub-model, avoiding the semantic drift problem caused by using a single model to process text in a single language.
[0047] In one implementation, a preset first deep learning model is invoked to process the first text information based on the identified first language type. The parsed first text information is input into the first deep learning model to obtain structured second text information. For example, when the language of the first text information is detected as Chinese, a BERT model for Chinese is invoked to perform deep semantic parsing on expressions such as "leading three projects worth tens of millions" to extract fields such as "project management experience" and "project scale". When the language of the first text information is detected as English, a RoBERTa model for English is invoked to extract fields such as "leading cross-functional teams" from "ledcross-functional teams". For paragraphs with mixed Chinese and English, an XLM model for cross-language processing is invoked to understand the technical relationship between "using Python to implement data crawling" and "built ETL pipelines using Pandas". In this way, by using deep learning models corresponding to different languages, language characteristics can be accurately matched, the accuracy of key field extraction can be improved, the semantic coherence problem of mixed-language text files can be effectively solved, and the processing efficiency and accuracy of mixed-language text files can be improved.
[0048] The second language of the output text information from all first deep learning models can be set to be the same. In this way, the outputs of all first deep learning models that process the same text file will be expressed in a structured second text information in a uniformly preset second language (such as Chinese), achieving accurate parsing of resumes in multiple languages and formats.
[0049] Step S104: Perform cross-field logical consistency verification on multiple fields in the second text information.
[0050] Step S105: If the verification passes, output the second text information.
[0051] Optionally, based on the semantic understanding of the first text information by the first deep learning model, after outputting the second text information and extracting field information, to ensure the logical consistency of the output second text information, cross-field logical association analysis can be performed on the extracted field information to verify the consistency of the second text information in time series and / or semantic association. The detection results are then marked and the verification results are output. If the verification passes, the second text information is output. If the verification fails, a corresponding prompt message is generated. In this way, by verifying the consistency of the second text information, logical inconsistencies in the text file can be automatically identified and marked, improving the credibility of the output second text information.
[0052] In one embodiment, cross-field logical consistency verification is performed on multiple fields of information in the second text information, including: extracting at least two time information from multiple fields of information, wherein the at least two time information come from the same text region in the second text; constructing a timeline between the at least two time information; and analyzing whether there are time logical conflicts in the events corresponding to the time information based on the timeline.
[0053] Optionally, at least two time-related fields are extracted from the same text region of the second text information, i.e., time information. The same text region refers to independent logical units segmented from the second text information based on language and content, such as work experience, educational background, or project experience. Constructing a timeline involves arranging and associating multiple time points or time periods extracted from the same logical unit according to their natural chronological order to form a continuous sequence. Time-related logical conflicts refer to detecting contradictions that should not exist in the timeline, such as overlaps between time periods, inversions in time order, or contradictions between a specific time point and results calculated based on other fields (such as age). For example, in the work experience text region, the start and end dates of two work experiences are extracted to form discrete time points. These discrete time points are then constructed into a continuous timeline, and logical conflict analysis is performed based on this timeline. The system detects whether there is a time overlap between adjacent work experiences; if the end date of the previous job is later than the start date of the next job, a time conflict is identified and marked as "work experience timeline contradiction." In this way, unreasonable time representations in text files can be effectively identified and marked, thereby improving the credibility and usability of the extracted information.
[0054] In one embodiment, performing cross-field logical consistency verification on multiple fields of information in the second text information further includes: extracting at least two related fields from the multiple fields of information, wherein the at least two related fields come from different text regions in the second text information and the different text regions have a preset association relationship; determining the matching degree of the semantic information of the at least two related fields based on preset semantic association rules; and determining that the semantic association of the at least two related fields is inconsistent if the semantic matching degree is lower than a preset threshold.
[0055] Optionally, at least two fields of information are extracted from different text regions with pre-defined relationships in the second text information for comparative analysis. The pre-defined relationships between different text regions can be reflected as semantically corroborative or logically supportive relationships. Examples include relationships between personal skills and job responsibilities, educational background and work experience, and project roles and project descriptions. Semantic association rules define the logical or quantitative standards for determining the consistency of field content under different association relationships. For example, determining whether the abilities declared in the skills list are reflected in the job responsibilities description, or whether the level of the job title matches the scale of the managed projects. Semantic matching degree can be measured by vectorizing and similarity calculating the text content of the associated fields, or by obtaining a quantitative score after comparing with a pre-defined knowledge graph. For example, the skills description field from the personal profile area and the job responsibilities field from the work experience area are extracted separately. Then, based on pre-defined semantic association rules or industry knowledge graphs, the semantic information of these two fields is calculated for matching degree. When a skill description is "Python programming" but the job description does not mention any Python programming-related technical development content, a semantic conflict is identified and marked as "skills do not match job responsibilities." This can significantly reduce the time cost of manual text review and lower the professional requirements for reviewers.
[0056] In one embodiment, the text file processing method further includes: real-time monitoring of performance indicators during the text file processing process, wherein the performance indicators include at least one of response time, error rate, and concurrency; and automatically sending an alarm message when any performance indicator exceeds a preset threshold.
[0057] During text file processing, the performance metrics of the hardware and software used for text file processing, such as response time, error rate, and concurrency, are monitored in real time to quantitatively evaluate the operational status, efficiency, and stability of the AIGC platform. The collected performance metrics are statistically analyzed, and alarm messages are automatically sent when one or more of them exceed preset thresholds. For example, if the response time of the OCR attachment parsing module increases from the normal 200 milliseconds to 1500 milliseconds, and the concurrent processing count continuously exceeds the preset threshold of 500 documents / second, an alarm is triggered, and a notification message containing specific abnormal metrics is sent to the operations and maintenance personnel. As another example, if the error rate of the large model processing module climbs from 1% to 8% within 5 minutes, exceeding the 5% threshold, the associated GPU server node is flagged, and an error report with stack trace information is generated. In this way, abnormal states of various modules in the AIGC platform can be detected and addressed promptly, ensuring the platform's stable operation and improving the efficiency of text file processing.
[0058] In summary, the text file processing method provided in the above embodiments involves receiving and parsing a text file to obtain first text information; identifying at least two first languages contained in the first text information; routing the text regions containing the first languages in the first text information to a prompt word template library and a first deep learning model corresponding to the first language to obtain second text information in the second language, the second text information including multiple field information; performing cross-field logical consistency verification on the multiple field information in the second text; and outputting the second text information if the verification passes. Thus, this application, by parsing and identifying the language of the text file and calling the corresponding deep learning model for processing, combined with the guidance of prompt word templates, achieves automatic parsing and structuring of multilingual text. Combined with logical consistency verification of the processed text information, this significantly improves the efficiency and accuracy of text file processing.
[0059] like Figure 4 As shown, based on the same inventive concept as the foregoing embodiments, this embodiment of the invention provides a text file processing system 30, comprising: The receiving and parsing module 301 is used to receive and parse the text file to obtain the first text information; The language recognition module 302 is used to identify at least two first languages contained in the first text information; The routing processing module 303 is used to route the text region containing the first language in the first text information to the prompt word template library and the first deep learning model corresponding to the first language, so as to obtain the second text information using the second language. The second text information includes multiple field information. The verification module 304 is used to perform logical consistency verification on multiple fields of information in the second text information. The output module 305 is used to output second text information when the verification module passes the verification.
[0060] In one embodiment, receiving and parsing a text file to obtain first text information includes: receiving the text file; verifying the received text file, including verifying the file's format, size, and integrity; intercepting the file if it fails verification; matching the corresponding parsing engine according to the text file's type; and routing the text file to the parsing engine to parse the text file and obtain the first text information.
[0061] In one embodiment, identifying at least two first languages contained in the first text information includes: inputting the first text information into a second deep learning model, wherein the number of parameters of the second deep learning model is smaller than the number of parameters of the first deep learning model; and identifying the language type and number of languages contained in the first text information based on the second deep learning model to determine at least two first languages.
[0062] In one embodiment, the text region containing the first language in the first text information is routed to a prompt word template library corresponding to the first language and a first deep learning model to obtain second text information in the second language. The second text information includes multiple field information, including: performing text segmentation on the first text information based on the first language to obtain at least two text regions; routing the at least two text regions to a prompt word template library corresponding to the first language to match prompt instructions, the prompt instructions being used to guide the first deep learning model to perform semantic understanding and field extraction on at least one text segmentation region, different first languages corresponding to different first deep learning models; determining the corresponding first deep learning model based on the first language, and inputting the at least two text regions and prompt instructions into the first deep learning model; performing semantic understanding and field extraction on the at least two text regions based on the first deep learning model to obtain second text information in the second language, the second text information including multiple field information; wherein, the second language of the second text information output by any first deep learning model is the same.
[0063] In one embodiment, cross-field logical consistency verification is performed on multiple fields of information in the second text information, including: extracting at least two time information from multiple fields of information, wherein the at least two time information come from the same text region in the second text; constructing a timeline between the at least two time information; and analyzing whether there are time logical conflicts in the events corresponding to the time information based on the timeline.
[0064] In one embodiment, performing cross-field logical consistency verification on multiple fields of information in the second text information further includes: extracting at least two related fields from the multiple fields of information, wherein the at least two related fields come from different text regions in the second text and the different text regions have a preset association relationship; determining the matching degree of the semantic information of the at least two related fields based on preset semantic association rules; and determining that the semantic association of the at least two related fields is inconsistent if the semantic matching degree is lower than a preset threshold.
[0065] In one embodiment, the text file processing system 30 is further configured to: monitor performance indicators during the text file processing process in real time, the performance indicators including at least one of response time, error rate and concurrency; and automatically send alarm information when any performance indicator exceeds a preset threshold.
[0066] For specific implementation details of the embodiments in this application, please refer to the relevant description of the text file processing method, which will not be repeated here.
[0067] Based on the same inventive concept as the foregoing embodiments, this embodiment of the invention provides a computing device, such as... Figure 5 As shown, the computing device includes: a processor 610 and a memory 611 storing computer programs; wherein, Figure 5 The processor 610 shown in the diagram does not indicate that there is only one processor 610, but only indicates the positional relationship of the processor 610 relative to other devices. In practical applications, there can be one or more processors 610; similarly, Figure 5 The memory 611 shown in the diagram has the same meaning, that is, it is only used to indicate the positional relationship of memory 611 relative to other devices. In practical applications, there can be one or more memories 611. When the processor 610 runs the computer program, the text file processing method described above is implemented.
[0068] The computing device may also include at least one network interface 612. The various components of the computing device are coupled together via a bus system 613. It is understood that the bus system 613 is used to implement communication between these components. In addition to a data bus, the bus system 613 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general designated all buses as Bus System 613.
[0069] Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a computer-readable storage medium storing a computer program. The computer-readable storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc. When the computer program stored in the computer-readable storage medium is executed by a processor, it implements the text file processing method described above. For the specific steps implemented when the computer program is executed by the processor, please refer to [link to relevant documentation]. Figures 1-5 The description of the illustrated embodiments will not be repeated here.
[0070] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A text file processing method, characterized in that, include: Receive and parse the text file to obtain the first text information; Identify at least two first languages contained in the first text information; The text region containing the first language in the first text information is routed to the prompt word template library and the first deep learning model corresponding to the first language to obtain the second text information using the second language. The second text information includes multiple field information. Perform cross-field logical consistency verification on multiple fields of information in the second text information; If the verification passes, the second text information will be output.
2. The method according to claim 1, characterized in that, The process of receiving and parsing the text file to obtain the first text information includes: Receive the text file; The received text file is verified, including verification of the file's format, size, and integrity. If the file fails verification, it will be blocked. Match the corresponding parsing engine according to the type of the text file; The text file is routed to the parsing engine to parse the text file and obtain the first text information.
3. The method according to claim 1, characterized in that, The identification of at least two first languages contained in the first text information includes: The first text information is input into the second deep learning model, where the number of parameters in the second deep learning model is smaller than the number of parameters in the first deep learning model. Based on the second deep learning model, the language type and number of languages contained in the first text information are identified to determine at least two first languages.
4. The method according to claim 1, characterized in that, The first text information containing the first language is routed to the prompt word template library and the first deep learning model corresponding to the first language to obtain second text information in the second language. The second text information includes multiple fields, including: Based on the first language, the first text information is segmented to obtain at least two text regions; The at least two text regions are routed to a prompt word template library corresponding to the first language to match prompt instructions. The prompt instructions are used to guide the first deep learning model to perform semantic understanding and field extraction on the at least one text segmentation region. Different first languages correspond to different first deep learning models. Based on the first language, determine the corresponding first deep learning model, and input the at least two text regions and the prompt instruction into the first deep learning model; Based on the first deep learning model, semantic understanding and field extraction are performed on the at least two text regions to obtain second text information in a second language, wherein the second text information includes multiple field information. Wherein, the second language of the second text information output by any of the first deep learning models is the same.
5. The method according to claim 1, characterized in that, The cross-field logical consistency check of multiple fields in the second text information includes: Extract at least two time information items from the plurality of field information, wherein the at least two time information items come from the same text region in the second text; Construct a timeline between the at least two pieces of time information; Based on the timeline, analyze whether there are any time logic conflicts in the events corresponding to the time information.
6. The method according to claim 1, characterized in that, The cross-field logical consistency check of multiple fields in the second text information further includes: Extract at least two related fields from the plurality of field information, wherein the at least two related fields come from different text regions in the second text information and the different text regions have a preset association relationship; Based on preset semantic association rules, determine the degree of matching of the semantic information of the at least two associated fields; If the semantic matching degree is lower than a preset threshold, it is determined that the semantic association of the at least two associated fields is inconsistent.
7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Real-time monitoring of performance metrics during the processing of the text file, wherein the performance metrics include at least one of response time, error rate, and concurrency; When any of the aforementioned performance indicators exceeds a preset threshold, an alarm message will be automatically sent.
8. A text file processing system, characterized in that, include: The receiving and parsing module is used to receive and parse the text file to obtain the first text information; A language recognition module is used to identify at least two first languages contained in the first text information; The routing processing module is used to route the text region containing the first language in the first text information to the prompt word template library and the first deep learning model corresponding to the first language, so as to obtain the second text information using the second language, wherein the second text information includes multiple field information. The verification module is used to perform logical consistency verification on multiple fields of information in the second text information. The output module is used to output the second text information when the verification module passes the verification.
9. A computing device, characterized in that, include: A processor and a memory storing a computer program, wherein when the processor runs the computer program, the text file processing method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the text file processing method according to any one of claims 1 to 7.