Text extraction method, device, equipment and storage medium for markup language file

Abstract

The present disclosure provides a text extraction method, device, setting and storage medium for a markup language file, the method comprising: obtaining a markup language file to be processed and the file size of the markup language file to be processed; judging whether the markup language file to be processed is a large file based on the file size and the current system remaining memory; in the case where the markup language file to be processed is judged to be a large file, performing streaming segmentation content reading and text extraction on the markup language file to be processed, and obtaining text extraction results corresponding to the content of each segment constituting the markup language file to be processed; sending the text extraction results corresponding to the content of each segment to a target server in a streaming segmentation manner, so that the target server combines the text extraction results corresponding to the content of each segment, and obtains a text file corresponding to the markup language file to be processed. By performing streaming segmentation processing on large files, the extraction efficiency of text information can be improved, and errors in the extraction of text information can be avoided.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data processing, and in particular to a text extraction method, device, equipment and storage medium for a markup language file. Background Art

[0002] Markup language files are widely used in various applications, such as XML files, HTML files, etc. These markup language files contain a large amount of tag information and text information. In some scenarios, it is necessary to extract the text information in the markup language file to obtain a text file containing only the text information. At present, when extracting text information from a markup language file, the entire markup language file is usually taken as a whole to extract the text information, which results in low efficiency and error-prone extraction of text information when the markup language file is relatively large. Summary of the invention

[0003] In view of this, the present disclosure proposes a text extraction method, device, equipment and storage medium for markup language files, which can improve the efficiency of text information extraction and avoid memory overflow caused by loading the entire markup language file to be processed into the memory, thereby avoiding errors in text information extraction.

[0004] According to a first aspect of the present disclosure, a method for extracting text from a markup language file is provided, comprising:

[0005] Obtaining a markup language file to be processed and the file size of the markup language file to be processed;

[0006] Based on the file size and the current remaining system memory, determining whether the markup language file to be processed is a large file;

[0007] When it is determined that the markup language file to be processed is a large file, streaming segment content reading and text extraction are performed on the markup language file to be processed to obtain text extraction results corresponding to the segment contents constituting the markup language file to be processed;

[0008] The text extraction results corresponding to the content of each segment are sent to the target server in a streaming segmented manner, so that the target server combines the text extraction results corresponding to the content of each segment to obtain a text file corresponding to the markup language file to be processed.

[0009] In a possible implementation, when streaming segment content reading and text extraction are performed on the markup language file to be processed to obtain text extraction results corresponding to the segment contents constituting the markup language file to be processed, the method includes:

[0010] Get the total system memory, current memory usage data, and the preset number of shards;

[0011] Calculate the current shard size based on the total memory, the current memory usage data, and the number of shards;

[0012] Based on the current moment segment size, the current moment segment content is read from the unread file content in the to-be-processed markup language file;

[0013] Detecting and deleting the mark information in the current moment segment content to obtain a text extraction result corresponding to the current moment segment content;

[0014] The above-mentioned streaming segment content reading and text extraction operations are repeated until the text extraction results corresponding to the segment contents constituting the markup language file to be processed are obtained.

[0015] In a possible implementation, after calculating the current fragment size, the method further includes:

[0016] Determine whether the current shard size is less than or equal to the current system remaining memory;

[0017] When it is determined that the current segment size is less than or equal to the current system remaining memory, an operation is performed to read the current segment content from the unread file content in the markup language file to be processed based on the current segment size.

[0018] In a possible implementation, when it is determined that the current shard size is greater than the current system remaining memory, the following operations are performed:

[0019] Adjust the number of shards according to a preset step size;

[0020] Based on the total memory, the memory usage data at the current moment and the adjusted number of shards, the shard size at the current moment is recalculated until the shard size at the current moment is less than or equal to the current system remaining memory, and then the operation of reading the shard content at the current moment from the unread file content in the markup language file to be processed based on the shard size at the current moment is performed.

[0021] In a possible implementation, when detecting and deleting the mark information in the current moment segment content to obtain the text extraction result corresponding to the current moment segment content, it includes:

[0022] Determine whether there is text content including complete markup information in the current moment segment content;

[0023] When it is determined that there is text content including complete mark information in the current time segment content, the text content including complete mark information is extracted, and the complete mark therein is deleted to obtain a text extraction result corresponding to the current time segment content.

[0024] In a possible implementation manner, when it is determined that there is no text content including complete markup information in the current time slice content, the following operations are performed:

[0025] Determine whether the current shard size is less than or equal to the current system remaining memory;

[0026] When it is determined that the current moment segment size is less than or equal to the current system remaining memory, based on the current moment segment size, read the next moment segment content from the to-be-processed markup language file;

[0027] Combining the next-time segment content with the current-time segment content in order to obtain the next-time segment content;

[0028] Determine whether there is text content including complete markup information in the next moment segment content;

[0029] When it is determined that there is text content including complete tag information in the next moment segment content, the text content including complete tag information is extracted, and the complete tag therein is deleted to obtain a text extraction result corresponding to the next moment segment content.

[0030] In a possible implementation, when it is determined that the markup language file to be processed is not a large file, all the contents of the markup language file to be processed are directly read, the tag information in the markup language file to be processed is detected and deleted, and a text file corresponding to the markup language file to be processed is obtained.

[0031] According to a second aspect of the present disclosure, a device for extracting text from a markup language file is provided, comprising:

[0032] A file acquisition module, used to acquire a markup language file to be processed and the file size of the markup language file to be processed;

[0033] A file size identification module, used to determine whether the markup language file to be processed is a large file based on the file size and the current remaining system memory;

[0034] A text extraction module is used to read the stream segment content and extract the text of the markup language file to be processed when it is determined that the markup language file to be processed is a large file, so as to obtain the text extraction result corresponding to each segment content constituting the markup language file to be processed;

[0035] The text output module is used to send the text extraction results corresponding to each of the fragment contents to the target server in a streaming fragmented manner, so that the target server combines the text extraction results corresponding to each of the fragment contents to obtain a text file corresponding to the markup language file to be processed.

[0036] According to a third aspect of the present disclosure, a text extraction device for a markup language file is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the method described in the first aspect of the present disclosure.

[0037] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the method described in the first aspect of the present disclosure.

[0038] The present disclosure provides a method, device, setting and storage medium for extracting text from a markup language file, the method comprising: obtaining a markup language file to be processed and the file size of the markup language file to be processed; judging whether the markup language file to be processed is a large file based on the file size and the current system remaining memory; in the case where the markup language file to be processed is judged to be a large file, performing streaming segmentation content reading and text extraction on the markup language file to be processed to obtain text extraction results corresponding to the content of each segment constituting the markup language file to be processed; sending the text extraction results corresponding to the content of each segment to a target server in a streaming segmentation manner, so that the target server combines the text extraction results corresponding to the content of each segment to obtain a text file corresponding to the markup language file to be processed. In the present disclosure, in the case where the markup language file to be processed is a large file, it will be processed by streaming segmentation, thereby improving the extraction efficiency of text information, and avoiding memory overflow caused by loading the entire markup language file to be processed into the memory, thereby avoiding errors in the extraction of text information.

[0039] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0041] Figure 1 A flowchart showing a method for extracting text from a markup language file according to an embodiment of the present disclosure is shown;

[0042] Figure 2An example flow chart showing a method for extracting text from a markup language file according to an embodiment of the present disclosure;

[0043] Figure 3 A schematic block diagram showing a text extraction device for a markup language file according to an embodiment of the present disclosure;

[0044] Figure 4 A schematic block diagram of a device for extracting text from a markup language file according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0045] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0046] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0047] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0048] <Method Example>

[0049] Figure 1 A flowchart of a method for extracting text from a markup language file according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes steps S1100-S1400.

[0050] S1100, obtaining a markup language file to be processed and the file size of the markup language file to be processed, wherein the markup language file to be processed is the markup language file that currently needs to be subjected to text extraction.

[0051] S1200: Based on the file size of the markup language file to be processed and the remaining memory of the current system, determine whether the markup language file to be processed is a large file.

[0052] It should be noted here that the remaining memory of the system executing the method disclosed herein changes in real time. Therefore, each time the system remaining memory is used, it is necessary to calculate the system remaining content at the current moment (hereinafter referred to as the current system remaining memory) based on the system's total memory and the memory usage data at the current moment.

[0053] In a possible implementation, the current memory usage data may include the currently occupied memory and the number of threads currently executing the processing task. In this embodiment, when calculating the current system remaining memory based on the system total memory and the current memory usage data, it can be calculated by the following calculation formula.

[0054]

[0055] In the formula, Mr is the current system remaining memory, Mt is the system total memory, Mo is the currently occupied memory, m is the number of threads currently executing the processing task, and a is a preset ratio value, wherein the ratio value can range from 70% to 90%, and preferably, the ratio value can be set to 80%.

[0056] After calculating the current system remaining memory, it is possible to determine whether the markup language file to be processed is a large file based on the file size of the markup language file to be processed and the current system remaining memory. Specifically, when the file size of the markup language file to be processed is greater than or equal to the current system remaining memory, it is determined that the markup language file to be processed is a large file; when the file size of the markup language file to be processed is less than the current system remaining memory, it is determined that the markup language file to be processed is not a large file, that is, the markup language file to be processed is a small file.

[0057] When it is determined that the markup language file to be processed is not a large file, it means that the remaining memory of the current system can meet the text extraction and processing requirements of the entire markup language file to be processed. Therefore, the entire content of the markup language file to be processed can be directly read into the memory, and the tag information in the markup language file to be processed can be detected and deleted to obtain the text file corresponding to the markup language file to be processed, and then the obtained text file is sent to the target server, so that the target server performs subsequent operations based on the extracted text file.

[0058] If it is determined that the markup language file to be processed is a large file, it means that the remaining memory of the current system cannot meet the text extraction processing requirements of the entire markup language file to be processed. At this time, if the entire markup language file to be processed is loaded into the memory, it will cause memory overflow, resulting in the inability to perform the text extraction processing operation normally, thereby causing a text extraction error. In order to solve this problem, if it is determined that the markup language file to be processed is a large file, step S1300 will be executed.

[0059] S1300, when it is determined that the markup language file to be processed is a large file, stream segment content reading and text extraction are performed on the markup language file to be processed to obtain text extraction results corresponding to the segment contents constituting the markup language file to be processed. Specifically, the following steps may be included:

[0060] First, the total memory of the system, the current memory usage data and the preset number of shards are obtained. The preset number of shards can be set according to the specific application scenario. For example, the number of shards can be set to 2 in some scenarios.

[0061] Second, based on the total system memory, the current memory usage data and the preset number of shards, the current shard size is calculated, wherein the current shard size represents the intercepted size of the shard content at the current moment.

[0062] In an embodiment where the memory usage data at the current moment includes the currently occupied memory and the number of threads currently executing the processing task, the slice size at the current moment may be expressed as follows:

[0063]

[0064] Where n is the current shard size, and c is the preset number of shards.

[0065] For example, when the total system memory is 4G, the currently occupied memory is 3G+512M, the number of threads currently executing the processing task is 8, the preset number of shards is 2, and the preset ratio value is 90%, the current shard size = (1024*4*90% - 1024*3-512) / 8 / 2 = 6.4M. Among them, 3G of the currently occupied memory is the memory occupied by other application programs, and 512M is the memory occupied by the application program executing the method disclosed in the present invention.

[0066] Third, based on the current segment size, the current segment content is read from the unread file content of the markup language file to be processed. Specifically, when the current segment size is n, the file content of the file size n is read in sequence from the head end of the unread file content of the markup language file to be processed as the current segment content. For example, when the current segment size is 6.4M, the file content of 6.4M is read in sequence from the head end of the unread file content of the markup language file to be processed as the current segment content.

[0067] After reading the current shard content, it is imported into the memory so that the processor can execute the fourth step.

[0068] Fourth, detect and delete the markup information in the current moment segment content to obtain the text extraction result corresponding to the current moment segment content. The specific steps are as follows:

[0069] First, it is determined whether there is text content including complete tag information in the current moment segment content. Specifically, after the current moment segment content is imported into the memory, the tags in the current moment segment content are traversed in order, and when the start tag is detected, the end tag matching the start tag is continued to be searched in the current moment segment content, and when the end tag matching the start tag is found, it is determined that there is text content including complete tag information in the current moment segment content, wherein the text content including complete tag information is the text content starting with the start tag and ending with the end tag matching the start tag.

[0070] Secondly, when it is determined that there is text content including complete tag information in the current time segment content, the text content including complete tag information is extracted from the memory, and the complete tag therein is deleted to obtain the text extraction result corresponding to the current time segment content. Among them, the start tag in the text content and the end tag matching the start tag become a complete tag.

[0071] Fifth, by repeating the above-mentioned streaming segment content reading and text extraction operations, the text extraction results corresponding to the segment contents constituting the markup language file to be processed can be obtained.

[0072] It should be noted here that in some cases, there is no text content including complete tag information in the current moment segment content. For example, when the tags in the current moment segment content are traversed in order, only the start tag can be detected, and the end tag matching the start tag cannot be detected. In this case, the tag information in the current moment segment content is not detected and deleted for the time being, but the following operations are performed: determine whether the current moment segment size is less than or equal to the current system remaining memory. When it is determined that the current moment segment size is less than or equal to the current system remaining memory, based on the current moment segment size, read the next segment content from the unread file content of the markup language file to be processed into the memory; combine the next segment content with the current moment segment content in order to obtain the next moment segment content; determine whether there is text content including complete tag information in the next moment segment content; when it is determined that there is text content including complete tag information in the next moment segment content, extract the text content including complete tag information in the next moment segment content, delete the complete tag therein, and obtain the text extraction result corresponding to the next moment segment content. The specific implementation steps are as mentioned above and will not be repeated here.

[0073] In a possible implementation, after calculating the current shard size, the following steps are further included:

[0074] First, determine whether the current shard size is less than or equal to the current system remaining memory.

[0075] Second, when it is determined that the current shard size is less than or equal to the current system remaining memory, an operation based on the current shard size is performed to read the current shard content from the markup language file to be processed.

[0076] In this embodiment, when it is determined that the current shard size is greater than the current system remaining memory, the following operations are further performed:

[0077] First, adjust the number of shards according to the preset step size. The preset step size can be the nth power of 2, the value range of n is 0 to max, and the value range of max can be set to 1 to 11. Specifically, the sum of the preset number of shards and the preset step size is used as the adjusted shard data. For example, if the preset number of shards is 2 and the preset step size is 1, the sum of the preset number of shards and the preset step size 3 is used as the adjusted number of shards.

[0078] Second, based on the total memory, the current memory usage data and the adjusted number of shards, the current shard size is recalculated until the current shard size is less than or equal to the current system remaining memory, and then the operation of reading the current shard content from the markup language file to be processed based on the current shard size is executed.

[0079] S1400, the text extraction results corresponding to each fragment content are sent to the target server in a streaming fragmented manner, so that the target server combines the text extraction results corresponding to each fragment content to obtain a text file corresponding to the markup language file to be processed. Specifically, each time a text extraction result of a fragment content is generated, it is sent to the target server, and each time the target server receives a text extraction result, it splices the newly received text extraction result at the end of the previous text extraction result until all text extraction results are spliced ​​to obtain a text file corresponding to the markup language file to be processed.

[0080] The present disclosure provides a text extraction method for a markup language file, comprising: obtaining a markup language file to be processed and the file size of the markup language file to be processed; judging whether the markup language file to be processed is a large file based on the file size and the current system remaining memory; in the case where the markup language file to be processed is judged to be a large file, performing streaming segmentation content reading and text extraction on the markup language file to be processed, and obtaining text extraction results corresponding to the content of each segment constituting the markup language file to be processed; sending the text extraction results corresponding to the content of each segment to a target server in a streaming segmentation manner, so that the target server combines the text extraction results corresponding to the content of each segment to obtain a text file corresponding to the markup language file to be processed. In the present disclosure, in the case where the markup language file to be processed is a large file, it will be streamed and segmented, that is, only a part of the content is read into the memory each time according to the memory situation of the system, and when there is text content including complete tag information in the content stored in the memory, it will be extracted from the memory to remove the complete tag, and the text extraction result obtained after the complete tag is removed will be sent to the target server, thereby releasing a part of the system memory in time and continuing the next streaming segmentation processing. Since the size of the text loaded into the system memory each time is determined according to the current system memory usage, text extraction failure caused by memory overflow can be avoided. At the same time, only part of the text content is processed each time, thereby improving text extraction efficiency.

[0081] It should be noted here that, referring to the inventive concept of the present disclosure, text extraction can also be performed on files in other formats such as PDF, which will not be described in detail here.

[0082] <Method Example>

[0083] Figure 2 An example flow chart of a method for extracting text from a markup language file according to an embodiment of the present disclosure is shown. Figure 2 As shown, the method includes steps S2100-S3200.

[0084] S2100: Obtain a markup language file to be processed and the file size of the markup language file to be processed.

[0085] S2200, based on the file size and the current remaining system memory, determine whether the markup language file to be processed is a large file, if it is not a large file, execute step S2300, if it is a large file, execute step S2500.

[0086] S2300, reading all the contents of the markup language file to be processed into the memory, detecting and deleting the tag information in the markup language file to be processed, and obtaining a text file corresponding to the markup language file to be processed.

[0087] S2400, sending a text file corresponding to the markup language file to be processed to a target server.

[0088] S2500 obtains the total memory of the system, the memory usage data at the current moment, and the preset number of shards, and calculates the shard size at the current moment.

[0089] S2600, determine whether the current shard size is less than or equal to the current system remaining memory. If it is determined that the current shard size is greater than the current system remaining memory, execute step S2700; if it is determined that the current shard size is less than or equal to the current system remaining memory, execute step S2800.

[0090] S2700, adjusting the number of shards according to a preset step size, and returning to step S2500, recalculating the shard size at the current moment based on the adjusted number of shards.

[0091] S2800: Based on the current segment size, read the current segment content from the unread file content in the markup language file to be processed into the system memory.

[0092] S2900, determine whether there is text content including complete tag information in the current time segment content, and when it is determined that there is no text content including complete tag information in the current time segment content, return to execute step S2600, continue to extract the next segment content, and store the extraction result of the next segment content in the memory, and combine it with the content stored in the memory to obtain the next segment content, until there is text content including complete tag information in the next time segment content, execute step S3000, and when it is determined that there is text content including complete tag information in the current time segment content, execute step S3000.

[0093] S3000, extracting the text content including the complete tag information from the memory, deleting the complete tag therein, and obtaining the text extraction result corresponding to the segment content at the current moment.

[0094] S3100, sending the text extraction result corresponding to the segment content at the current moment to the target server.

[0095] S3200, determining whether there is any unread file content in the markup language file to be processed, and if so, returning to step S2500, and if not, terminating the process.

[0096] <Device Example>

[0097] Figure 3 A schematic block diagram of a text extraction device for a markup language file according to an embodiment of the present disclosure is shown. Figure 3 As shown, the device 100 includes:

[0098] The file acquisition module 110 is used to acquire the markup language file to be processed and the file size of the markup language file to be processed;

[0099] The file size identification module 120 is used to determine whether the markup language file to be processed is a large file based on the file size and the current remaining system memory;

[0100] The text extraction module 130 is used to read the stream segment content and extract the text of the markup language file to be processed when it is determined that the markup language file to be processed is a large file, and obtain the text extraction results corresponding to the segment contents constituting the markup language file to be processed;

[0101] The text output module 140 is used to send the text extraction results corresponding to each fragment content to the target server in a streaming fragmented manner, so that the target server combines the text extraction results corresponding to each fragment content to obtain a text file corresponding to the markup language file to be processed.

[0102] <Equipment Embodiment>

[0103] Figure 4 A schematic block diagram of a text extraction device for a markup language file according to an embodiment of the present disclosure is shown. Figure 4 As shown, the text extraction device 200 for markup language files includes: a processor 210 and a memory 220 for storing executable instructions of the processor 210. The processor 210 is configured to implement any of the above-mentioned text extraction methods for markup language files when executing the executable instructions.

[0104] Here, it should be noted that the number of processors 210 may be one or more. Meanwhile, in the text extraction device 200 of the markup language file of the embodiment of the present disclosure, an input device 230 and an output device 240 may also be included. The processor 210, the memory 220, the input device 230 and the output device 240 may be connected via a bus or in other ways, which are not specifically limited here.

[0105] The memory 220 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and various modules, such as the program or module corresponding to the text extraction method of the markup language file of the embodiment of the present disclosure. The processor 210 executes various functional applications and data processing of the text extraction device 200 of the markup language file by running the software program or module stored in the memory 220.

[0106] The input device 230 may be used to receive input numbers or signals. The signals may be key signals related to user settings and function control of the device / terminal / server. The output device 240 may include display devices such as display screens.

[0107] <Storage Medium Embodiment>

[0108] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is also provided, on which computer program instructions are stored. When the computer program instructions are executed by the processor 210, any of the above-mentioned methods for extracting text from a markup language file is implemented.

[0109] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A text extraction method for a markup language file, characterized in that: include: Obtaining a markup language file to be processed and the file size of the markup language file to be processed; Based on the file size and the current remaining system memory, determining whether the markup language file to be processed is a large file; When it is determined that the markup language file to be processed is a large file, streaming segment content reading and text extraction are performed on the markup language file to be processed to obtain text extraction results corresponding to the segment contents constituting the markup language file to be processed; Sending the text extraction results corresponding to the content of each segment to the target server in a streaming segmented manner, so that the target server combines the text extraction results corresponding to the content of each segment to obtain a text file corresponding to the markup language file to be processed; When streaming segment content reading and text extraction are performed on the markup language file to be processed to obtain text extraction results corresponding to the segment contents constituting the markup language file to be processed, the method includes: Get the total system memory, current memory usage data, and the preset number of shards; Calculate the current moment slice size based on the total memory, the current moment memory usage data and the number of slices, wherein the current moment slice size represents the intercepted size of the current moment slice content; Based on the current moment segment size, the current moment segment content is read from the unread file content in the to-be-processed markup language file; Detecting and deleting the mark information in the current moment segment content to obtain a text extraction result corresponding to the current moment segment content; Repeat the above streaming segment content reading and text extraction operations until the text extraction results corresponding to the segment contents constituting the markup language file to be processed are obtained; When detecting and deleting the mark information in the current moment segment content to obtain the text extraction result corresponding to the current moment segment content, it includes: Determine whether there is text content including complete markup information in the current moment segment content; When it is determined that the current moment segment content contains text content including complete tag information, the text content including the complete tag information is extracted, and the complete tag is deleted therein to obtain a text extraction result corresponding to the current moment segment content; When it is determined that there is no text content including complete markup information in the current time slice content, the following operations are performed: Determine whether the current shard size is less than or equal to the current system remaining memory; When it is determined that the current moment segment size is less than or equal to the current system remaining memory, based on the current moment segment size, read the next moment segment content from the to-be-processed markup language file; Combining the next-time segment content with the current-time segment content in order to obtain the next-time segment content; Determine whether there is text content including complete markup information in the next moment segment content; When it is determined that there is text content including complete tag information in the next moment segment content, the text content including complete tag information is extracted, and the complete tag therein is deleted to obtain a text extraction result corresponding to the next moment segment content.

2. The method according to claim 1, characterized in that After calculating the current shard size, the method further includes: Determine whether the current shard size is less than or equal to the current system remaining memory; When it is determined that the current segment size is less than or equal to the current system remaining memory, an operation is performed to read the current segment content from the unread file content in the markup language file to be processed based on the current segment size.

3. The method according to claim 2, characterized in that When it is determined that the current shard size is greater than the current system remaining memory, the following operations are performed: Adjust the number of shards according to a preset step size; Based on the total memory, the memory usage data at the current moment and the adjusted number of shards, the shard size at the current moment is recalculated until the shard size at the current moment is less than or equal to the current system remaining memory, and then the operation of reading the shard content at the current moment from the unread file content in the markup language file to be processed based on the shard size at the current moment is performed.

4. The method according to any one of claims 1 to 3, characterized in that When it is determined that the markup language file to be processed is not a large file, all the contents of the markup language file to be processed are directly read, the tag information in the markup language file to be processed is detected and deleted, and a text file corresponding to the markup language file to be processed is obtained.

5. A text extraction device for a markup language file, characterized in that: include: A file acquisition module, used to acquire a markup language file to be processed and the file size of the markup language file to be processed; A file size identification module, used to determine whether the markup language file to be processed is a large file based on the file size and the current remaining system memory; A text extraction module is used to read the stream segment content and extract the text of the markup language file to be processed when it is determined that the markup language file to be processed is a large file, so as to obtain the text extraction result corresponding to each segment content constituting the markup language file to be processed; A text output module, used for sending the text extraction results corresponding to the content of each segment to the target server in a streaming segmented manner, so that the target server combines the text extraction results corresponding to the content of each segment to obtain a text file corresponding to the markup language file to be processed; The text extraction module, when performing streaming segment content reading and text extraction on the markup language file to be processed to obtain text extraction results corresponding to the segment contents constituting the markup language file to be processed, is specifically used to: Get the total system memory, current memory usage data, and the preset number of shards; Calculate the current moment slice size based on the total memory, the current moment memory usage data and the number of slices, wherein the current moment slice size represents the intercepted size of the current moment slice content; Based on the current moment segment size, the current moment segment content is read from the unread file content in the to-be-processed markup language file; Detecting and deleting the mark information in the current moment segment content to obtain a text extraction result corresponding to the current moment segment content; Repeat the above streaming segment content reading and text extraction operations until the text extraction results corresponding to the segment contents constituting the markup language file to be processed are obtained; The text extraction module, when detecting and deleting the mark information in the current moment segment content to obtain the text extraction result corresponding to the current moment segment content, is specifically used to: Determine whether there is text content including complete markup information in the current moment segment content; When it is determined that the current time segment content contains text content including complete mark information, the text content including complete mark information is extracted, and the complete mark therein is deleted to obtain a text extraction result corresponding to the current time segment content; When it is determined that there is no text content including complete markup information in the current moment segment content, the following operations are performed: Determine whether the current shard size is less than or equal to the current system remaining memory; When it is determined that the current moment segment size is less than or equal to the current system remaining memory, based on the current moment segment size, read the next moment segment content from the to-be-processed markup language file; Combining the next-time segment content with the current-time segment content in order to obtain the next-time segment content; Determine whether there is text content including complete markup information in the next moment segment content; When it is determined that there is text content including complete tag information in the next moment segment content, the text content including complete tag information is extracted, and the complete tag therein is deleted to obtain a text extraction result corresponding to the next moment segment content.

6. A text extraction device for a markup language file, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 4 when executing the executable instructions.

7. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images