Text field extraction method, device, medium and equipment

Abstract

The present invention discloses a text field extraction method, device, medium and equipment, comprising: first extracting multiple elements from a sample text, each element including an alias, content, a starting position and an ending position of the content. Then, according to the starting and ending positions of each element, the relevant pre- and post-contents are determined, and added to the corresponding element to more accurately locate the field. Then, the alias, pre-content and post-content in the element set are extracted to form a rule set for describing the position and related information of each element in the sample text. Finally, the text to be processed is obtained, and according to the pre- and post-content in the rule set, the matching content is intercepted as the target field, and the middle field thereof is extracted and assigned a corresponding alias. The present invention does not need to write a complex regular expression for each type of field, and utilizes the element information in the sample text to extract the field, thereby reducing the development workload and the possibility of error, while improving the processing efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field, and in particular to a text field extraction method, device, medium and equipment. Background Art

[0002] As the degree of enterprise informatization continues to deepen, the complexity of information systems continues to increase, generating a large amount of text data, which makes it increasingly difficult for auditors to find evidence in disorganized texts. The text analysis system can structure the disordered text, extract the characteristic fields, and use the analysis capabilities of big data to help auditors find problems in massive texts.

[0003] However, traditional text parsing and field extraction methods require programmers to write complex regular expressions for each type of field, which results in low processing efficiency and huge resource consumption when facing large amounts of text data. Summary of the invention

[0004] Based on this, it is necessary to provide a text field extraction method, device, medium and equipment to solve the problems of low processing efficiency and huge resource consumption in the prior art.

[0005] A text field extraction method, the method comprising:

[0006] Acquire a sample text, and extract elements from the sample text to obtain an element set including a plurality of elements; wherein each element includes an alias, content, a content start position, and a content end position;

[0007] In the example text, determining the preceding content associated with each content based on the starting position of the content, and determining the following content associated with each content based on the ending position of the content, and adding the preceding content and following content of each content to the corresponding element;

[0008] In the sample text, extract the alias, the preceding content and the following content of each element in the element set to form a rule set;

[0009] Obtain a text to be processed, extract the content in the text to be processed that is identical to the pre-content and post-content of the rule set as the target pre-content and target post-content, extract the field between the target pre-content and the target post-content, and assign a corresponding alias based on the rule set; wherein the text type of the text to be processed is consistent with that of the example text.

[0010] In one embodiment, determining the preceding content associated with each content based on the starting position of the content in the example text includes:

[0011] In the example text, boundary judgment is performed character by character from the starting position of each content to the starting position of the text, and the characters from the last character that meets the boundary judgment condition to before the starting position of the content are used as the preceding content; wherein the boundary judgment condition includes that the character is not a space and is not content.

[0012] In one embodiment, determining the subsequent content associated with each content based on the content termination position includes:

[0013] The boundary judgment is performed character by character from the content end position of each content to the text end position of the text, and the characters from the last character that meets the boundary judgment condition to the content end position are taken as the post-content.

[0014] In one embodiment, in the sample text, extracting the alias, the preceding content and the following content of each element in the element set to form a rule set includes:

[0015] In the example text, the leading content in each element is simplified into the leading simplified content, and the trailing content in each element is simplified into the trailing simplified content; wherein the leading simplified content and the trailing simplified content include unique simplified characters and serial numbers;

[0016] Extract the alias, the preceding simplified content, and the following simplified content of each element to combine and generate an analytical expression;

[0017] The alias in the parsing expression and the simplified characters in the preceding simplified content and the following simplified content are extracted based on a preset regular expression to form a rule set.

[0018] In one embodiment, the step of simplifying the preceding content in each element into the preceding simplified content includes:

[0019] In the example text, characters are added to the character group one by one from the end position of the preceding content of each element to the starting position of the text, and the quantity of each character group is judged. The first character group that meets the quantity judgment condition is used as the first simplified character group, and the part of the preceding content except the first simplified character group is replaced with different preset serial numbers, and the combination of the first simplified character group and the different preset serial numbers is used as the preceding simplified content; wherein the quantity judgment condition is that the number of character groups in the example text is 1.

[0020] In one embodiment, the step of simplifying the post-positioned content in each element into post-positioned simplified content includes:

[0021] Add characters one by one to the character group from the starting position of the post-content of each element to the ending position of the text, perform quantity judgment on each character group, take the first character group that meets the quantity judgment condition as the second simplified character group, replace the part of the post-content except the second simplified character group with different preset serial numbers, and take the combination of the second simplified character group and the different preset serial numbers as the post-simplified content.

[0022] In one embodiment, the method further comprises:

[0023] The characters in the sample text that are not extracted and fall between two spaces are regarded as unextracted content, and the space format of each unextracted content is determined based on the starting position and the ending position of the unextracted content in the sample text;

[0024] Obtain the alias of each unextracted content, and add the space format and alias of each unextracted content as a set of new elements to the rule set;

[0025] Fields in the text to be processed that have the same space format as the rule set are intercepted, and corresponding aliases are assigned based on the rule set.

[0026] A text field extraction device, the device comprising:

[0027] An element extraction module is used to obtain a sample text and extract elements from the sample text to obtain an element set including multiple elements; wherein each element includes an alias, content, a content start position, and a content end position; in the sample text, a pre-content associated with each content is determined based on the content start position, and a post-content associated with each content is determined based on the content end position, and the pre-content and post-content of each content are added to the corresponding element;

[0028] A rule forming module, used for extracting the alias, the preceding content and the following content of each element in the element set in the sample text to form a rule set;

[0029] A field extraction module is used to obtain a text to be processed, intercept the content in the text to be processed that is identical to the pre-content and post-content of a rule set as the target pre-content and target post-content, extract the field between the target pre-content and the target post-content, and assign a corresponding alias based on the rule set; wherein the text to be processed is consistent with the text type of the example text.

[0030] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the above-mentioned text field extraction method.

[0031] A text field extraction device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned text field extraction method.

[0032] The present invention provides a text field extraction method, device, medium and equipment. First, multiple elements are extracted from a sample text, each element including an alias, content, a content start position and a content end position. The extraction of these elements can be used for rule generation in subsequent steps and field extraction of a text to be processed. Then, according to the content start position and content end position of each element, the leading content and trailing content associated with each content are determined, and they are added to the corresponding element. This helps to locate the position of each field more accurately. The alias, leading content and trailing content of each element in the extracted element set can form a rule set. These rules are used to describe the position and related information of each element in the sample text. Finally, the text to be processed is obtained, and according to the leading content and trailing content in the rule set, the content matching the rule in the text to be processed is intercepted as a target field. Then, the middle field is extracted from the target field, and the corresponding alias is assigned according to the rule set. The beneficial effect of this method is that it does not need to write a complex regular expression for each type of field, but uses the existing element information in the sample text to extract the field, which can reduce the workload of developers and reduce the possibility of error. In addition, since the element information in the sample text is utilized during the processing, the processing efficiency will also be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] in:

[0035] Figure 1 It is a flowchart of the text field extraction method in the first embodiment;

[0036] Figure 2 A schematic diagram of the process of forming a set of rules;

[0037] Figure 3 This is a performance comparison chart between this method and traditional regular expressions;

[0038] Figure 4 It is a flowchart of the text field extraction method in the second embodiment;

[0039] Figure 5 It is a structural schematic diagram of a text field extraction device;

[0040] Figure 6 Block diagram of the device for text field extraction. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0043] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0044] like Figure 1 As shown, Figure 1 1 is a flowchart of the text field extraction method in the first embodiment. The steps provided by the text field extraction method in the first embodiment include:

[0045] S101, obtaining a sample text, and extracting elements from the sample text to obtain an element set containing multiple elements.

[0046] Specifically, the sample text is sample data used for analysis and definition of extraction rules, which is called SmapleLog in this article. The text usually contains multiple records with similar structures. By parsing and extracting elements from the sample text, rules and templates for processing the same type of data can be established. Optionally, the sample text can be log text within a period of time.

[0047] Element extraction is the process of identifying and extracting meaningful information units (i.e. elements) from sample text. This process can use string matching or other parsing methods. All extracted elements form an element set, which is called element-list in this article. Each element contains the following attributes:

[0048] Alias ​​(label): Name the element to identify the element type.

[0049] Content: The actual text data of an element.

[0050] Content start position (content-start): The starting position of the content in the sample text.

[0051] Content-end: The end position of the content in the sample text.

[0052] For example, the sample text is:

[0053] <5> time:2010-08-13 15:35:38;

[0054] danger_degree:1; breaking_sighn:0; event:

[40432] HTTP; src_addr:192.168.10.238; src_port:2724; dst_addr:202.111.185.5; dst_port:80; proto:TCP.HTT

[0055] The first element extracted is represented as:

[0056] element{

[0057] content 2010-08-13 15:35:38

[0058] content-start 9

[0059] content-end 27

[0060] label log_time}

[0061] Similarly, continue to extract the remaining elements and add them to the element set element-list. Furthermore, the element set element-list is reordered from small to large according to the starting position of each element content-start.

[0062] S102, in the example text, determine the preceding content associated with each content based on the starting position of the content, and determine the following content associated with each content based on the ending position of the content, and add the preceding content and following content of each content to the corresponding element.

[0063] Among them, the leading content refers to the character sequence in a specific range before the starting position of the content in the text, and the trailing content refers to the character sequence in a specific range after the ending position of the content in the text. This part of the content provides contextual information of the element, which helps to understand and parse the element in the context.

[0064] In a specific embodiment, the specific steps of determining the preceding content associated with each content based on the starting position of the content include: in the example text, starting from the starting position of the content of each content to the starting position of the text, boundary judgment is performed character by character, and the characters from the last character that meets the boundary judgment condition to before the starting position of the content are used as the preceding content; wherein the boundary judgment condition includes that the character is not a space and is not content.

[0065] For example, the sample text is:

[0066] <5> time:2010-08-13 15:35:38;

[0067] danger_degree:1; breaking_sighn:0; event:

[40432] HTTP; src_addr:192.168.10.238; src_port:2724; dst_addr:202.111.185.5; dst_port:80; proto:TCP.HTT

[0068] The content start position (content-start) is 9. Starting from position 9, check each character forward:

[0069] Position 8: Character:, satisfies the boundary judgment condition.

[0070] Position 7: Character e, which meets the boundary judgment condition.

[0071] Position 6: Character m, which meets the boundary judgment condition.

[0072] Position 5: Character i, which meets the boundary judgment condition.

[0073] Position 4: Character t, which meets the boundary judgment condition.

[0074] Position 3: Character >, which satisfies the boundary judgment condition.

[0075] Position 2: Character 5, which meets the boundary judgment condition.

[0076] Position 1: Character <, which satisfies the boundary judgment condition.

[0077] Therefore, all characters from the last character that meets the boundary judgment condition to the content start position (9) are " <5> time:” as the preceding content, this article calls it content-start-pre.

[0078] In a specific embodiment, the specific steps of determining the post-content associated with each content based on the content end position include: performing boundary judgment character by character from the content end position of each content to the text end position of the text, and taking the characters from the last character that meets the boundary judgment condition to the content end position as the post-content.

[0079] Exemplary, exemplary, example text is:

[0080] <5> time:2010-08-13 15:35:38;

[0081] danger_degree:1; breaking_sighn:0; event:

[40432] HTTP; src_addr:192.168.10.238; src_port:2724; dst_addr:202.111.185.5; dst_port:80; proto:TCP.HTT

[0082] The content end position (content-end) is 27. Starting from position 27, check each character backwards:

[0083] Position 28: Character ;, does not meet the boundary judgment condition.

[0084] Position 29: Character d, which meets the boundary judgment condition.

[0085] Position 30: Character a, which meets the boundary judgment condition.

[0086] Position 31: Character n, which meets the boundary judgment condition.

[0087] Position 32: Character g, which meets the boundary judgment condition.

[0088] Position 33: Character e, which meets the boundary judgment condition.

[0089] Position 34: Character r, which meets the boundary judgment condition.

[0090] Position 35: Character _, which meets the boundary judgment condition.

[0091] Position 36: Character d, which meets the boundary judgment condition.

[0092] Position 37: Character e, which meets the boundary judgment condition.

[0093] Position 38: Character g, which meets the boundary judgment condition.

[0094] Position 39: Character r, which meets the boundary judgment condition.

[0095] Position 40: Character e, which meets the boundary judgment condition.

[0096] Position 41: Character e, which meets the boundary judgment condition.

[0097] Position 42: Character;, satisfies the boundary judgment condition.

[0098] Position 43: Character 1 does not meet the boundary judgment condition. Because this character is content, all characters from the last character that meets the boundary judgment condition to the content start position (27) "; danger_degree:" are used as post-content, which is called content-end-pre in this article.

[0099] Add the preceding and following contents of the content to the corresponding elements, expressed as:

[0100] element{

[0101] content 2010-08-13 15:35:38

[0102] content-start 9

[0103] content-end 27

[0104] label log_time

[0105] content-start-pre <5> time:

[0106] content-end-pre;danger_degree:}

[0107] S103, in the sample text, extract the alias, the preceding content and the following content of each element in the element set to form a rule set.

[0108] The main goal of this step is to extract the information of each element from the sample text and generate a set of rules. These rules are used to describe the location and related information of each element in the sample text.

[0109] Furthermore, considering that the preceding content and the following content may be too redundant, which in turn leads to too redundant rule sets, we can simplify the extraction process to a certain extent to improve the efficiency of subsequent field extraction. In a specific embodiment, Figure 2 As shown, the specific steps of forming a rule set include:

[0110] S1031, in the example text, simplify the leading content in each element into the leading simplified content, and simplify the trailing content in each element into the trailing simplified content.

[0111] The leading simplified content and trailing simplified content include unique simplified characters and serial numbers. This step is used to simplify the leading content and trailing content of the element so as to more effectively generate parsing expressions and rule sets.

[0112] Optionally, when simplifying the preceding content, in the sample text, characters are added to the character group one by one from the end position of the preceding content of each element to the start position of the text, and the quantity of each character group is judged, and the first character group that meets the quantity judgment condition is used as the first simplified character group, and the part of the preceding content other than the first simplified character group is replaced with different preset serial numbers, and the combination of the first simplified character group and the different preset serial numbers is used as the preceding simplified content. Among them, the quantity judgment condition is that the number of character groups in the sample text is 1.

[0113] For example, the sample text is:

[0114] <5> time:2010-08-13 15:35:38;

[0115] danger_degree:1; breaking_sighn:0; event:

[40432] HTTP; src_addr:192.168.10.238; src_port:2724; dst_addr:202.111.185.5; dst_port:80; proto:TCP.HTT

[0116] Among them, the preceding content of 2010-08-13 15:35:38 is " <5> time:", when the character ":" is added to the character group starting from the end position of the element's preceding content, it is found that the quantity judgment condition is not met; then when the character "e:" is added to the character group, it is found that the quantity judgment condition is not met; then when the character "me:" is added to the character group, it is found that the quantity judgment condition is met, so me: is used as the first simplified character group, and "ti" is represented by a preset serial number in the form of %{?xi}, i∈[1,2,3…], here represented as %{?x1}, then the preceding simplified content (content-start-pre-tmp) is finally represented as "%{?x1}me:".

[0117] Optionally, when simplifying the post-content, characters are added to the character group one by one from the starting position of the post-content of each element to the ending position of the text, the quantity of each character group is judged, the first character group that meets the quantity judgment condition is used as the second simplified character group, the part of the post-content except the second simplified character group is replaced with different preset serial numbers, and the combination of the second simplified character group and the different preset serial numbers is used as the post-simplified content.

[0118] For example, the sample text is:

[0119] <5> time:2010-08-13 15:35:38;

[0120] danger_degree:1; breaking_sighn:0; event:

[40432] HTTP; src_addr:192.168.10.238; src_port:2724; dst_addr:202.111.185.5; dst_port:80; proto:TCP.HTT

[0121] Among them, the post-content of 2010-08-13 15:35:38 is ";danger_degree:". When the character ";" is added to the character group from the starting position of the post-content of the element, it is found that the quantity judgment condition is not satisfied; then the character ";d" is added to the character group, it is found that the quantity judgment condition is not satisfied; then the character ";da" is added to the character group, it is found that the quantity judgment condition is satisfied, so ;da is used as the second simplified character group, and "danger_degree:" is represented by a preset sequence number in the form of %{? xi}, i∈[1,2,3…], which is represented as %{? x2} here, then the post-simplified content (content-end-pre-tmp) is finally represented as ";da%{?x2}".

[0122] S1032, extract the alias, the leading simplified content and the trailing simplified content of each element to combine and generate an analytical expression.

[0123] That is, traverse the element collection element-list, and for each element element, append the content-start-pre-tmp and %{lable} and content-end-pre-tmp combination to the parsed expression.

[0124] S1033, extracting the alias in the parsed expression and the simplified characters in the leading simplified content and the trailing simplified content based on a preset regular expression to form a rule set.

[0125] Specifically, we extract the parsed expression according to the preset regular expression (.*?)%\\{([^}]*?)} to generate a rule set fields containing multiple fields, each field containing (lable: alias, previous: simplified characters in the preceding simplified content, next: simplified characters in the following simplified content).

[0126] Through the processing of S1031-S1033, we abstract the element information in the sample text into a set of simplified rule sets. These rule sets can be used for text field extraction more efficiently.

[0127] S104, obtaining the text to be processed, intercepting the content in the text to be processed that is the same as the pre-content and post-content of the rule set as the target pre-content and target post-content, extracting the field between the target pre-content and the target post-content, and assigning a corresponding alias based on the rule set.

[0128] The text type of the to-be-processed text and the sample text is the same, that is, the format and structure of the two should be similar, for example, they can be log texts of different time periods. However, the sample text cannot be a log text and the to-be-processed text cannot be a transaction record, because there are obvious differences in the format and structure of the two.

[0129] For example, there is the following text to be processed:

[0130] <5> time:2024-05-21 10:00:00;

[0131] danger_degree:2; breaking_sign:1; event:

[12345] ALERT; src_addr:192.168.0.1; src_port:8080; dst_addr:10.0.0.1; dst_port:80; proto:TCP; HTTP

[0132] In the rule set, it is assumed that the record has the same element as the previous content " <5> time:" and the following content "; danger_degree:". After matching, we can determine that there is a set of target preceding content in the text to be processed" <5> time:" and the target post-content "; danger_degree:", we can extract the target pre-content " <5> time:” and the target post-content “; danger_degree:”, and assign the corresponding alias “log_time”.

[0133] If the rule set records the simplified characters "me:" of the simplified content before and ";da" of the simplified content after the same element, then the text to be processed can also be determined to have a set of target prefix content "me:" and target post-content ";da". We can extract the field "2024-05-21 10:00:00" between the target prefix content "me:" and the target post-content ";da" and assign the corresponding alias "log_time".

[0134] like Figure 3 As shown, Figure 3 This is a performance comparison chart between this method and traditional regular expressions. It can be seen that this method consumes less time than the traditional regular expression method for the same number of parsing times, and as the number of parsing times increases (that is, the amount of text to be processed increases), the time difference becomes more obvious.

[0135] The above-mentioned text field extraction method first extracts multiple elements from the sample text, each element including an alias, content, a content start position and a content end position. The extraction of these elements can be used for rule generation in subsequent steps and field extraction of the text to be processed. Then, according to the content start position and content end position of each element, the leading content and trailing content associated with each content are determined, and they are added to the corresponding element. This helps to locate the position of each field more accurately. The alias, leading content and trailing content of each element in the extracted element set can form a rule set. These rules are used to describe the position and related information of each element in the sample text. Finally, the text to be processed is obtained, and according to the leading content and trailing content in the rule set, the content matching the rule in the text to be processed is intercepted as the target field. Then, the middle field is extracted from the target field, and the corresponding alias is assigned according to the rule set. The beneficial effect of this method is that it does not need to write complex regular expressions for each type of field, but uses the existing element information in the sample text to extract the field, which can reduce the workload of developers and reduce the possibility of errors. In addition, since the element information in the sample text is utilized during the processing, the processing efficiency will also be improved.

[0136] In the first embodiment, the rule set is constructed based on only the extracted elements. However, in fact, in addition to the elements that can be directly extracted, the sample text usually includes some parts that cannot be directly extracted. These parts can also be used to construct the rule set to further improve the rule set and improve the accuracy of subsequent field extraction. Based on this, the present invention also provides another embodiment, such as Figure 4 As shown, Figure 4 This is a flowchart of the text field extraction method in the second embodiment. The steps provided by the text field extraction method in the second embodiment include:

[0137] S401, obtaining a sample text, and extracting elements from the sample text to obtain an element set containing multiple elements.

[0138] S402, in the example text, determine the preceding content associated with each content based on the starting position of the content, and determine the following content associated with each content based on the ending position of the content, and add the preceding content and following content of each content to the corresponding element.

[0139] S403, in the sample text, extract the alias, the preceding content and the following content of each element in the element set to form a rule set.

[0140] Optionally, steps S401-S403 are substantially the same as steps S101-S103 of the text field extraction method in the first embodiment of the present invention, and will not be described in detail herein.

[0141] S404: Characters that are not extracted in the sample text and fall between two spaces are regarded as unextracted content, and the space format of each unextracted content is determined based on the starting position and the ending position of the unextracted content in the sample text.

[0142] Specifically, the main purpose of S404 is to identify the unextracted parts of the sample text, that is, those characters that fall between two spaces. These unextracted contents may contain useful information but have been ignored in the previous element extraction process. Therefore, S404 scans the sample text, finds these unextracted parts, and determines their starting and ending positions in the sample text. Then, based on this position information, S404 determines the space format of each unextracted content, that is, the number and position of the spaces where the unextracted content is located.

[0143] S405, obtaining the alias of each unextracted content, and adding the space format and alias of each unextracted content as a group of new elements into the rule set.

[0144] Specifically, the task of S405 is to assign an alias to each unextracted content, and add its space format and alias as a set of new elements to the rule set. The purpose of this is to ensure that the rule set contains all the information in the sample text, even if this information is not recognized during the element extraction process. By adding these unextracted contents to the rule set, the accuracy and completeness of subsequent field extraction can be improved, ensuring that the system can fully utilize all the information in the sample text.

[0145] S406, obtaining the text to be processed.

[0146] S407, intercepting the content in the text to be processed that is the same as the pre-content and post-content of the rule set as the target pre-content and target post-content, extracting the field between the target pre-content and the target post-content, and assigning a corresponding alias based on the rule set.

[0147] Optionally, steps S406-S407 are substantially consistent with step S104 of the text field extraction method in the first embodiment of the present invention, and will not be described in detail herein.

[0148] S408, intercepting the field in the text to be processed that has the same space format as the rule set, and assigning a corresponding alias based on the rule set.

[0149] Specifically, this step checks the space layout in the text to be processed, finds and extracts the parts that match the space format defined in the rule set, and then assigns each matching field to the corresponding alias in the rule set.

[0150] In one embodiment, Figure 5 As shown, a text field extraction device is proposed, the device comprising:

[0151] The element extraction module 501 is used to obtain a sample text and extract elements from the sample text to obtain an element set including multiple elements; each element includes an alias, content, a content start position, and a content end position; in the sample text, the leading content associated with each content is determined based on the content start position, and the trailing content associated with each content is determined based on the content end position, and the leading content and trailing content of each content are added to the corresponding element;

[0152] A rule forming module 502 is used to extract the alias, the preceding content and the following content of each element in the element set in the sample text to form a rule set;

[0153] The field extraction module 503 is used to obtain the text to be processed, intercept the content in the text to be processed that is the same as the pre-content and post-content of the rule set as the target pre-content and target post-content, extract the field between the target pre-content and the target post-content, and assign a corresponding alias based on the rule set; wherein the text type of the text to be processed is consistent with that of the example text.

[0154] Figure 6 FIG. 2 shows an internal structure diagram of a text field extraction device in one embodiment. Figure 6 As shown, the text field extraction device includes a processor, a memory and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the text field extraction device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement the text field extraction method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can implement the text field extraction method. Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a partial structure related to the present application scheme, and does not constitute a limitation on the text field extraction device to which the present application scheme is applied. The specific text field extraction device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0155] A computer-readable storage medium stores a computer program, which implements the following steps when executed by a processor: obtaining a sample text, and extracting elements from the sample text to obtain an element set containing multiple elements; wherein each element includes an alias, content, a content start position, and a content end position; determining the leading content associated with each content in the sample text based on the content start position, and determining the trailing content associated with each content based on the content end position, and adding the leading content and trailing content of each content to the corresponding element; in the sample text, extracting the alias, leading content, and trailing content of each element in the element set to form a rule set; obtaining a text to be processed, intercepting the content in the text to be processed that is identical to the leading content and trailing content of the rule set as the target leading content and the target trailing content, extracting the field between the target leading content and the target trailing content, and assigning the corresponding alias based on the rule set; wherein the text type of the text to be processed is consistent with that of the sample text.

[0156] A text field extraction device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: obtaining a sample text, and extracting elements from the sample text to obtain an element set containing multiple elements; wherein each element comprises an alias, content, a content start position and a content end position; determining the leading content associated with each content in the sample text based on the content start position, and determining the trailing content associated with each content based on the content end position, and adding the leading content and trailing content of each content to the corresponding element; in the sample text, extracting the alias, leading content and trailing content of each element in the element set to form a rule set; obtaining a text to be processed, intercepting the content in the text to be processed that is identical to the leading content and trailing content of the rule set as the target leading content and the target trailing content, extracting the field between the target leading content and the target trailing content, and assigning the corresponding alias based on the rule set; wherein the text type of the text to be processed is consistent with that of the sample text.

[0157] It should be noted that the above-mentioned text field extraction method, apparatus, device and computer-readable storage medium belong to a general inventive concept, and the contents in the embodiments of the text field extraction method, apparatus, device and computer-readable storage medium are applicable to each other.

[0158] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0159] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0160] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A text field extraction method, characterized in that: The method comprises: Acquire a sample text, and extract elements from the sample text to obtain an element set including a plurality of elements; wherein each element includes an alias, content, a content start position, and a content end position; In the example text, determining the preceding content associated with each content based on the starting position of the content, and determining the following content associated with each content based on the ending position of the content, and adding the preceding content and following content of each content to the corresponding element; In the sample text, extract the alias, the preceding content and the following content of each element in the element set to form a rule set; Acquire a text to be processed, intercept the content in the text to be processed that is identical to the pre-content and post-content of the rule set as the target pre-content and target post-content, extract the field between the target pre-content and the target post-content, and assign a corresponding alias based on the rule set; wherein the text type of the text to be processed is consistent with that of the sample text; In the example text, extracting the alias, the preceding content and the following content of each element in the element set to form a rule set includes: In the example text, the leading content in each element is simplified into the leading simplified content, and the trailing content in each element is simplified into the trailing simplified content; wherein the leading simplified content and the trailing simplified content include unique simplified characters and serial numbers; Extract the alias, the preceding simplified content, and the following simplified content of each element to combine and generate an analytical expression; The alias in the parsing expression and the simplified characters in the preceding simplified content and the following simplified content are extracted based on a preset regular expression to form a rule set.

2. The method according to claim 1, characterized in that The determining, in the example text, the preceding content associated with each content based on the starting position of the content comprises: In the example text, boundary judgment is performed character by character from the starting position of each content to the starting position of the text, and the characters from the last character that meets the boundary judgment condition to before the starting position of the content are used as the preceding content; wherein the boundary judgment condition includes that the character is not a space and is not content.

3. The method according to claim 2, characterized in that The determining of the post-content associated with each content based on the content termination position includes: The boundary judgment is performed character by character from the content end position of each content to the text end position of the text, and the characters from the last character that meets the boundary judgment condition to the content end position are taken as the post-content.

4. The method according to claim 1, characterized in that: The method of simplifying the preceding content in each element into the preceding simplified content includes: In the example text, characters are added to the character group one by one from the end position of the preceding content of each element to the starting position of the text, and the quantity of each character group is judged. The first character group that meets the quantity judgment condition is used as the first simplified character group, and the part of the preceding content except the first simplified character group is replaced with different preset serial numbers, and the combination of the first simplified character group and the different preset serial numbers is used as the preceding simplified content; wherein the quantity judgment condition is that the number of character groups in the example text is 1.

5. The method according to claim 4, characterized in that The step of simplifying the post-positioned content in each element into post-positioned simplified content includes: Add characters one by one to the character group from the starting position of the post-content of each element to the ending position of the text, perform quantity judgment on each character group, take the first character group that meets the quantity judgment condition as the second simplified character group, replace the part of the post-content except the second simplified character group with different preset serial numbers, and take the combination of the second simplified character group and the different preset serial numbers as the post-simplified content.

6. The method according to claim 1, characterized in that The method further comprises: The characters in the sample text that are not extracted and fall between two spaces are regarded as unextracted content, and the space format of each unextracted content is determined based on the starting position and the ending position of the unextracted content in the sample text; Obtain the alias of each unextracted content, and add the space format and alias of each unextracted content as a set of new elements to the rule set; Fields in the text to be processed that have the same space format as the rule set are intercepted, and corresponding aliases are assigned based on the rule set.

7. A text field extraction device, characterized in that: The device comprises: An element extraction module is used to obtain a sample text and extract elements from the sample text to obtain an element set including multiple elements; wherein each element includes an alias, content, a content start position, and a content end position; in the sample text, a pre-content associated with each content is determined based on the content start position, and a post-content associated with each content is determined based on the content end position, and the pre-content and post-content of each content are added to the corresponding element; A rule forming module, used for extracting the alias, the preceding content and the following content of each element in the element set in the sample text to form a rule set; A field extraction module is used to obtain a text to be processed, intercept the content in the text to be processed that is the same as the pre-content and post-content of the rule set as the target pre-content and target post-content, extract the field between the target pre-content and the target post-content, and assign a corresponding alias based on the rule set; wherein the text type of the text to be processed is consistent with that of the sample text; In the example text, extracting the alias, the preceding content and the following content of each element in the element set to form a rule set includes: In the example text, the leading content in each element is simplified into the leading simplified content, and the trailing content in each element is simplified into the trailing simplified content; wherein the leading simplified content and the trailing simplified content include unique simplified characters and serial numbers; Extract the alias, the preceding simplified content, and the following simplified content of each element to combine and generate an analytical expression; The alias in the parsing expression and the simplified characters in the preceding simplified content and the following simplified content are extracted based on a preset regular expression to form a rule set.

8. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to execute the steps of the method according to any one of claims 1 to 6.

9. A text field extraction device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 6.

Images