Developing intelligent document parsing methods

Abstract

The present invention discloses a method for intelligent parsing of development documents, which is used to parse a structured data set from a development document. First, a first category set and a second category set are defined, a first semantic recognition network and a second semantic recognition network are constructed, and several keyword extraction modules are constructed; then the development document is divided into multiple parts, and then each part is traversed in turn, and for each part, different processing is performed according to whether it is a text paragraph or a table. During the processing, the first semantic recognition network, the second semantic recognition network and each keyword extraction module are used to obtain a structured data set. The present invention can automatically read the development document and accurately extract the required structured data, thereby greatly saving the time required for developers to read and organize documents, improving development efficiency, and also has the advantages of high parsing accuracy and good integrity.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning natural language processing, and specifically relates to a method for parsing and generating structured data from development documents. Background Art

[0002] In the process of software development, there is a common need to connect with third-party systems. In order to achieve this need, developers need to spend a lot of time reading and understanding the development documents provided by third parties to complete tasks such as connection and debugging.

[0003] The traditional way of processing is that developers need to read relevant documents one by one, interpret the document content manually and complete the code development. This method is not only labor-intensive, but also requires re-reading the document every time a change is made, wasting a lot of time and energy.

[0004] To solve the above problems, technicians in this field often organize the corresponding structured data according to the content while reading the document. Such structured data generally includes information such as URL address, request method, request body, response body, etc. Subsequent developers can not only refer to the organized structured data for code development, but also use structured data and some tools to quickly generate simple code. Although this method improves development efficiency to a certain extent and reduces the repetitive work of developers, there is still the problem of large workload of structured data organization. Summary of the invention

[0005] The present invention proposes a method for intelligent parsing of development documents, the purpose of which is to solve the problem of heavy workload and low efficiency in collating structured data based on development documents.

[0006] The technical solution of the present invention is as follows:

[0007] A development document intelligent parsing method, the parsing method is used to parse a structured data set from a development document; the structured data set includes a plurality of data items, each of which includes a first attribute for describing the data item, and also includes an entity attribute and a state quantity; the entity attribute is a set including a plurality of parameter items, each of which includes a plurality of second attributes for describing the parameter item; the state quantity is used to point to the entity attribute currently being edited;

[0008] A first category set and a second category set are defined; the first category in the first category set includes a category for describing a first attribute in a data item, and also includes a "wrapper entity" and a "table" category corresponding to an entity attribute, and a "noise" category representing content that does not require attention; the second category in the second category set is a category for describing a second attribute under a parameter item;

[0009] Constructing a first semantic recognition network and a second semantic recognition network, wherein the first semantic recognition network is used to determine the first category to which the input content belongs, and the second semantic recognition network is used to determine the second category to which the input content belongs;

[0010] Constructing a plurality of keyword extraction modules, each of which corresponds to a different first category, and the keyword extraction module is used to extract keywords from the input content;

[0011] The parsing steps are:

[0012] Step 1: Initialize a structured data set, a paragraph stored, and a first category global id. The paragraph stored is used to store the paragraphs that have been read and are to be processed. The first category global id is used to record the first category to which the content in the current paragraph stored belongs.

[0013] Step 2: Divide the development document into multiple parts, and then traverse each part in turn. For each part, process it according to step 3;

[0014] Step 3: Determine whether the currently read part is a text paragraph or a table. If it is a text paragraph, execute step 4; otherwise, execute step 5.

[0015] Step 4: Take the currently read text paragraph as paragraph current, use the first semantic recognition network to obtain the first category corresponding to paragraph current, and after confirming that paragraph current does not belong to "noise", determine whether the first category of paragraph current is consistent with the first category global id. If they are consistent, add paragraph current to paragraph stored and end the processing of the currently read part. If they are inconsistent, settle the content in paragraph stored, then clear paragraph stored, add paragraph current to paragraph stored, and finally use the first category of paragraph current as the first category global id;

[0016] Step 5, determine whether the paragraph stored contains content, if it does, execute steps 5-1 and 5-2 in sequence, otherwise, directly execute step 5-2;

[0017] Step 5-1: settle the content in the paragraph stored, and then clear the paragraph stored;

[0018] Step 5-2, traverse and process each row of the table currently being read; for the current row, first merge the contents of the row into a string, then use the first semantic recognition network to recognize the string, if the identified first category is "packaged entity" or "table", then divide the row into multiple cells by column, and create a parameter item; for each cell, use the second semantic recognition network to identify the contents of the cell to obtain the second category to which it belongs, and then assign the contents of the cell to the second attribute corresponding to the identified second category in the created parameter item; after the cell traversal of each row is completed, add the parameter item corresponding to the row to the entity attribute pointed to by its state quantity in the currently pointed data item.

[0019] As a further improvement of the intelligent parsing method for development documents, the settlement processing means: if the first category global id indicates that a new interface description is to be started, a new data item is created, and then the keyword extraction module corresponding to the first category global id is used to extract keywords from the paragraph stored, and the extracted keywords are assigned to the first attribute corresponding to the first category global id in the data item, and then the first category global id is assigned to the state quantity in the data item, and then the data item is added to the structured data set and points to the newly added data item; otherwise, the keyword extraction module is used to extract keywords from the paragraph stored, and the extracted keywords are assigned to the first attribute corresponding to the first category global id in the currently pointed data item.

[0020] As a further improvement of the intelligent parsing method for development documents, the working process of the keyword extraction module is: extracting several states from the input content to obtain a state sequence, then querying the state value set obtained in the training phase to obtain the value of each state in the state sequence, and then obtaining the position of the keyword in the input content according to the judgment rules to complete the keyword extraction.

[0021] As a further improvement of the development document intelligent parsing method, the training process of the keyword extraction module is as follows:

[0022] Step A-1, constructing a training set, wherein each training sample in the training set contains a keyword to be extracted, and the position of the keyword in the training sample is marked;

[0023] Step A-2, extracting states from each training sample to obtain a state set, and determining the state in the state set that is directly adjacent to the keyword in the training sample as the terminal state;

[0024] Step A-3, traverse each state in the state set, and create an associated state set for each state;

[0025] Step A-4: Set up reward and penalty mechanisms, build a value function for the state, and use the policy iteration algorithm to obtain the value of each state in the state set.

[0026] As a further improvement of the development document intelligent parsing method, the process of obtaining the state set in step A-2 is:

[0027] Step A-2-1, starting from the first word in the input content, traverse each word to the right, and process each word through the following steps:

[0028] Step A-2-1-1, determine whether the currently traversed word belongs to the keyword, if it does, end step A-2-1, otherwise extract the current word as a state, and use the current word as the judgment range, and continue to execute step A-2-1-2;

[0029] Step A-2-1-2, in the input content, expand one word to the right based on the current judgment range to obtain a new judgment range. If the expanded word belongs to a keyword, the processing of the currently traversed word is terminated, otherwise, step A-2-1-3 is continued;

[0030] Step A-2-1-3, extract the content in the current judgment range as a state, and then jump to step A-2-1-2;

[0031] Step A-2-2, starting from the last word in the input content, traverse each word to the left, and process each word through the following steps:

[0032] Step A-2-2-1, determine whether the currently traversed word belongs to the keyword, if it does, end step A-2-2, otherwise extract the current word as a state, and use the current word as the judgment range, and continue to execute step A-2-2-2;

[0033] Step A-2-2-2, in the input content, expand one word to the left based on the current judgment range to obtain a new judgment range. If the expanded word belongs to a keyword, the processing of the currently traversed word is terminated, otherwise, step A-2-2-3 is continued;

[0034] Step A-2-2-3: extract the content in the current judgment range as a state, and then jump to step A-2-2-2.

[0035] As a further improvement of the intelligent parsing method for development documents, the specific method of step A-3 is: for the currently traversed state, if other states in the state set can be obtained by extending this state by one word to the left or one word to the right, then these other states are added to the associated state set of this state.

[0036] As a further improvement of the development document intelligent parsing method, in step A-4, a positive reward is set for the terminal state in the state set, and a negative reward is set for the state in the state set whose associated state set is empty. The value function of the constructed state is:

[0037] ;

[0038] in, Yes Status The value function of Yes Status The associated state set of It is the slave state obtained based on the associated state set Transition to state The probability of From the state Transition to state The rewards you get, is the discount factor.

[0039] As a further improvement of the development document intelligent parsing method, the process of the keyword extraction module obtaining the state sequence from the input content is as follows:

[0040] Step B-1, traverse each word in the input content from left to right, and process each word according to step B-2;

[0041] Step B-2: Explore rightward from the currently traversed word, find the longest string that can match the state in the state set as the state corresponding to the word, and add the state to the state sequence.

[0042] As a further improvement of the development document intelligent parsing method, step B-2 specifically includes:

[0043] Step B-2-1, taking the currently traversed word as the judgment range and creating a state object;

[0044] Step B-2-2, determine whether the string in the current judgment range matches any state in the state set. If it matches, overwrite the value in the state object with the state, and then execute step B-2-3. Otherwise, determine whether there is a valid state in the state object. If so, add the state to the state sequence, and end the exploration of the currently traversed word;

[0045] Step B-2-3: Expand one word to the right from the current judgment range in the input content, and then jump to step B-2-2.

[0046] As a further improvement of the development document intelligent parsing method: in the keyword extraction module, when the position of the keyword in the input content is obtained according to the judgment rule, the state sequence is regarded as peaks of different heights according to the value of each state; first find the state with the highest value in the state sequence, and define the state as the highest peak; then use the highest peak as the boundary, define the state with the highest value in the states on the left side of the highest peak as the left peak, and define the state with the highest value in the states on the right side of the highest peak as the right peak; and define all terminal states in the state sequence as terminal peaks, and then extract keywords according to the following rules:

[0047] Rule 1, Definition Condition 1: The highest peak is the terminal peak, one of the left peak and the right peak is also the terminal peak, and the character range of the highest peak in the input content does not intersect with the character range of the left peak and the right peak;

[0048] If condition 1 is met, the characters between the two terminal peaks are extracted as keywords;

[0049] Rule 2, Definition Condition 2: The highest peak is the terminal peak, and neither the left peak nor the right peak is the terminal peak;

[0050] If condition 2 is met, the characters between the highest value peak and the highest peak in the left peak and the right peak are extracted as keywords;

[0051] Rule 3: If neither condition 1 nor condition 2 is satisfied, and there is no peak on the right side of the highest peak, and there are characters on the right side of the highest peak in the input content, all characters on the right side of the highest peak are extracted as keywords;

[0052] Rule 4: If neither condition 1 nor condition 2 is met, and there is no peak on the right side of the highest peak, and there is no character on the right side of the highest peak, all characters on the left side of the highest peak are extracted as keywords;

[0053] Rule 5: If neither condition 1 nor condition 2 is met, and there is no peak on the left side of the highest peak, and there are characters on the left side of the highest peak, all characters on the left side of the highest peak are extracted as keywords;

[0054] Rule 6: If neither condition 1 nor condition 2 is met, and there is no peak on the left side of the highest peak, and there is no character on the left side of the highest peak, then all characters on the right side of the highest peak are extracted as keywords;

[0055] Rule 7: If neither condition 1 nor condition 2 is met, and there are peaks on both sides of the highest peak, the right peak is higher than the left peak, and the character range of the highest peak does not intersect with the character range of the right peak, then the characters between the highest peak and the right peak are extracted as keywords;

[0056] Rule 8: If neither condition 1 nor condition 2 is met, and there are peaks on both sides of the highest peak, the right peak is higher than the left peak, and the character range of the highest peak intersects with the character range of the right peak, then the characters covered by the right peak are extracted as keywords;

[0057] Rule 9: If neither condition 1 nor condition 2 is met, and there are peaks on both sides of the highest peak, the left peak is higher than the right peak, and the character range of the highest peak does not intersect with the character range of the left peak, then the characters between the highest peak and the left peak are extracted as keywords;

[0058] Rule 10: If neither condition 1 nor condition 2 is met, and there are peaks on both sides of the highest peak, the left peak is higher than the right peak, and the character range of the highest peak intersects with the character range of the left peak, the characters covered by the left peak are extracted as keywords.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. The present invention can automatically read development documents and accurately extract required structured data, thereby greatly saving the time required for developers to read and organize documents and improving development efficiency.

[0061] 2. The present invention uses two semantic recognition networks to respectively identify the content category in a paragraph and the content of a table cell. It is not only applicable to documents of different formats, but also can fully identify the important information in the document to ensure the integrity of the parsing results.

[0062] 3. The keyword extraction module of the present invention is based on a reinforcement learning dynamic programming algorithm. It first defines and calculates the value functions of different states, and then automatically extracts keywords according to the value size and judgment rules of each state contained in the paragraph. It has high accuracy and good adaptability. DETAILED DESCRIPTION

[0063] The technical solution of the present invention is described in detail below. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0064] A method for intelligently parsing development documents is used to parse a structured data set from a development document. The development document generally contains text paragraphs and tables, which contain information required for development such as the URL address of the interface, the request method, the request body structure, the response body structure, and a large amount of text descriptions. The purpose of this method is to parse the above required information into a structured data set for developers to directly consult and use.

[0065] The structured data set includes a plurality of data items, each of which includes a first attribute for describing the data item, and also includes entity attributes and state quantities. The entity attributes are a set of a plurality of parameter items, each of which includes a plurality of second attributes for describing the parameter items. The state quantity is used to point to the entity attribute currently being edited.

[0066] Generally, a data item corresponds to an interface in a document. The first attribute under the data item is used to store general information describing the interface, such as "URL address" and "request method", while the entity attribute is used to store complex information of the interface, such as "request body" and "response body". The parameter items under the entity attribute correspond to different parameters, such as "name" and "gender" in the request body. The second attribute in the parameter item records "parameter name", "data type", "required", "description", etc., which are used to describe the corresponding parameters.

[0067] Before parsing begins, a first category set and a second category set are defined. The first category in the first category set includes a category for describing the first attribute in the data item, and also includes "wrapper entity" and "table" categories corresponding to entity attributes and a "noise" category representing content that does not need attention. The second category in the second category set is a category for describing the second attribute under the parameter item.

[0068] In this embodiment, the first category set includes the following first categories:

[0069] 1. Interface title;

[0070] 2. Interface address (URL);

[0071] 3. Submission method (post / get);

[0072] 4. Request body;

[0073] 5. Packaging entity;

[0074] 6. Response body;

[0075] 7. Request header;

[0076] 8. Dictionary title;

[0077] 9. Interface description;

[0078] 10. Request data type;

[0079] 11. Return data type;

[0080] 12. Table (usually contains "parameter name", "data type", "required", etc., each row needs to be integrated into a parameter item, and the entire table corresponds to an entity attribute);

[0081] 13. Noise (content that does not require attention).

[0082] In this embodiment, the second category set includes the following second categories:

[0083] 1. Parameter name;

[0084] 2. Data type;

[0085] 3. Is it necessary?

[0086] 4. Description.

[0087] A first semantic recognition network and a second semantic recognition network are constructed, and both the first semantic recognition network and the second semantic recognition network are constructed based on Transformer. The first semantic recognition network is used to determine the first category to which the input content belongs, and the second semantic recognition network is used to determine the second category to which the input content belongs.

[0088] A plurality of keyword extraction modules are constructed, each of which corresponds to a different first category, and the keyword extraction module is used to extract keywords from the input content.

[0089] For example, for a text: "This URL address is http: / / xxxx / xxx", you need to use the keyword extraction module corresponding to the first category "interface address" to extract the keyword "http: / / xxxx / xxx". For a text: "Submission method: POST request", you need to use the keyword extraction module corresponding to the first category "Submission method" to extract the keyword "POST".

[0090] Furthermore, the working process of the keyword extraction module is: extracting several states from the input content to obtain a state sequence, then querying the state value set obtained in the training phase to obtain the value of each state in the state sequence, and then obtaining the position of the keyword in the input content according to the judgment rules to complete the keyword extraction.

[0091] The training process of the keyword extraction module is as follows:

[0092] Step A-1: ​​construct a training set, where each training sample in the training set contains a keyword to be extracted, and the position of the keyword in the training sample is marked.

[0093] Step A-2: extract states from each training sample to obtain a state set, and determine the state in the state set that is directly adjacent to the keyword in the training sample as the terminal state.

[0094] The process of obtaining the state set is:

[0095] Step A-2-1: Traverse each character from the first character in the input content to the right. For each character, process it through the following steps:

[0096] Step A-2-1-1: Determine whether the currently traversed character belongs to the keyword. If it does, end Step A-2-1. Otherwise, extract the current character as a state, and use the current character as the judgment range, then continue to execute Step A-2-1-2.

[0097] Step A-2-1-2: Expand one character to the right based on the current judgment range in the input content to obtain a new judgment range. If the expanded character belongs to the keyword, end the processing of the currently traversed character. Otherwise, continue to execute Step A-2-1-3.

[0098] Step A-2-1-3: Extract the content in the current judgment range as a state, and then jump to Step A-2-1-2.

[0099] Taking "submission method: POST request" as an example, here "POST" is the pre-labeled keyword. Starting from the first character "提", 5 states can be extracted: "提", "提交", "提交方", "提交方法", "提交方法:". When the keyword is encountered when expanding to "P" next time, stop extraction. Then start extracting multiple states from the second character "交", and so on.

[0100] Step A-2-2: Traverse each character from the last character in the input content to the left. For each character, process it through the following steps:

[0101] Step A-2-2-1: Determine whether the currently traversed character belongs to the keyword. If it does, end Step A-2-2. Otherwise, extract the current character as a state, and use the current character as the judgment range, then continue to execute Step A-2-2-2.

[0102] Step A-2-2-2: Expand one character to the left based on the current judgment range in the input content to obtain a new judgment range. If the expanded character belongs to the keyword, end the processing of the currently traversed character. Otherwise, continue to execute Step A-2-2-3.

[0103] Step A-2-2-3: Extract the content in the current judgment range as a state, and then jump to Step A-2-2-2.

[0104] Step A-3: Traverse each state in the state set and create an associated state set for each state.

[0105] The specific method is: for the currently traversed state, if other states in the state set can be obtained by extending this state by one word to the left or one word to the right, then these other states are added to the associated state set of this state.

[0106] Step A-4: Set up reward and penalty mechanisms, build a value function for the state, and use the policy iteration algorithm to obtain the value of each state in the state set.

[0107] Specifically, a positive reward is set for the terminal state in the state set, and a negative reward is set for the state whose associated state set is empty in the state set. The value function of the constructed state is:

[0108] ;

[0109] in, Yes Status The value function of Yes Status The associated state set of It is the slave state obtained based on the associated state set Transition to state The probability of (according to the state in the associated state set ), From the state Transition to state The rewards you get, is the discount factor.

[0110] The value of each state can be obtained through the policy iteration algorithm commonly used in the field of reinforcement learning dynamic programming. It should be noted that the policy iteration algorithm is used in dynamic programming to obtain an optimal strategy (a set of optimal actions), but at the same time, the value of each state in the optimal strategy will also be obtained. This method is not for the purpose of implementing dynamic programming, but to use this method to obtain the value of each state for keyword extraction.

[0111] The process of getting the state sequence from the input content is:

[0112] Step B-1, traverse each word in the input content from left to right, and process each word according to step B-2;

[0113] Step B-2: Explore rightward from the currently traversed word, find the longest string that can match the state in the state set as the state corresponding to the word, and add the state to the state sequence.

[0114] Specifically include:

[0115] Step B-2-1: Use the currently traversed word as the judgment range and create a state object.

[0116] Step B-2-2, determine whether the string in the current judgment range matches any state in the state set. If it matches, use the state to overwrite the value in the state object, and then execute step B-2-3. Otherwise, determine whether there is a valid state in the state object. If so, add the state to the state sequence, and end the exploration of the currently traversed word.

[0117] Step B-2-3: Expand one word to the right from the current judgment range in the input content, and then jump to step B-2-2.

[0118] When the position of the keyword in the input content is obtained according to the judgment rules, the state sequence should be regarded as peaks of different heights according to the value of each state. First, find the state with the highest value in the state sequence and define it as the highest peak. Then, using the highest peak as the boundary, define the state with the highest value on the left side of the highest peak as the left peak, and define the state with the highest value on the right side of the highest peak as the right peak. And define all terminal states in the state sequence as terminal peaks, and then extract keywords according to the following rules:

[0119] Rule 1, definition condition 1: the highest peak is the terminal peak, one of the left peak and the right peak is also the terminal peak, and the character range of the highest peak in the input content does not intersect with the character range of the left peak and the right peak.

[0120] If condition 1 is met, the characters between the two terminal peaks are extracted as keywords.

[0121] Rule 2, Definition Condition 2: The highest peak is the terminal peak, and neither the left peak nor the right peak is the terminal peak.

[0122] If condition 2 is satisfied, the characters between the highest value peak and the highest peak of the left peak and the right peak are extracted as keywords.

[0123] Rule 3: If neither condition 1 nor condition 2 is met, and there is no peak on the right side of the highest peak, and there are characters on the right side of the highest peak in the input content, all characters on the right side of the highest peak are extracted as keywords.

[0124] Rule 4: If neither condition 1 nor condition 2 is met, and there is no peak on the right side of the highest peak, and there is no character on the right side of the highest peak, all characters on the left side of the highest peak are extracted as keywords.

[0125] Rule 5: If neither condition 1 nor condition 2 is met, and there is no peak on the left side of the highest peak, and there are characters on the left side of the highest peak, all characters on the left side of the highest peak are extracted as keywords.

[0126] Rule 6: If neither condition 1 nor condition 2 is met, and there is no peak on the left side of the highest peak, and there is no character on the left side of the highest peak, all characters on the right side of the highest peak are extracted as keywords.

[0127] Rule 7: If neither condition 1 nor condition 2 is met, and there are peaks on both sides of the highest peak, the right peak is higher than the left peak, and the character range of the highest peak does not intersect with the character range of the right peak, then the characters between the highest peak and the right peak are extracted as keywords.

[0128] Rule 8: If neither condition 1 nor condition 2 is met, and there are peaks on both sides of the highest peak, the right peak is higher than the left peak, and the character range of the highest peak intersects with the character range of the right peak, then the characters covered by the right peak are extracted as keywords.

[0129] Rule 9: If neither condition 1 nor condition 2 is met, and there are peaks on both sides of the highest peak, the left peak is higher than the right peak, and the character range of the highest peak does not intersect with the character range of the left peak, then the characters between the highest peak and the left peak are extracted as keywords.

[0130] Rule 10: If neither condition 1 nor condition 2 is met, and there are peaks on both sides of the highest peak, the left peak is higher than the right peak, and the character range of the highest peak intersects with the character range of the left peak, the characters covered by the left peak are extracted as keywords.

[0131] In this embodiment, three keyword extraction modules are constructed, corresponding to the aforementioned first category 1, first category 2 and first category 3. The algorithms and training steps executed by the three keyword extraction modules are consistent, and the only difference is that different training samples are used.

[0132] The steps of the analytical method are:

[0133] Step 1: Initialize a structured data set, a paragraph stored, and a first category global id. The paragraph stored is used to store the paragraphs that have been read and are to be processed. The first category global id is used to record the first category to which the content in the current paragraph stored belongs.

[0134] Step 2: Divide the development document into multiple parts, and then traverse each part in turn. For each part, process it according to step 3.

[0135] Step 3: Determine whether the currently read portion is a text paragraph or a table. If it is a text paragraph, execute step 4; otherwise, execute step 5.

[0136] In this step, the determination of text paragraphs / tables is implemented based on the interface of Word, which belongs to the prior art and will not be described in detail.

[0137] Step 4: Take the currently read text paragraph as paragraph current, use the first semantic recognition network to obtain the first category corresponding to paragraph current, and after confirming that paragraph current does not belong to "noise", determine whether the first category of paragraph current is consistent with the first category global id. If they are consistent, add paragraph current to paragraph stored and end the processing of the currently read part. If they are inconsistent, settle the content in paragraph stored, then clear paragraph stored, add paragraph current to paragraph stored, and finally use the first category of paragraph current as the first category global id.

[0138] The settlement processing means: if the first category global id indicates that a new interface description is to be started (for example, "interface title" or "interface address"), a new data item is created, and then the keyword extraction module corresponding to the first category global id is used to extract keywords from the paragraph stored, and the extracted keywords are assigned to the first attribute corresponding to the first category global id in the data item, and then the first category global id is assigned to the state quantity in the data item, and then the data item is added to the structured data set and points to the newly added data item; otherwise, the keyword extraction module is used to extract keywords from the paragraph stored, and the extracted keywords are assigned to the first attribute corresponding to the first category global id in the currently pointed data item.

[0139] Step 5: Determine whether the paragraph stored contains content. If it does, execute steps 5-1 and 5-2 in sequence; otherwise, execute step 5-2 directly.

[0140] Step 5-1: settle the contents of the paragraph stored, and then clear the paragraph stored.

[0141] Step 5-2, traverse and process each row of the table currently being read; for the current row, first merge the contents of the row into a string, then use the first semantic recognition network to recognize the string, if the identified first category is "packaged entity" or "table", then divide the row into multiple cells by column, and create a parameter item; for each cell, use the second semantic recognition network to identify the contents of the cell to obtain the second category to which it belongs, and then assign the contents of the cell to the second attribute corresponding to the identified second category in the created parameter item; after the cell traversal of each row is completed, add the parameter item corresponding to the row to the entity attribute pointed to by its state quantity in the currently pointed data item.

[0142] In the development document, each row of the table usually corresponds to a parameter, and the table generally includes four columns describing each parameter: "Parameter Name", "Data Type", "Required", and "Description". Through the above processing, the attribute values ​​of each column of each row can be packaged together into a parameter item, and then added to the entity attribute pointed to by the state quantity, thereby gradually completing the parsing conversion from the table to the entity attribute.

[0143] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. The scope of the present invention is defined by the claims rather than the above description.

Claims

1. A method for intelligent parsing of development documents, characterized in that: The parsing method is used to parse a structured data set from a development document; the structured data set includes a plurality of data items, each of which includes a first attribute for describing the data item, and also includes an entity attribute and a state quantity; the entity attribute is a set including a plurality of parameter items, each of which includes a plurality of second attributes for describing the parameter item; the state quantity is used to point to the entity attribute currently being edited; A first category set and a second category set are defined; the first category in the first category set includes a category for describing a first attribute in a data item, and also includes a "wrapper entity" and a "table" category corresponding to an entity attribute, and a "noise" category representing content that does not require attention; the second category in the second category set is a category for describing a second attribute under a parameter item; Constructing a first semantic recognition network and a second semantic recognition network, wherein the first semantic recognition network is used to determine the first category to which the input content belongs, and the second semantic recognition network is used to determine the second category to which the input content belongs; Constructing a plurality of keyword extraction modules, each of which corresponds to a different first category, and the keyword extraction module is used to extract keywords from the input content; The parsing steps are: Step 1: Initialize a structured data set, a paragraph stored, and a first category global id. The paragraph stored is used to store the paragraphs that have been read and are to be processed. The first category global id is used to record the first category to which the content in the current paragraph stored belongs. Step 2: Divide the development document into multiple parts, and then traverse each part in turn. For each part, process it according to step 3; Step 3: Determine whether the currently read part is a text paragraph or a table. If it is a text paragraph, execute step 4; otherwise, execute step 5. Step 4: Take the currently read text paragraph as paragraph current, use the first semantic recognition network to obtain the first category corresponding to paragraph current, and after confirming that paragraph current does not belong to "noise", determine whether the first category of paragraph current is consistent with the first category global id. If they are consistent, add paragraph current to paragraph stored and end the processing of the currently read part. If they are inconsistent, settle the content in paragraph stored, then clear paragraph stored, add paragraph current to paragraph stored, and finally use the first category of paragraph current as the first category global id; Step 5, determine whether the paragraph stored contains content, if it does, execute steps 5-1 and 5-2 in sequence, otherwise, directly execute step 5-2; Step 5-1: settle the content in the paragraph stored, and then clear the paragraph stored; Step 5-2, traverse and process each row of the table currently being read; for the current row, first merge the contents of the row into a string, then use the first semantic recognition network to recognize the string, if the recognized first category is "packaged entity" or "table", divide the row into multiple cells by column, and create a parameter item; for each cell, use the second semantic recognition network to recognize the contents of the cell to obtain the second category to which it belongs, and then assign the contents of the cell to the second attribute corresponding to the recognized second category in the created parameter item; after the cell traversal of each row is completed, add the parameter item corresponding to the row to the entity attribute pointed to by its state quantity in the currently pointed data item.

2. The method for intelligent parsing of development documents according to claim 1, characterized in that: The settlement processing means: if the first category global id indicates that a new interface description is to be started, a new data item is created, and then the keyword extraction module corresponding to the first category global id is used to extract keywords from the paragraph stored, and the extracted keywords are assigned to the first attribute corresponding to the first category global id in the data item, and then the first category global id is assigned to the state quantity in the data item, and then the data item is added to the structured data set and points to the newly added data item; otherwise, the keyword extraction module is used to extract keywords from the paragraph stored, and the extracted keywords are assigned to the first attribute corresponding to the first category global id in the currently pointed data item.

3. The method for intelligent parsing of development documents according to claim 1 or 2, characterized in that: The working process of the keyword extraction module is as follows: extracting several states from the input content to obtain a state sequence, then querying the state value set obtained in the training phase to obtain the value of each state in the state sequence, and then obtaining the position of the keyword in the input content according to the judgment rules to complete the keyword extraction.

4. The method for intelligent parsing of development documents as claimed in claim 3, characterized in that: The training process of the keyword extraction module is as follows: Step A-1, constructing a training set, wherein each training sample in the training set contains a keyword to be extracted, and the position of the keyword in the training sample is marked; Step A-2, extracting states from each training sample to obtain a state set, and determining the state in the state set that is directly adjacent to the keyword in the training sample as the terminal state; Step A-3, traverse each state in the state set, and create an associated state set for each state; Step A-4: Set up reward and penalty mechanisms, build a value function for the state, and use the policy iteration algorithm to obtain the value of each state in the state set.

5. The method for intelligent parsing of development documents as claimed in claim 4, characterized in that: The process of obtaining the state set in step A-2 is: Step A-2-1, starting from the first word in the input content, traverse each word to the right, and process each word through the following steps: Step A-2-1-1, determine whether the currently traversed word belongs to the keyword, if it does, end step A-2-1, otherwise extract the current word as a state, and use the current word as the judgment range, and continue to execute step A-2-1-2; Step A-2-1-2, in the input content, expand one word to the right based on the current judgment range to obtain a new judgment range. If the expanded word belongs to a keyword, the processing of the currently traversed word is terminated, otherwise, step A-2-1-3 is continued; Step A-2-1-3, extract the content in the current judgment range as a state, and then jump to step A-2-1-2; Step A-2-2, starting from the last word in the input content, traverse each word to the left, and process each word through the following steps: Step A-2-2-1, determine whether the currently traversed word belongs to the keyword, if it does, end step A-2-2, otherwise extract the current word as a state, and use the current word as the judgment range, and continue to execute step A-2-2-2; Step A-2-2-2, in the input content, expand one word to the left based on the current judgment range to obtain a new judgment range. If the expanded word belongs to a keyword, the processing of the currently traversed word is terminated, otherwise, step A-2-2-3 is continued; Step A-2-2-3: extract the content in the current judgment range as a state, and then jump to step A-2-2-2.

6. The method for intelligent parsing of development documents as claimed in claim 4, characterized in that: The specific method of step A-3 is: for the currently traversed state, if other states in the state set can be obtained by extending this state by one word to the left or one word to the right, then these other states are added to the associated state set of this state.

7. The method for intelligent parsing of development documents as claimed in claim 4, characterized in that: In step A-4, a positive reward is set for the terminal state in the state set, and a negative reward is set for the state whose associated state set is empty in the state set. The value function of the constructed state is: ； in, Yes Status The value function of Yes Status The associated state set of It is the slave state obtained based on the associated state set Transition to state The probability of From the state Transition to state The rewards you get, is the discount factor.

8. The method for intelligent parsing of development documents as claimed in claim 4, characterized in that: The process of the keyword extraction module obtaining the state sequence from the input content is as follows: Step B-1, traverse each word in the input content from left to right, and process each word according to step B-2; Step B-2: Explore rightward from the currently traversed word, find the longest string that can match the state in the state set as the state corresponding to the word, and add the state to the state sequence.

9. The method for intelligent parsing of development documents as claimed in claim 4, characterized in that: Step B-2 specifically includes: Step B-2-1, taking the currently traversed word as the judgment range and creating a state object; Step B-2-2, determine whether the string in the current judgment range matches any state in the state set. If it matches, overwrite the value in the state object with the state, and then execute step B-2-3. Otherwise, determine whether there is a valid state in the state object. If so, add the state to the state sequence, and end the exploration of the currently traversed word; Step B-2-3: Expand one word to the right from the current judgment range in the input content, and then jump to step B-2-2.

10. The method for intelligent parsing of development documents according to claim 4, characterized in that: In the keyword extraction module, when the position of the keyword in the input content is obtained according to the judgment rule, the state sequence is regarded as peaks of different heights according to the value of each state; first find the state with the highest value in the state sequence, and define the state as the highest peak; then use the highest peak as the boundary, define the state with the highest value in the states on the left side of the highest peak as the left peak, and define the state with the highest value in the states on the right side of the highest peak as the right peak; and define all terminal states in the state sequence as terminal peaks, and then extract keywords according to the following rules: Rule 1, Definition Condition 1: The highest peak is the terminal peak, one of the left peak and the right peak is also the terminal peak, and the character range of the highest peak in the input content does not intersect with the character range of the left peak and the right peak; If condition 1 is met, the characters between the two terminal peaks are extracted as keywords; Rule 2, Definition Condition 2: The highest peak is the terminal peak, and neither the left peak nor the right peak is the terminal peak; If condition 2 is met, the characters between the highest value peak and the highest peak in the left peak and the right peak are extracted as keywords; Rule 3: If neither condition 1 nor condition 2 is satisfied, and there is no peak on the right side of the highest peak, and there are characters on the right side of the highest peak in the input content, all characters on the right side of the highest peak are extracted as keywords; Rule 4: If neither condition 1 nor condition 2 is met, and there is no peak on the right side of the highest peak, and there is no character on the right side of the highest peak, all characters on the left side of the highest peak are extracted as keywords; Rule 5: If neither condition 1 nor condition 2 is met, and there is no peak on the left side of the highest peak, and there are characters on the left side of the highest peak, all characters on the left side of the highest peak are extracted as keywords; Rule 6: If neither condition 1 nor condition 2 is met, and there is no peak on the left side of the highest peak, and there is no character on the left side of the highest peak, then all characters on the right side of the highest peak are extracted as keywords; Rule 7: If neither condition 1 nor condition 2 is met, and there are peaks on both sides of the highest peak, the right peak is higher than the left peak, and the character range of the highest peak does not intersect with the character range of the right peak, then the characters between the highest peak and the right peak are extracted as keywords; Rule 8: If neither condition 1 nor condition 2 is met, and there are peaks on both sides of the highest peak, the right peak is higher than the left peak, and the character range of the highest peak intersects with the character range of the right peak, then the characters covered by the right peak are extracted as keywords; Rule 9: If neither condition 1 nor condition 2 is met, and there are peaks on both sides of the highest peak, the left peak is higher than the right peak, and the character range of the highest peak does not intersect with the character range of the left peak, then the characters between the highest peak and the left peak are extracted as keywords; Rule 10: If neither condition 1 nor condition 2 is met, and there are peaks on both sides of the highest peak, the left peak is higher than the right peak, and the character range of the highest peak intersects with the character range of the left peak, the characters covered by the left peak are extracted as keywords.

Images