Unstructured transaction information recognition method and system based on natural language processing
Through natural language processing technology, including preprocessing, word segmentation, group segmentation and Bayesian network modeling, the high cost problem of manually identifying transaction quotation information in the interbank trading market is solved, and efficient automatic identification of transaction information and Correction of errors reduces labor costs.
Patent Information
- Application Number
- CN202111051611.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-08
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-09-08
AI Technical Summary
In the prior art, transaction matching in the interbank trading market relies on manual collection of market conditions and inquiry and quotations, resulting in high labor costs and low degree of automation, and lack of intelligent methods to identify market quotation information.
Using natural language processing methods, the transaction language text that obtains transaction quotation information is pre-processed, conditional random word segmentation and part-of-speech recognition, group segmentation and paragraph completion processing, and the Bayesian network is used for global optimal recognition. And identify transaction quotation information through matrix structure processing.
Improve the processing efficiency of transaction quotation information, realize automatic recognition and intelligent error correction of transaction language text, reduce the labor cost of OTC transaction inquiry and quotation, and support multi-machine distributed deployment and real-time messaging of massive chat groups deal with.
Smart Images

Figure CN113761900B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer information processing, and in particular to unstructured transaction information recognition and system based on natural language processing. Background Art
[0002] At present, the transaction matching of fixed-income products and non-standardized securities is negotiated by various market participants. With the prosperity of the inter-bank trading market, a large number of transactions require manual collection of market conditions, price inquiries, and negotiation, which consumes a lot of human communication costs. At present, the market is still in a state of high manualization and low automation. Therefore, how to realize intelligent identification of market quotes and other information is a problem that technicians in this field need to solve. Summary of the invention
[0003] The embodiment of the present invention provides a method and system for identifying unstructured transaction information based on natural language processing, aiming to improve the processing efficiency of transaction quotation information.
[0004] In a first aspect, an embodiment of the present invention provides a method for identifying unstructured transaction information based on natural language processing, comprising:
[0005] Acquire the transaction language text corresponding to the transaction quotation information sent by the user, and pre-process the transaction language text;
[0006] Based on the conditional random field, the preprocessed transaction language text is segmented in the transaction context and the part of speech is identified;
[0007] According to the word segmentation results and part-of-speech recognition results of the transaction context, the transaction language text is segmented into meaning groups and paragraphs are completed;
[0008] Extracting part-of-speech features from the transaction context word segmentation results and part-of-speech recognition results, and vectorizing and modeling the part-of-speech features through a Bayesian network to perform global optimal recognition of the transaction language text;
[0009] The global optimal recognition result is checked in matrix format, and based on the check result, the meaning group segmentation and paragraph completion processing results are subjected to matrix structuring processing, and the matrix structuring processing result is output as the transaction quotation information recognition result.
[0010] In a second aspect, an embodiment of the present invention provides a device for identifying unstructured transaction information based on natural language processing, comprising:
[0011] A preprocessing unit, used to obtain a transaction language text corresponding to the transaction quotation information sent by the user, and preprocess the transaction language text;
[0012] A part-of-speech processing unit is used to perform transaction context word segmentation and part-of-speech recognition on the preprocessed transaction language text based on conditional random fields;
[0013] A paragraph processing unit, used to perform meaning group segmentation and paragraph completion processing on the transaction language text according to the transaction context word segmentation result and part-of-speech recognition result;
[0014] An optimal recognition unit, used for extracting part-of-speech features from the transaction context word segmentation results and part-of-speech recognition results, and vectorizing and modeling the part-of-speech features through a Bayesian network to perform global optimal recognition of the transaction language text;
[0015] The first structured processing unit is used to perform matrix format verification on the global optimal recognition result, and based on the verification result, perform matrix structured processing on the meaning group segmentation and paragraph completion processing results, and output the matrix structured processing results as transaction quotation information recognition results.
[0016] In the third aspect, an embodiment of the present invention provides a distributed computer single-point and clustered deployment device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for identifying unstructured transaction information based on natural language processing as described in any one of claims 1 to 7 is implemented, and single-machine computing and multi-machine parallel computing are supported to implement the method for identifying unstructured transaction information based on natural language processing as described in the first aspect.
[0017] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for identifying unstructured transaction information based on natural language processing as described in the first aspect is implemented.
[0018] The embodiment of the present invention provides a method and system for identifying unstructured transaction information based on natural language processing, the method comprising: obtaining a transaction language text corresponding to the transaction quotation information sent by a user, and preprocessing the transaction language text; performing transaction context word segmentation and part-of-speech recognition on the preprocessed transaction language text based on a conditional random field; performing meaning group segmentation and paragraph completion processing on the transaction language text according to the transaction context word segmentation result and the part-of-speech recognition result; extracting part-of-speech features from the transaction context word segmentation result and the part-of-speech recognition result, and vectorizing and modeling the part-of-speech features through a Bayesian network to perform global optimal recognition on the transaction language text; performing matrix format verification on the global optimal recognition result, and performing matrix structured processing on the meaning group segmentation and paragraph completion processing results based on the verification result, and outputting the matrix structured processing result as a transaction quotation information recognition result. The embodiment of the present invention parses various types of transaction "jargon" into specific quotation and transaction information by performing part-of-speech processing and paragraph processing on the user's transaction quotation information, thereby improving the processing efficiency of transaction quotation information. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0020] Figure 1 A flowchart of a method for identifying unstructured transaction information based on natural language processing provided by an embodiment of the present invention;
[0021] Figure 2 A schematic diagram of a sub-flow of step S101 in a method for identifying unstructured transaction information based on natural language processing provided by an embodiment of the present invention;
[0022] Figure 3 A schematic diagram of a sub-flow of step S102 in a method for identifying unstructured transaction information based on natural language processing provided by an embodiment of the present invention;
[0023] Figure 4 A schematic diagram of a sub-flow of step S103 in a method for identifying unstructured transaction information based on natural language processing provided by an embodiment of the present invention;
[0024] Figure 5 A schematic diagram of a sub-flow of step S104 in a method for identifying unstructured transaction information based on natural language processing provided by an embodiment of the present invention;
[0025] Figure 6A schematic diagram of a sub-flow of step S105 in a method for identifying unstructured transaction information based on natural language processing provided by an embodiment of the present invention;
[0026] Figure 7 A schematic block diagram of a device for identifying unstructured transaction information based on natural language processing provided by an embodiment of the present invention;
[0027] Figure 8 A schematic sub-block diagram of a preprocessing unit in a device for identifying unstructured transaction information based on natural language processing provided by an embodiment of the present invention;
[0028] Fig. 9 A schematic block diagram of a part-of-speech processing unit in a non-structured transaction information identification device based on natural language processing provided by an embodiment of the present invention;
[0029] Fig.10 A sub-schematic block diagram of a paragraph processing unit in a non-structured transaction information identification device based on natural language processing provided by an embodiment of the present invention;
[0030] Fig.11 A sub-schematic block diagram of a dispute correction unit in a non-structured transaction information identification device based on natural language processing provided by an embodiment of the present invention;
[0031] Fig.12 A sub-schematic block diagram of a first structured processing unit in an unstructured transaction information identification device based on natural language processing provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0034] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0035] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0036] See below Figure 1 , Figure 1 A flowchart of a method for identifying unstructured transaction information based on natural language processing provided by an embodiment of the present invention specifically includes: steps S101 to S105.
[0037] S101, obtaining a transaction language text corresponding to the transaction quotation information sent by the user, and preprocessing the transaction language text;
[0038] S102, based on the conditional random field, performing transaction context word segmentation and part-of-speech recognition on the preprocessed transaction language text;
[0039] S103, performing meaning group segmentation and paragraph completion processing on the transaction language text according to the transaction context word segmentation result and part-of-speech recognition result;
[0040] S104, extracting part-of-speech features from the transaction context word segmentation results and part-of-speech recognition results, and vectorizing and modeling the part-of-speech features through a Bayesian network to perform global optimal recognition of the transaction language text;
[0041] S105. Perform matrix format verification on the global optimal recognition result, and perform matrix structuring processing on the meaning group segmentation and paragraph completion processing results based on the verification result, and output the matrix structuring processing result as the transaction quotation information recognition result.
[0042] In this embodiment, the transaction quotation information sent by the user is first preprocessed into a transaction language text to achieve the effect of semantic group perception and aggregation. Of course, the transaction quotation information can also refer to other securities transaction information, such as inquiry information, etc. On the basis of the conditional random field, the transaction language text is subjected to part-of-speech processing, that is, the transaction context segmentation and part-of-speech recognition, such as determining what the transaction context to which the part-of-speech in the transaction language text belongs is, and determining whether the part-of-speech is a verb or a noun, or other parts of speech, etc. Then, the transaction language text is subjected to paragraph processing, that is, semantic group segmentation and paragraph completion processing. The so-called semantic group refers to the various components of the sentence in the transaction language text divided according to the meaning and structure. Each component is called a semantic group, and the words in the same semantic group are closely related to each other and cannot be split at will, otherwise it will cause misunderstanding. Then, according to the results of the part-of-speech processing and paragraph processing, the part-of-speech features are extracted from the transaction language text, and a vectorization model is established based on this to achieve global optimal recognition of the transaction language text. Then, the transaction language text is checked in matrix format according to the global optimal recognition result, and the result obtained by the check can be output as the recognition processing result of the transaction language text.
[0043] This embodiment performs part-of-speech processing and paragraph processing on the transaction quotation information of the user, and parses various types of transaction "jargon" into specific quotation and transaction information, thereby improving the processing efficiency of transaction quotation information. This embodiment can automatically identify inquiry, quotation, and transaction information of the transaction language text generated by negotiation and communication. And intelligent error correction can be performed on the transaction language text. In a specific application scenario, the recognition accuracy of the transaction language text of this embodiment reaches 99.94%. In another specific application scenario, the average time consumed by this embodiment from capturing to structured output for each group of market conditions is in the order of 20ms. In summary, this embodiment can greatly reduce the labor cost of over-the-counter transaction inquiry and quotation, and improve the processing efficiency of transaction quotation information. In addition, this embodiment also supports multi-machine distributed deployment, and accesses a large number of chat groups and friend users to realize the processing of real-time messages, and can be encapsulated into a high-performance streaming computing unit on the message stream bus.
[0044] In one embodiment, if Figure 2 As shown, the step S101 includes: steps S201 to S204.
[0045] S201, performing coarse sense group perception on the transaction language text through a preset transaction verb table, and removing non-market information in the transaction language text according to the coarse sense group perception result;
[0046] S202, using index message queue technology and constructing an index queue from the transaction language text through a hash algorithm to achieve streaming short-term deduplication;
[0047] S203, performing context tracing according to the order of the transaction language texts in the index queue, and performing aggregation processing on the transaction language texts by marking line separators according to the acquisition time of the transaction language texts;
[0048] S204: construct an adaptive sentence segmentation algorithm by combining the decision tree, the text punctuation attributes, and the character distribution characteristics between punctuation marks, and use the adaptive sentence segmentation algorithm to pre-segment the transaction language text into word segments.
[0049] In this embodiment, the transaction language text is first collected and analyzed, that is, the transaction language meaning group is roughly perceived through the preset transaction verb table to judge the market information in the transaction language text. Here, the preset transaction verb table can be a verb table extracted based on the regular expression matching algorithm, and the transaction verb table is formed after the trader manually reviews the verbs that are meaningful to the scene. Next, the multi-source data in the transaction language text is deduplicated. Specifically, the transaction language text is pressed into the deduplication queue after hash processing, and the hash is discarded if it is repeated. After the message in the deduplication queue times out a certain limit or reaches the end of the queue, it will be pressed out of the queue. When the transaction language text is aggregated, the deduplication messages in the previous step are aggregated according to the length of the sender's sending interval, and the aggregated information is marked with a line separator. Then the transaction language text is pre-segmented using an adaptive sentence segmentation algorithm. The adaptive sentence segmentation algorithm specifically comprises: constructing a decision tree (e.g., a C4.5 decision tree, a decision tree algorithm), setting the separation level of the punctuation according to the text punctuation attributes and the character distribution characteristics between the punctuation, for example, setting the level of the period and carriage return to be higher than the level of the comma and semicolon.
[0050] In one embodiment, if Figure 3 As shown, the step S102 includes: steps S301 to S306.
[0051] S301, constructing a target part-of-speech vector corresponding to a key element of the trading market based on the trading context of the trading language text, and constructing a part-of-speech determination expert rule for each element in the target part-of-speech vector based on a regularized expression, thereby constructing a part-of-speech determiner;
[0052] S302, using the part-of-speech determiner to perform part-of-speech determination on the semi-structured information in the transaction language text, and correcting the part-of-speech determination result through a keyword matching header recognizer;
[0053] S303, embedding the part-of-speech determiner into a conditional random field model, and using a pre-constructed continuous character cursor to consume the unstructured information in the transaction language text character by character to perform part-of-speech determination;
[0054] S304, recording the start and end positions of each character on the continuous character cursor, and connecting the start and end positions of each character by the greediness of the hit determination rule to obtain the corresponding local conditional cost distance;
[0055] S305, constructing a corresponding position transfer matrix according to the field length of the character, and using the local conditional cost distance to sequentially replace the coordinate distance values of the corresponding start and end positions in the position transfer matrix;
[0056] S306. Based on the coordinate distance value, the shortest path of the position transfer matrix is calculated by a shortest path algorithm model, and matching words on the shortest path are used as an optimal word segmentation scheme to perform transaction context word segmentation and part-of-speech recognition on the transaction language text.
[0057] In this embodiment, the target part-of-speech vector (i.e., the key element type of the market) of the market is constructed based on the transaction context, and the part-of-speech determination expert rules are constructed one by one for the elements in the target part-of-speech vector based on the regular expression rules, and are uniformly encapsulated as a part-of-speech determiner. The part-of-speech determiner supports access to expert rules for the same part-of-speech, and counts the rule greediness t based on the frequency of matching existing part-of-speech samples on different rules. The higher the pass rate, the greedier it is.
[0058] For semi-structured messages with good segment separation, the part-of-speech determiner can be used directly for determination, and the header recognizer based on keyword matching can be used for correction. For unstructured messages, such as when the paragraph or punctuation separation is unclear or irregular, the word adhesion separation algorithm is used for processing, that is, a continuous character cursor Cursor is constructed, the part-of-speech determiner is embedded in the CRF conditional random field model, and the segment to be identified is eaten character by character. Each time it moves, the part-of-speech is determined for the cursor context, the start and end positions of the determined words (or the start and end positions) are recorded, and the start and end positions are connected with the greediness t of the hit determination rule to form a local conditional cost distance d. The more greedy the matching distance, the higher the cost. At the same time, assuming that the length of the segment to be identified is n, a position transfer matrix of (n+1)*(n+1) is constructed, and the adjacent position transfer distances are all large numbers. Then, the local conditional cost distance d matched by the cursor is replaced in turn with the distance value of the start and end position coordinates corresponding to the position transfer matrix. When the cursor has completed the entire word segment, the shortest path algorithm model (such as the Dijkstra algorithm) is used to calculate the shortest path for the position transfer matrix, and the matching words on the shortest path are used as the optimal word segmentation scheme. In the process of continuous segment recognition and processing, the above-mentioned word segmentation is used for the word segment that meets the market part-of-speech vector and is continuously recorded until the current vector has repeated attributes and then the sentence is broken. It can also realize large-scale real-time message processing while maintaining a certain degree of intelligence.
[0059] In one embodiment, if Figure 4As shown, the step S103 includes: steps S401 to S403.
[0060] S401, performing context distance calculation on the target part-of-speech vector according to the transaction context word segmentation result and part-of-speech recognition result, and calculating the context difference using an edit distance algorithm;
[0061] S402, using a cumulative loss method to determine the context difference, and separating the transaction language text into segments according to the determination result;
[0062] S403: performing part-of-speech alignment on the transaction language text sentence by sentence, and completing the missing value in each sentence by using a previous value filling method.
[0063] In this embodiment, based on the transaction context word segmentation results and part-of-speech identification results obtained in the above steps, the constructed target part-of-speech vector is subjected to context distance calculation, and the context difference is calculated using an edit distance algorithm (such as the LevenshteinDistance Levenshtein distance algorithm), and the context difference is determined using the cumulative loss method. That is, if the context difference is zero, the cumulative loss is forgotten, and if the context difference is not zero, the cumulative loss is accumulated. The specific size of the context difference is used to determine whether to truncate the meaning group to separate the paragraph. For the separated paragraphs, the part-of-speech alignment is performed sentence by sentence, and the missing values in the paragraph are completed by filling in the previous value.
[0064] In one embodiment, the step S104 includes:
[0065] A controversial part-of-speech identifier for a meaning group is constructed to correct the group controversial problem generated by the part-of-speech recognition result, and the correction process is classified step by step based on priority rules.
[0066] In this embodiment, since there are often group disputes in the process of part-of-speech recognition, it is necessary to construct a meaning group controversial part-of-speech identifier to correct group disputes. Specifically, a priority rule set can be used to divert and classify group disputes. In a specific embodiment, group disputes can be divided into three categories (or three levels) for processing: header feature recognition, value feature recognition, and statistical distribution feature recognition. If the current level cannot effectively resolve the dispute, it can flow into the next level for resolution.
[0067] Furthermore, in one embodiment, if Figure 5 As shown, the constructing of the controversial part-of-speech identifier corrects the group controversial problem generated by the part-of-speech recognition result, and classifies the correction process step by step based on the priority rule, including: steps S501 to S504.
[0068] S501, performing header feature recognition on group dispute issues in unstructured messages by matching header key characters;
[0069] S502, modifying the category of the group dispute problem by combining the numerical value range boundary constraint and text enumeration type assertion of the group dispute problem to achieve value feature recognition;
[0070] S503, obtaining text types and numerical types corresponding to the group dispute issues in the structured information, performing distribution statistics on the text types and numerical types, and marking part-of-speech tags on the group dispute issues according to the distribution statistics results;
[0071] S504: construct a corresponding dispute feature matrix based on the part-of-speech tags, and use a supervised model to train the dispute feature matrix to obtain a corresponding part-of-speech classification result, and then use the part-of-speech classification result as the global optimal recognition result.
[0072] In this embodiment, header feature recognition is achieved through header key character matching, and header feature recognition is usually mined from redundant information in unstructured messages. When performing value feature recognition, category correction is achieved through numerical value domain boundary constraints and text enumeration type assertions for specific parts of speech in group dispute issues. When performing distribution feature recognition, distribution statistics are performed on the text type and numerical type of group dispute issues, where the text type counts features such as character frequency, field length, and digital proportion; the numerical type counts features such as median, variance, range, and coefficient of variation gain. After part-of-speech tags are annotated according to the distribution statistics results, a dispute feature matrix can be constructed and used. A fast supervised model such as Bayes (Naive Bayes) and / or CART tree is used to train the dispute feature matrix, and the trained model is used to classify the part of speech of the collective dispute issues and form a final decision.
[0073] This embodiment can effectively improve the accuracy of market identification. In a specific application scenario, the accuracy of market identification can be improved from about 95% to more than 99.9%, so that the overall system can be put into production reliably.
[0074] In one embodiment, if Figure 6 As shown, the step S105 includes: steps S601 to S604.
[0075] S601, constructing a part-of-speech projection matrix according to the results of the meaning group segmentation and paragraph completion processing, and calculating the inner product between the part-of-speech projection matrix and a preset test vector, and then judging whether the basic market elements of the transaction language text are complete according to the inner product;
[0076] S602, performing matrix structuring processing on the part-of-speech projection matrix by common attribute aggregation, non-common column data alignment or column reordering method;
[0077] S603, performing standardized conversion on the parts of speech in the transaction language text, translating the word values corresponding to the parts of speech and reducing the numerical values to a standardized data structure that complies with production specifications;
[0078] S604: Output the standardized data structure through multiple asynchronous threads.
[0079] In this embodiment, the matrix format check is mainly carried out by constructing a check vector T, calculating the inner product p of the part-of-speech projection matrix formed by the completed phrase paragraph, and if the value of the inner product p is not zero, it indicates whether the basic elements of the market are complete and can be output. The output format specification mainly structures the phrase matrix by means of common attribute aggregation, non-common column data alignment, column reordering, etc. In addition, by constructing an automatic part-of-speech translator, the standardized conversion of the word values of each part of speech is realized, and through the marking in the above-mentioned recognition process and the accumulated expert translation rules, the word values are translated and numerically reduced to a standardized data structure that meets the production specifications. Multiple asynchronous threads are used to push the results after translation and reduction to the message queue in real time, and quickly output by writing to a database and / or writing to an archive file.
[0080] This embodiment performs automatic part-of-speech translation and numerical value reduction on the transaction language text, thereby realizing structured generation and real-time push of market information and transaction information.
[0081] Figure 7 A schematic block diagram of an unstructured transaction information identification device 700 based on natural language processing provided by an embodiment of the present invention, the device 700 includes:
[0082] The preprocessing unit 701 is used to obtain the transaction language text corresponding to the transaction quotation information sent by the user, and preprocess the transaction language text;
[0083] The part-of-speech processing unit 702 is used to perform transaction context word segmentation and part-of-speech recognition on the pre-processed transaction language text based on the conditional random field;
[0084] The paragraph processing unit 703 is used to perform meaning group segmentation and paragraph completion processing on the transaction language text according to the transaction context word segmentation result and part-of-speech recognition result;
[0085] The optimal recognition unit 704 is used to extract part-of-speech features from the transaction context word segmentation results and part-of-speech recognition results, and to vectorize and model the part-of-speech features through a Bayesian network to perform global optimal recognition on the transaction language text;
[0086] The first structured processing unit 705 is used to perform matrix format verification on the global optimal recognition result, and based on the verification result, perform matrix structured processing on the meaning group segmentation and paragraph completion processing results, and output the matrix structured processing results as transaction quotation information recognition results.
[0087] In one embodiment, if Figure 8 As shown, the pre-processing unit 701 includes:
[0088] The perception unit 801 is used to perform coarse perception of meaning groups on the transaction language text by using a preset transaction verb table, and remove non-market information in the transaction language text according to the coarse perception result of meaning groups;
[0089] The deduplication unit 802 is used to utilize the index message queue technology and construct an index queue with the transaction language text through a hash algorithm to achieve streaming short-term deduplication;
[0090] An aggregation unit 803 is used to perform context tracing according to the order of the transaction language text in the index queue, and to perform aggregation processing on the transaction language text by marking line separators according to the acquisition time of the transaction language text;
[0091] The pre-segmentation unit 804 is used to construct an adaptive sentence segmentation algorithm by combining the decision tree with the text punctuation attributes and the character distribution characteristics between punctuation marks, and to pre-segment the transaction language text into word segments by using the adaptive sentence segmentation algorithm.
[0092] In one embodiment, if Fig. 9 As shown, the part-of-speech processing unit 702 includes:
[0093] A construction unit 901 is used to construct a target part-of-speech vector corresponding to a key element of the trading market based on the trading context of the trading language text, and construct a part-of-speech determination expert rule for each element in the target part-of-speech vector based on a regularized expression, thereby constructing a part-of-speech determiner;
[0094] A correction unit 902, configured to use the part-of-speech determiner to perform part-of-speech determination on the semi-structured information in the transaction language text, and to correct the part-of-speech determination result through a keyword matching header recognizer;
[0095] The determination unit 903 is used to embed the part-of-speech determiner into the conditional random field model, and use the pre-constructed continuous character cursor to consume the unstructured information in the transaction language text character by character to perform part-of-speech determination;
[0096] A connection unit 904 is used to record the start and end positions of each character on the continuous character cursor, and connect the start and end positions of each character according to the greediness of the hit determination rule to obtain the corresponding local conditional cost distance;
[0097] A replacement unit 905 is used to construct a corresponding position transfer matrix according to the field length of the character, and use the local conditional cost distance to sequentially replace the coordinate distance values of the corresponding start and end positions in the position transfer matrix;
[0098] The path calculation unit 906 is used to calculate the shortest path of the position transfer matrix through the shortest path algorithm model based on the coordinate distance value, and use the matching words on the shortest path as the optimal word segmentation scheme to perform transaction context word segmentation and part of speech recognition on the transaction language text.
[0099] In one embodiment, if Fig.10 As shown, the paragraph processing unit 703 includes:
[0100] The context distance calculation unit 1001 is used to perform context distance calculation on the target part-of-speech vector according to the transaction context word segmentation result and the part-of-speech recognition result, and calculate the context difference degree by using an edit distance algorithm;
[0101] The difference determination unit 1002 is used to determine the context difference by using a cumulative loss method, and separate the transaction language text into segments according to the determination result;
[0102] The missing value filling unit 1003 is used to perform part-of-speech alignment on the transaction language text sentence by sentence, and fill in the missing value in each sentence by using the previous value filling method.
[0103] In one embodiment, the optimal identification unit 704 includes:
[0104] The dispute correction unit is used to construct a controversial part-of-speech identifier for a meaning group to correct the group dispute problem generated by the part-of-speech recognition result, and classify the correction process step by step based on the priority rule.
[0105] In one embodiment, if Fig.11 As shown, the dispute correction unit includes:
[0106] The header feature recognition unit 1101 is used to perform header feature recognition on group dispute issues in unstructured messages by matching header key characters;
[0107] The value feature identification unit 1102 is used to modify the category of the group dispute problem by combining the numerical value range boundary constraint and the text enumeration type assertion of the group dispute problem to achieve value feature identification;
[0108] A distribution feature identification unit 1103 is used to obtain text types and numerical types corresponding to group dispute issues in the structured information, perform distribution statistics on the text types and numerical types, and label the group dispute issues with part-of-speech tags according to the distribution statistics results;
[0109] The matrix construction unit 1104 is used to construct a corresponding dispute feature matrix based on the part-of-speech tags, and use a supervised model to train the dispute feature matrix to obtain a corresponding part-of-speech classification result, and then use the part-of-speech classification result as the global optimal recognition result.
[0110] In one embodiment, if Fig.12 As shown, the first structured processing unit 705 includes:
[0111] The inner product calculation unit 1201 is used to construct a part-of-speech projection matrix according to the results of the meaning group segmentation and paragraph completion processing, and calculate the inner product between the part-of-speech projection matrix and a preset test vector, and then determine whether the basic market elements of the transaction language text are complete according to the inner product;
[0112] A second structured processing unit 1202 is used to perform matrix structured processing on the part-of-speech projection matrix by common attribute aggregation, non-common column data alignment or column reordering method;
[0113] A standardization conversion unit 1203 is used to perform standardization conversion on the parts of speech in the transaction language text, translate the word values corresponding to the parts of speech, and convert the values into a standardized data structure that meets the production specifications;
[0114] The output unit 1204 is used to output the standardized data structure through multiple asynchronous threads.
[0115] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.
[0116] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0117] The embodiment of the present invention also provides a distributed computer single point and cluster deployment device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment can be implemented. Of course, the computer device may also include various network interfaces, power supplies and other components, and support single-machine computing and multi-machine parallel computing to implement the steps provided in the above embodiment.
[0118] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.
[0119] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
Claims
1. A method for identifying unstructured transaction information based on natural language processing, characterized in that: include: Acquire the transaction language text corresponding to the transaction quotation information sent by the user, and pre-process the transaction language text; Based on the conditional random field, the preprocessed transaction language text is segmented in the transaction context and the part of speech is identified; According to the word segmentation results and part-of-speech recognition results of the transaction context, the transaction language text is segmented into meaning groups and paragraphs are completed; Extracting part-of-speech features from the transaction context word segmentation results and part-of-speech recognition results, and vectorizing and modeling the part-of-speech features through a Bayesian network to perform global optimal recognition of the transaction language text; The global optimal recognition result is checked in matrix format, and based on the check result, the meaning group segmentation and paragraph completion processing results are matrix structured, and the matrix structured processing results are output as the transaction quotation information recognition result.
2. The method for identifying unstructured transaction information based on natural language processing according to claim 1, characterized in that: The acquiring of the transaction language text corresponding to the transaction quotation information sent by the user and preprocessing the transaction language text includes: The transaction language text is roughly sensed by a preset transaction verb table, and non-market information in the transaction language text is removed according to the result of the rough sense group perception; Utilize the index message queue technology and construct the index queue of the transaction language text through the hash algorithm to achieve streaming short-term deduplication; Performing context tracing according to the order of the transaction language texts in the index queue, and performing aggregation processing on the transaction language texts by marking line separators according to the acquisition time of the transaction language texts; An adaptive sentence segmentation algorithm is constructed by combining the decision tree with text punctuation attributes and character distribution characteristics between punctuation marks, and the transaction language text is pre-segmented into word segments using the adaptive sentence segmentation algorithm.
3. The method for identifying unstructured transaction information based on natural language processing according to claim 1, characterized in that: The process of performing transaction context word segmentation and part-of-speech recognition on the preprocessed transaction language text based on the conditional random field includes: Constructing a target part-of-speech vector corresponding to the key elements of the trading market based on the trading context of the trading language text, and constructing a part-of-speech determination expert rule for each element in the target part-of-speech vector based on a regularized expression, thereby constructing a part-of-speech determiner; Using the part-of-speech determiner to determine the part-of-speech of the semi-structured information in the transaction language text, and correcting the part-of-speech determination result by a keyword matching header recognizer; The part-of-speech determiner is embedded in a conditional random field model, and a pre-constructed continuous character cursor is used to consume the unstructured information in the transaction language text character by character to perform part-of-speech determination; Record the start and end positions of each character on the continuous character cursor, and connect the start and end positions of each character through the greediness of the hit determination rule to obtain the corresponding local conditional cost distance; Constructing a corresponding position transfer matrix according to the field length of the character, and using the local conditional cost distance to sequentially replace the coordinate distance values of the corresponding start and end positions in the position transfer matrix; Based on the coordinate distance value, the shortest path of the position transfer matrix is calculated through the shortest path algorithm model, and the matching words on the shortest path are used as the optimal word segmentation scheme to perform transaction context word segmentation and part of speech recognition on the transaction language text.
4. The method for identifying unstructured transaction information based on natural language processing according to claim 3, characterized in that: The processing of segmenting the transaction language text into sense groups and completing paragraphs according to the transaction context word segmentation results and part-of-speech recognition results includes: According to the transaction context word segmentation results and part-of-speech recognition results, context distance calculation is performed on the target part-of-speech vector, and the context difference degree is calculated using an edit distance algorithm; Using a cumulative loss method, the context difference is determined, and the transaction language text is separated into segments according to the determination result; The transaction language text is aligned sentence by sentence based on the part of speech, and the missing values in each sentence are completed by using the previous value filling method.
5. The method for identifying unstructured transaction information based on natural language processing according to claim 1, characterized in that: The extracting part-of-speech features from the transaction context word segmentation results and part-of-speech recognition results, and vectorizing and modeling the part-of-speech features through a Bayesian network to perform global optimal recognition of the transaction language text, includes: A controversial part-of-speech identifier for a meaning group is constructed to correct the group controversial problem generated by the part-of-speech recognition result, and the correction process is classified step by step based on priority rules.
6. The method for identifying unstructured transaction information based on natural language processing according to claim 5, characterized in that: The method of constructing a controversial part-of-speech identifier for a meaning group corrects the controversial group problem generated by the part-of-speech recognition result, and classifies the correction process based on a priority rule, including: The header features of group controversial issues in unstructured messages are identified by matching key characters in the header; Combined with the numerical domain boundary constraints and text enumeration type assertions of group dispute issues, the category of group dispute issues is modified to achieve value feature recognition; Obtaining text types and numerical types corresponding to group dispute issues in the structured information, performing distribution statistics on the text types and numerical types, and marking part-of-speech tags on the group dispute issues according to the distribution statistics results; A corresponding dispute feature matrix is constructed based on the part-of-speech tags, and the dispute feature matrix is trained using a supervised model to obtain a corresponding part-of-speech classification result, which is then used as the global optimal recognition result.
7. The method for identifying unstructured transaction information based on natural language processing according to claim 6, characterized in that: The matrix format verification of the global optimal recognition result and matrix structured processing of the meaning group segmentation and paragraph completion processing results based on the verification result include: Constructing a part-of-speech projection matrix according to the results of the meaning group segmentation and paragraph completion processing, and calculating the inner product between the part-of-speech projection matrix and a preset test vector, and then judging whether the basic market elements of the transaction language text are complete according to the inner product; Performing matrix structuring processing on the part-of-speech projection matrix by common attribute aggregation, non-common column data alignment or column reordering method; Standardizing and converting the parts of speech in the transaction language text, translating the word values corresponding to the parts of speech, and numerically reducing them to a standardized data structure that complies with production specifications; The standardized data structure is outputted through a plurality of asynchronous threads.
8. A device for identifying unstructured transaction information based on natural language processing, characterized in that: include: A preprocessing unit, used to obtain a transaction language text corresponding to the transaction quotation information sent by the user, and preprocess the transaction language text; A part-of-speech processing unit is used to perform transaction context word segmentation and part-of-speech recognition on the preprocessed transaction language text based on conditional random fields; A paragraph processing unit, used to perform meaning group segmentation and paragraph completion processing on the transaction language text according to the transaction context word segmentation result and part-of-speech recognition result; An optimal recognition unit, used for extracting part-of-speech features from the transaction context word segmentation results and part-of-speech recognition results, and vectorizing and modeling the part-of-speech features through a Bayesian network to perform global optimal recognition of the transaction language text; The first structured processing unit is used to perform matrix format verification on the global optimal recognition result, and based on the verification result, perform matrix structured processing on the meaning group segmentation and paragraph completion processing results, and output the matrix structured processing results as the transaction quotation information recognition result.
9. A distributed computer single point and cluster deployment device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for identifying unstructured transaction information based on natural language processing as described in any one of claims 1 to 7, and supports single-machine computing and multi-machine parallel computing, so as to implement the method for identifying unstructured transaction information based on natural language processing as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for identifying unstructured transaction information based on natural language processing according to any one of claims 1 to 7 is implemented.
