Natural language processing method and device

Abstract

The embodiment of the invention provides a natural language processing method and equipment, and the method comprises the steps: obtaining a lexical item set corresponding to a to-be-processed initial input statement, and enabling the lexical item set to comprise an initial lexical item included in the initial input statement, and an extended lexical item of the initial lexical item; performing word order reduction on lexical items in the lexical item set to obtain an extended statement of the initial input statement; and obtaining a processing result based on the extension statement and a natural language processing model. According to the embodiment of the invention, the initial lexical item and the extended lexical item included in the initial input statement can be acquired and restored into the extended statement with a smooth word order, so that the initial input statement can be extended and enhanced, and the obtained extended statement can be better understood by a natural language processing model; and then natural language processing is performed in combination with the natural language processing model, so that the accuracy and stability of a processing result can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer and network communication technology, and in particular to a method and apparatus for natural language processing. Background Technology

[0002] Search engines, such as Opensearch and Elasticsearch, typically have a rich ecosystem of analyzers that can intervene in the query input, such as word segmentation, synonyms, ICU (International Components for Unicode), and pinyin. The analyzer ecosystem is built on terms, which is more suitable for full-text search scenarios.

[0003] However, with the recent surge in applications related to Natural Language Processing (NLP) models, there is a lack of alternative intervention methods similar to search engine analyzers in these applications. Term-based analyzers like search engine analyzers are not suitable for NLP models. Summary of the Invention

[0004] This disclosure provides a natural language processing method and apparatus to overcome the above-mentioned problems.

[0005] In a first aspect, embodiments of this disclosure provide a natural language processing method, including:

[0006] Obtain the term set corresponding to the initial input statement to be processed, wherein the term set includes the initial term included in the initial input statement and the extended term of the initial term;

[0007] The word order of the words in the word set is restored to obtain the expanded sentence of the initial input sentence;

[0008] Based on the expanded statement and the natural language processing model, the processing result is obtained.

[0009] Secondly, embodiments of this disclosure provide a natural language processing device, comprising:

[0010] The analysis unit is used to obtain the term set corresponding to the initial input statement to be processed, wherein the term set includes the initial term included in the initial input statement and the extended term of the initial term;

[0011] The word order restoration unit is used to restore the word order of the word items in the word set to obtain the expanded sentence of the initial input sentence;

[0012] The model processing unit is used to obtain the processing result based on the extended statement and the natural language processing model.

[0013] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0014] The memory stores computer-executed instructions;

[0015] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the natural language processing method as described in the first aspect and various possible designs of the first aspect.

[0016] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the natural language processing method described in the first aspect and various possible designs of the first aspect.

[0017] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the natural language processing method as described in the first aspect and various possible designs of the first aspect.

[0018] The natural language processing method and apparatus provided in this disclosure acquire a term set corresponding to an initial input statement to be processed, wherein the term set includes initial terms included in the initial input statement and extended terms of the initial terms; the term set is used to restore the word order of the terms in the term set to obtain an extended statement of the initial input statement; and a processing result is obtained based on the extended statement and a natural language processing model. In this disclosure, by acquiring the initial terms and extended terms included in the initial input statement and restoring them into a grammatically correct extended statement, the initial input statement can be expanded and enhanced, making the obtained extended statement better understood by the natural language processing model. Furthermore, by combining this extended statement with the natural language processing model for natural language processing, the accuracy and stability of the processing result can be improved. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1A schematic diagram illustrating a scenario of a natural language processing method provided in an embodiment of this disclosure;

[0021] Figure 2 A schematic flowchart of a natural language processing method provided in an embodiment of this disclosure;

[0022] Figure 3 A schematic flowchart of a natural language processing method provided in another embodiment of this disclosure;

[0023] Figure 4 A structural block diagram of a natural language processing device provided in an embodiment of this disclosure;

[0024] Figure 5 This is a schematic diagram of the hardware structure of a natural language processing device provided in an embodiment of the present disclosure. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0026] Search engines, such as Opensearch and Elasticsearch, typically have a rich ecosystem of analyzers that intervene in processing the input (query), such as word segmentation, synonyms, ICU, pinyin, etc. The analyzer ecosystem is built based on terms, which is more suitable for full-text search scenarios.

[0027] However, with the recent surge in applications related to natural language processing (NLP) models, there is a lack of alternative intervention methods similar to search engine analyzers. Term-based analyzers are not suitable for NLP models because they are highly sensitive to delimiters and word order. Term-based analyzers, on the other hand, produce terms that, if directly used as input, would result in very poor processing performance.

[0028] For example, when searching for "traffic accident leading to work injury disputes", after intervention by the search engine analyzer, results similar to the following can be obtained:

[0029] "Work-related injuries caused by traffic accidents should be excluded from disputes and should be caused by other work-related reasons or scenarios."

[0030] This type of query, which is broken down into terms, is suitable for full-text retrieval, but not for input to natural language models, especially for semantic retrieval or as a natural language processing prompt.

[0031] To address the aforementioned technical problems, this disclosure provides a natural language processing method that can reuse a search engine analyzer to obtain initial terms and extended terms included in the initial input statement, resulting in a term set. The term set is then restored to its original word order, forming a coherent extended statement. This method expands and enhances the initial input statement, enabling the extended statement to be better understood by the natural language processing model. Furthermore, by combining this extended statement with the natural language processing model for further processing, the accuracy and stability of the processing results can be improved.

[0032] Application scenarios of the natural language processing method of this disclosure embodiment are as follows: Figure 1 As shown, it can be applied to any electronic device such as a terminal device or server. Specific scenarios can be any application of natural language processing models, such as various chatbots, data retrieval such as professional literature knowledge bases and help document knowledge bases, shopping guides, after-sales service robots, and any other natural language processing scenario. First, the initial input statement to be processed is obtained. The search engine analyzer is reused to obtain the term set corresponding to the initial input statement. The term set includes the initial terms included in the initial input statement and the extended terms of the initial terms. The word order of the terms in the term set is restored to obtain the extended statement of the initial input statement. Based on the extended statement and the natural language processing model, the processing result is obtained.

[0033] The natural language processing method of this disclosure will be described in detail below with reference to specific embodiments.

[0034] refer to Figure 2 , Figure 2 This is a schematic flowchart of a natural language processing method according to an embodiment of the present disclosure. The method of this embodiment can be applied to any electronic device such as a terminal device or server. The natural language processing method includes:

[0035] S201. Obtain the term set corresponding to the initial input statement to be processed, wherein the term set includes the initial term included in the initial input statement and the extended term of the initial term.

[0036] In this embodiment, in scenarios such as semantic retrieval based on natural language (statement) processing, the initial input statement (query) to be processed can be obtained first. The initial input statement is a natural language statement. Then, based on the initial input statement to be processed, the initial terms included in the initial input statement and the extended terms of the initial terms can be obtained to form a term set.

[0037] Optionally, in this embodiment, the analyzer of a search engine can be invoked to obtain the term set corresponding to the initial input statement. The search engine can be, but is not limited to, Opensearch, Elasticsearch, etc., and its analyzer is a component within the search engine used to process text data. It can be used to fulfill specific text processing needs and may include, but is not limited to, one or more of the following: a word segmenter, a thesaurus analyzer, an ICU analyzer (International Components for Unicode, also known as a multilingual text analyzer), a pinyin analyzer, etc. The input to the analyzer can be text (statements or terms) of any form, and the output is terms. Of course, this embodiment is not limited to using the analyzer of a search engine to obtain the term set corresponding to the initial input statement; other feasible methods can also be used to obtain the term set corresponding to the initial input statement, such as using other word segmentation tools, thesaurus tools, pinyin tools, ICU tools, etc.

[0038] In scenarios where retrieval is based on terms, such as full-text search, a rich ecosystem of analyzers is typically used to intervene in the initial query. These analyzers include term segmentation, synonyms, ICU (International Components for Unicode), pinyin, and other term-based analyzers, which enrich the final terms and improve retrieval accuracy. In this embodiment, in scenarios based on natural language (sentence) processing, such as semantic retrieval, in order to enrich and expand the initial input statement in the same way, the analyzer ecosystem of the search engine is reused to intervene in the initial input statement. That is, one or more term-based analyzers, including but not limited to term segmentation, synonyms, ICU, pinyin, etc., can be used to process the initial input statement and obtain the term set corresponding to the initial input statement.

[0039] The term set corresponding to the initial input statement may specifically include the initial term included in the initial input statement (which can be directly obtained through the word segmenter) and the extended term of the initial term (obtained through other term-based analyzers). For example, the extended term of the initial term includes, but is not limited to, one or more of the following: synonyms of the initial term (obtained through the synonym analyzer), pinyin of the initial term (obtained through the pinyin analyzer), standard format terms corresponding to the initial term (obtained through the ICU analyzer), etc.

[0040] For example, suppose the initial input statement is "traffic accident leading to work injury dispute". After intervention by multiple analyzers, the following term set can be obtained:

[0041] "Work-related injuries caused by traffic accidents should be excluded from disputes and should be caused by other work-related reasons or scenarios."

[0042] Furthermore, when retrieving individual terms from the term set, especially when retrieving terms through the search engine's analyzer, the position information of each term can also be obtained, including its position and offset in the initial input statement. The position indicates the order of the term in the initial statement, while the offset indicates the position of the term's starting character and syllable number, so that it can be accurately mapped back to the initial input statement, including the start offset and end offset. The position and offset of the extended terms of the initial term usually have a certain inheritance relationship with the position and offset of the initial term.

[0043] S202. Restore the word order of the words in the word set to obtain the expanded sentence of the initial input sentence.

[0044] In this embodiment, since the term set contains not only the initial term but also extended terms of the initial term, and the order of the initial and extended terms may not conform to word order, it is necessary to reorder the terms in the term set to obtain a complete and fluent sentence, i.e., an extended sentence of the initial input sentence. This is more compatible with the natural language processing model in the following steps and more suitable for semantic retrieval scenarios. It may include the initial input sentence and the sentence constructed based on the extended terms. The ordering of the terms can be achieved using any known feasible method, and this embodiment does not impose any restrictions.

[0045] For example, in the above example, the word order is restored for the words in the word set:

[0046] "Work-related injuries resulting from traffic accidents should be excluded from disputes and should be caused by other work-related reasons or scenarios."

[0047] The result is that "traffic accidents lead to work-related injury disputes, and work-related injuries are caused by work-related reasons, and other scenarios should be excluded."

[0048] S203. Based on the extended statement and the natural language processing model, obtain the processing result.

[0049] In this embodiment, after obtaining the expanded sentence, a natural language processing (NLP) model can be used for further intervention. This includes directly using it as input to the NLP model to perform NLP tasks, including but not limited to retrieval and question answering by the NLP model. Alternatively, the expanded sentence can be further processed by the NLP model, and the processing results can be subjected to other NLP processing. NLP models are subfields of computer science, information engineering, and artificial intelligence, focusing on human-computer language interaction and exploring how to process and utilize natural language. In this embodiment, the expanded sentence can be further processed using the NLP model, or semantic retrieval can be achieved based on the NLP model.

[0050] In one alternative embodiment, the extended statement can be input into a natural language processing model, the extended statement can be rewritten by the natural language processing model to obtain the rewritten extended statement, and natural language processing can be performed on the rewritten extended statement to obtain the processing result.

[0051] In this embodiment, the extended statement can be further rewritten based on a natural language processing model to make its semantics richer and more comprehensive. It can also correct errors in the extended statement or rewrite it into a specific format (e.g., a prompt). The rewritten extended statement can then be applied to any natural language processing scenario, such as searching in a search engine (or in other scenarios), or any other natural language processing task, to obtain more accurate results. The specific rewriting process and subsequent applications of the rewritten extended statement are not limited in this embodiment. For example, when searching a database (such as a knowledge base or any other database) based on natural language, the database search can be performed based on the rewritten extended statement; when conducting question-and-answer sessions based on natural language (such as chatbots, shopping guides, or after-sales robots), questions can be asked based on the rewritten extended statement to obtain answers.

[0052] In another alternative embodiment, the extended statement can be input into a natural language processing model, the natural language processing model can obtain the sentence vector corresponding to the extended statement, and natural language processing can be performed based on the sentence vector to obtain the processing result.

[0053] In this embodiment, the extended statement can be embedded using a natural language processing model to obtain the sentence vector corresponding to the extended statement. Vector processing, such as vector retrieval, can then be performed based on the sentence vector to obtain more accurate results. The specific embedding and vector retrieval processes are not limited in this embodiment. For example, when retrieving data from a database (such as a knowledge base or any other database) based on natural language, vector retrieval can be performed on the database based on the sentence vector of the extended statement.

[0054] In another alternative embodiment, the extended statement can be queried using a natural language model to obtain query results and a matching score between the query results and the extended statement.

[0055] In this embodiment, semantic retrieval of the extended statement can be performed directly using a natural language processing model. For example, semantic retrieval can be performed from a preset database, and query results matching the extended statement can be output. A matching score between the query results and the extended statement can also be provided. The specific process of performing the retrieval based on the natural language processing model and providing the matching score is not limited in this embodiment. For example, when retrieving data from a database (such as a knowledge base or any other database) based on natural language, the natural language processing model can be directly used to retrieve data from the database, obtaining one or more search results, and providing a matching score between the search results and the extended statement.

[0056] Optionally, the above three embodiments based on extended sentences and natural language processing models can be combined in any way without conflict.

[0057] The natural language processing method provided in this embodiment obtains a term set corresponding to the initial input statement to be processed. This term set includes initial terms included in the initial input statement and extended terms of those initial terms. The term set is then used to restore the word order of the terms in the term set, resulting in an extended statement of the initial input statement. Based on the extended statement and a natural language processing model, the processing result is obtained. This embodiment achieves the expansion and enhancement of the initial input statement by obtaining the initial terms and extended terms included in the initial input statement and restoring them into a grammatically correct extended statement. This allows the obtained extended statement to be better understood by the natural language processing model, and further, by combining it with the natural language processing model for natural language processing, the accuracy and stability of the processing result can be improved.

[0058] Furthermore, the main workflow of natural language processing remains almost unchanged, with only minor interventions added to the recall and invocation of the natural language processing model. Specifically, this involves obtaining the term set and restoring the word order to obtain an expanded sentence from the initial input. Moreover, recall and question-answering optimization can be performed quickly anytime, anywhere. In actual implementation, all bad cases converged rapidly, for example:

[0059] In fresh food shopping scenarios, the poor response to questions due to regional dialects can be resolved by simply configuring a dialect explanation.

[0060] The "clone function" is not understood by the "multi-instance" natural language processing model. This can be resolved by directly configuring the clone function instructions.

[0061] The responses to "work injury" scenarios are inconsistent, so we directly configure work injury descriptions and command control statements.

[0062] Based on any of the above embodiments, step S202, which involves restoring the word order of the words in the word set to obtain the expanded sentence of the initial input sentence, may specifically include:

[0063] Traverse the terms in the term set, obtain the terms with connected position information from the term set, and construct an extended statement, wherein the position information is the position and / or offset value of the term in the initial input statement.

[0064] In this embodiment, when the initial terms and extended terms of the initial input statement are obtained through the analyzer of the search engine, the position information of each term, including the corresponding position and offset in the initial input statement, can also be obtained. This can be used to reflect the word order of the terms. The position and offset of the extended terms of the initial terms usually have a certain inheritance relationship with the position and offset of the initial terms. Therefore, the word order of the terms in the term set can be restored based on the position information. By sequentially obtaining the terms connected by the position information, the extended statement can be constructed.

[0065] Optionally, when traversing the terms in the term set, obtaining the terms with connected positional information from the term set, and constructing the extended sentence, such as... Figure 3 As shown, it may specifically include:

[0066] S301. The terms in the term set that have not been added with any extended statement are traversed sequentially, and it is determined whether the position information of the current term and the previous term that has been added with the current extended statement are connected.

[0067] If the location information is connected, then execute S302; if the location information is not connected, then execute S303.

[0068] S302. If the position information is connected, the current term is added to the current extended statement and is located after the previous term added to the current extended statement.

[0069] S303. If the location information is not connected, skip the current term;

[0070] Furthermore, after S302 or S303, it is determined whether the current traversal has ended. If it has not ended, the next term in the term set that has not been added to any extended statement is traversed, and S301 is executed again; if it is received, S304 is executed.

[0071] S304. At the end of the traversal, the construction of the current extended statement is completed, and the traversal of the terms in the term set that have not been added to any extended statement is restarted to construct new extended statements.

[0072] In this embodiment, since the position and offset of the extended terms of the initial term are usually inherited from the position and offset of the initial term, there may be overlap between the positions and offsets. Specifically, the startoffset of the extended terms of the initial term is the same as the startoffset of the initial term, but the position will keep increasing. In particular, the more parsers are used, the more severe the overlap between positions and offsets becomes.

[0073] For example, in the above example, the initial input statement is "traffic accident leading to work injury dispute". The word segmenter can obtain initial terms including "traffic accident leading to work injury dispute". After processing by some analyzers, extended terms (or phrases) of the initial term "work injury" are obtained, such as "work injury is caused by work-related reasons" and "other scenarios should be excluded". The startoffset of the initial term "dispute" is connected to the endoffset of the initial term "work injury". The startoffsets of the two extended terms (or phrases) of the initial term "work injury" are the same as those of the initial term "work injury", but their endoffsets are different from those of the initial term "work injury". Therefore, they are not connected to the startoffset of the initial term "dispute". Thus, for the initial term "work injury", the rule that the endoffset of the preceding term is connected to the startoffset of the following term can be used to determine that the subsequent term is "dispute". Therefore, whether terms are connected can be determined by whether their offsets are connected.

[0074] For example, if the analyzer further segments the extended terms of the initial term, such as segmenting the extended terms (or phrases) of the initial term "work injury" ("work injury is caused by work-related reasons") and "other scenarios should be excluded"), resulting in the extended term "work injury should exclude injuries caused by other work-related reasons," the start offsets of these extended terms are all the same as the initial term "work injury." However, the positions of the extended terms (or phrases) after segmentation of the same extended term (or phrase) show an increasing relationship. Therefore, for terms with the same start offset, the connection between the segmented extended terms (or phrases) can be determined by whether their positions are connected, thus reconstructing "work injury should exclude injuries caused by other work-related reasons" into "work injury is caused by work-related reasons" and "other scenarios should be excluded." Therefore, even with the same start offset, the connection between terms can be determined by whether their positions are connected.

[0075] Based on this, in this embodiment, the terms in the term set can be traversed sequentially, and each term is used only once. Therefore, the unused terms in the term set (i.e., terms that have not been added to any extension statement) can be traversed sequentially. During the traversal, for the current term, it can be determined whether the position information of the current term and the previous term added to the current extension statement are connected. If the position information is connected, it means that the current term is connected to the previous term added to the current extension statement. The current term is added to the current extension statement and placed after the previous term added to the current extension statement. Then, the next term is traversed. If the position information is not connected, it means that the current term is not connected to the previous term added to the current extension statement. The current term is skipped, and the next term is traversed. When the traversal ends, the construction of the current extension statement is completed. Since some terms were skipped during this traversal—meaning these terms were not used (i.e., not added to any expansion statement)—the unused terms in the term set (i.e., those not added to any expansion statement) can be traversed again sequentially to construct new expansion statements. In the above process, each traversal constructs one expansion statement.

[0076] Specifically, when determining whether the position information of the currently traversed term is connected to the position information of the previous term added to the current expanded statement, it may include:

[0077] Determine whether the starting offset value of the current term is connected to the ending offset value of the previous term added to the current expansion statement, and / or determine whether the starting offset value of the current term is the same as the starting offset value of the previous term added to the current expansion statement, and whether the position of the previous term added to the current expansion statement is in an increasing relationship with the position of the current term;

[0078] If the starting offset of the current term is connected to the ending offset of the previous term added to the current expansion statement, or if the starting offset of the current term is the same as the starting offset of the previous term added to the current expansion statement, and the position of the previous term added to the current expansion statement is in an increasing relationship with the position of the current term, then the position information of the current term is connected to that of the previous term added to the current expansion statement.

[0079] In other words, if either of these two conditions is met, it can be determined that the position information of the current term is connected to that of the previous term added to the current expansion statement; or, if the starting offset value of the current term is the same as the starting offset value of the previous term added to the current expansion statement, and the position of the previous term added to the current expansion statement is in an increasing relationship with the position of the current term.

[0080] Based on any of the above embodiments, since each traversal process sequentially traverses the unused terms in the term set (i.e., terms that have not been added to any extension statement), in order to facilitate determining whether each term has been unused during traversal, that is, whether a term has not been added to any extension statement, an array can be created based on the term set. Each bit of the array is used to store the state corresponding to each term in the term set. The state corresponding to the term is used to represent whether the term has been added to any extension statement. Optionally, the array can be a boolean array, with 1 indicating that a term has been added to any extension statement and 0 indicating that a term has not been added to any extension statement. During the traversal process, the terms in the term set that have not been added to any extension statement can be traversed sequentially based on the array, that is, the terms with a state value of 0 in the array can be retrieved sequentially from the term set for traversal. If any term is added to any extension statement, the state corresponding to that term in the array is updated, that is, 0 is updated to 1. This can quickly determine whether each term has been unused and improve traversal efficiency.

[0081] Corresponding to the natural language processing method in the above embodiments, Figure 4 This is a structural block diagram of a natural language processing device provided for embodiments of this disclosure. For ease of explanation, only the parts relevant to embodiments of this disclosure are shown. (Refer to...) Figure 4The natural language processing device 400 includes: an analysis unit 401, a word order restoration unit 402, and a model processing unit 403.

[0082] The analysis unit 401 is used to obtain a set of terms corresponding to the initial input statement to be processed, wherein the set of terms includes the initial terms included in the initial input statement and the extended terms of the initial terms;

[0083] The word order restoration unit 402 is used to restore the word order of the word items in the word set to obtain the expanded sentence of the initial input sentence;

[0084] The model processing unit 403 is used to obtain the processing result based on the extended statement and the natural language processing model.

[0085] In one or more embodiments of this disclosure, when the word order restoration unit 402 performs word order restoration on the words in the word set to obtain the expanded sentence of the initial input sentence, it is used to:

[0086] Traverse the terms in the term set, obtain the terms with connected position information from the term set, and construct an extended statement, wherein the position information is the position and / or offset value of the term in the initial input statement.

[0087] In one or more embodiments of this disclosure, when the word order restoration unit 402 traverses the word items in the word item set, obtains word items with connected positional information from the word item set, and constructs an extended sentence, it is used to:

[0088] The terms in the term set that have not been added with any extended statement are traversed sequentially, and it is determined whether the position information of the current term and the previous term that has been added with the current extended statement is connected.

[0089] If the positional information is contiguous, the current term is added to the current expanded statement, following the previous term added to the current expanded statement; or

[0090] If the location information is not connected, skip the current term;

[0091] At the end of the traversal, the construction of the current extended statement is completed, and the traversal of the terms in the term set that have not been added to any extended statement is restarted to construct new extended statements.

[0092] In one or more embodiments of this disclosure, the word order restoration unit 402 is further configured to:

[0093] An array is created based on the term set, wherein each bit of the array is used to store the state corresponding to each term in the term set, and the state corresponding to the term is used to characterize whether the term has been added to any extended statement;

[0094] During the traversal, if any word is added with any extended statement, the state corresponding to the word is updated in the array.

[0095] Accordingly, when the word order restoration unit 402 sequentially traverses the word items in the word set that have not been supplemented with any extended sentences, it is used to:

[0096] Based on the array, the terms in the term set that have not been added with any extended statements are traversed sequentially.

[0097] In one or more embodiments of this disclosure, the word order restoration unit 402, when determining whether the position information of the current traversed word item is connected to the position information of the previous word item added to the current expanded sentence, is used to:

[0098] Determine whether the current term being iterated over is connected to the position and / or offset of the previous term added to the current expansion statement.

[0099] In one or more embodiments of this disclosure, the word order restoration unit 402, when determining whether the position and / or offset value of the currently traversed word item is connected to the position and / or offset value of the previous word item added to the current expanded sentence, is used to:

[0100] Determine whether the starting offset value of the current term is connected to the ending offset value of the previous term added to the current expansion statement, and / or determine whether the starting offset value of the current term is the same as the starting offset value of the previous term added to the current expansion statement, and whether the position of the previous term added to the current expansion statement is in an increasing relationship with the position of the current term;

[0101] If the starting offset of the current term is connected to the ending offset of the previous term added to the current expansion statement, or if the starting offset of the current term is the same as the starting offset of the previous term added to the current expansion statement, and the position of the previous term added to the current expansion statement is in an increasing relationship with the position of the current term, then the position information of the current term is connected to that of the previous term added to the current expansion statement.

[0102] In one or more embodiments of this disclosure, when the model processing unit 403 obtains the processing result based on the extended statement and the natural language processing model, it is used to:

[0103] The extended statement is input into the natural language processing model, which rewrites the extended statement to obtain the rewritten extended statement. Natural language processing is then performed on the rewritten extended statement to obtain the processing result.

[0104] In one or more embodiments of this disclosure, when the model processing unit 403 obtains the processing result based on the extended statement and the natural language processing model, it is used to:

[0105] The extended statement is input into the natural language processing model, the natural language processing model obtains the sentence vector corresponding to the extended statement, and vector processing is performed on the sentence vector to obtain the processing result.

[0106] In one or more embodiments of this disclosure, when the model processing unit 403 obtains the processing result based on the extended statement and the natural language processing model, it is used to:

[0107] The extended statement is queried using the natural language model to obtain query results and a matching score between the query results and the extended statement.

[0108] In one or more embodiments of this disclosure, the expanded terms of the initial term include one or more of the following:

[0109] Synonyms of the initial term, pinyin of the initial term, and standard format terms corresponding to the initial term.

[0110] In one or more embodiments of this disclosure, the analysis unit 401, when acquiring the term set corresponding to the initial input statement to be processed, is used to:

[0111] The analyzer of the search engine is invoked to obtain the set of terms corresponding to the initial input statement to be processed. The analyzer is a component in the search engine used to process text data. The analyzer includes one or more of the following analyzers: word segmenter, synonym analyzer, pinyin analyzer, and multilingual text analyzer.

[0112] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0113] To implement the above embodiments, this disclosure also provides an electronic device.

[0114] refer to Figure 5The diagram illustrates a structural schematic of an electronic device 500 suitable for implementing embodiments of the present disclosure. The electronic device 500 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0115] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0116] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0117] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0118] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0119] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0120] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0121] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0123] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0124] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0125] In a first aspect, according to one or more embodiments of this disclosure, a natural language processing method is provided, comprising:

[0126] Obtain the term set corresponding to the initial input statement to be processed, wherein the term set includes the initial term included in the initial input statement and the extended term of the initial term;

[0127] The word order of the words in the word set is restored to obtain the expanded sentence of the initial input sentence;

[0128] Based on the expanded statement and the natural language processing model, the processing result is obtained.

[0129] According to one or more embodiments of this disclosure, the step of restoring the word order of the words in the word set to obtain the expanded sentence of the initial input sentence includes:

[0130] Traverse the terms in the term set, obtain the terms with connected position information from the term set, and construct an extended statement, wherein the position information is the position and / or offset value of the term in the initial input statement.

[0131] According to one or more embodiments of this disclosure, the step of traversing the terms in the term set, obtaining terms with connected positional information from the term set, and constructing an extended sentence includes:

[0132] The terms in the term set that have not been added with any extended statement are traversed sequentially, and it is determined whether the position information of the current term and the previous term that has been added with the current extended statement is connected.

[0133] If the positional information is contiguous, the current term is added to the current expanded statement, following the previous term added to the current expanded statement; or

[0134] If the location information is not connected, skip the current term;

[0135] At the end of the traversal, the construction of the current extended statement is completed, and the traversal of the terms in the term set that have not been added to any extended statement is restarted to construct new extended statements.

[0136] According to one or more embodiments of this disclosure, the method further includes:

[0137] An array is created based on the term set, wherein each bit of the array is used to store the state corresponding to each term in the term set, and the state corresponding to the term is used to characterize whether the term has been added to any extended statement;

[0138] During the traversal, if any word is added with any extended statement, the state corresponding to the word is updated in the array.

[0139] Accordingly, the step of sequentially traversing the terms in the term set that have not been supplemented with any extended statement includes:

[0140] Based on the array, the terms in the term set that have not been added with any extended statements are traversed sequentially.

[0141] According to one or more embodiments of this disclosure, determining whether the position information of the currently traversed term is connected to the position information of the previous term added to the current expanded statement includes:

[0142] Determine whether the current term being iterated over is connected to the position and / or offset of the previous term added to the current expansion statement.

[0143] According to one or more embodiments of this disclosure, determining whether the position and / or offset value of the currently traversed term is connected to the position and / or offset value of the previous term added to the current expanded statement includes:

[0144] Determine whether the starting offset value of the current term is connected to the ending offset value of the previous term added to the current expansion statement, and / or determine whether the starting offset value of the current term is the same as the starting offset value of the previous term added to the current expansion statement, and whether the position of the previous term added to the current expansion statement is in an increasing relationship with the position of the current term;

[0145] If the starting offset of the current term is connected to the ending offset of the previous term added to the current expansion statement, or if the starting offset of the current term is the same as the starting offset of the previous term added to the current expansion statement, and the position of the previous term added to the current expansion statement is in an increasing relationship with the position of the current term, then the position information of the current term is connected to that of the previous term added to the current expansion statement.

[0146] According to one or more embodiments of this disclosure, obtaining the processing result based on the extended statement and the natural language processing model includes:

[0147] The extended statement is input into the natural language processing model, which rewrites the extended statement to obtain the rewritten extended statement. Natural language processing is then performed on the rewritten extended statement to obtain the processing result.

[0148] According to one or more embodiments of this disclosure, obtaining the processing result based on the extended statement and the natural language processing model includes:

[0149] The extended statement is input into the natural language processing model, the natural language processing model obtains the sentence vector corresponding to the extended statement, and vector processing is performed on the sentence vector to obtain the processing result.

[0150] According to one or more embodiments of this disclosure, obtaining the processing result based on the extended statement and the natural language processing model includes:

[0151] The extended statement is queried using the natural language model to obtain query results and a matching score between the query results and the extended statement.

[0152] According to one or more embodiments of this disclosure, the expanded terms of the initial term include one or more of the following:

[0153] Synonyms of the initial term, pinyin of the initial term, and standard format terms corresponding to the initial term.

[0154] According to one or more embodiments of this disclosure, obtaining the term set corresponding to the initial input statement to be processed includes:

[0155] The analyzer of the search engine is invoked to obtain the set of terms corresponding to the initial input statement to be processed. The analyzer is a component in the search engine used to process text data. The analyzer includes one or more of the following analyzers: word segmenter, synonym analyzer, pinyin analyzer, and multilingual text analyzer.

[0156] Secondly, according to one or more embodiments of this disclosure, a natural language processing apparatus is provided, comprising:

[0157] The analysis unit is used to obtain the term set corresponding to the initial input statement to be processed, wherein the term set includes the initial term included in the initial input statement and the extended term of the initial term;

[0158] The word order restoration unit is used to restore the word order of the word items in the word set to obtain the expanded sentence of the initial input sentence;

[0159] The model processing unit is used to obtain the processing result based on the extended statement and the natural language processing model.

[0160] According to one or more embodiments of this disclosure, when the word order restoration unit performs word order restoration on the words in the word set to obtain the expanded sentence of the initial input sentence, it is used to:

[0161] Traverse the terms in the term set, obtain the terms with connected position information from the term set, and construct an extended statement, wherein the position information is the position and / or offset value of the term in the initial input statement.

[0162] According to one or more embodiments of this disclosure, when the word order restoration unit traverses the word items in the word item set, obtains word items with connected positional information from the word item set, and constructs an extended sentence, it is used to:

[0163] The terms in the term set that have not been added with any extended statement are traversed sequentially, and it is determined whether the position information of the current term and the previous term that has been added with the current extended statement is connected.

[0164] If the positional information is contiguous, the current term is added to the current expanded statement, following the previous term added to the current expanded statement; or

[0165] If the location information is not connected, skip the current term;

[0166] At the end of the traversal, the construction of the current extended statement is completed, and the traversal of the terms in the term set that have not been added to any extended statement is restarted to construct new extended statements.

[0167] According to one or more embodiments of this disclosure, the word order restoration unit is further configured to:

[0168] An array is created based on the term set, wherein each bit of the array is used to store the state corresponding to each term in the term set, and the state corresponding to the term is used to characterize whether the term has been added to any extended statement;

[0169] During the traversal, if any word is added with any extended statement, the state corresponding to the word is updated in the array.

[0170] Accordingly, when the word order restoration unit sequentially traverses the word items in the word set that have not been supplemented with any extended sentences, it is used to:

[0171] Based on the array, the terms in the term set that have not been added with any extended statements are traversed sequentially.

[0172] According to one or more embodiments of this disclosure, when determining whether the position information of the current traversed term is connected to the position information of the previous term added to the current expanded statement, the word order restoration unit is used to:

[0173] Determine whether the current term being iterated over is connected to the position and / or offset of the previous term added to the current expansion statement.

[0174] According to one or more embodiments of this disclosure, when determining whether the position and / or offset value of the currently traversed term is connected to the position and / or offset value of the previous term added to the current expanded statement, the word order restoration unit is used to:

[0175] Determine whether the starting offset value of the current term is connected to the ending offset value of the previous term added to the current expansion statement, and / or determine whether the starting offset value of the current term is the same as the starting offset value of the previous term added to the current expansion statement, and whether the position of the previous term added to the current expansion statement is in an increasing relationship with the position of the current term;

[0176] If the starting offset of the current term is connected to the ending offset of the previous term added to the current expansion statement, or if the starting offset of the current term is the same as the starting offset of the previous term added to the current expansion statement, and the position of the previous term added to the current expansion statement is in an increasing relationship with the position of the current term, then the position information of the current term is connected to that of the previous term added to the current expansion statement.

[0177] According to one or more embodiments of this disclosure, when the model processing unit obtains the processing result based on the extended statement and the natural language processing model, it is used to:

[0178] The extended statement is input into the natural language processing model, which rewrites the extended statement to obtain the rewritten extended statement. Natural language processing is then performed on the rewritten extended statement to obtain the processing result.

[0179] According to one or more embodiments of this disclosure, when the model processing unit obtains the processing result based on the extended statement and the natural language processing model, it is used to:

[0180] The extended statement is input into the natural language processing model, the natural language processing model obtains the sentence vector corresponding to the extended statement, and vector processing is performed on the sentence vector to obtain the processing result.

[0181] According to one or more embodiments of this disclosure, when the model processing unit obtains the processing result based on the extended statement and the natural language processing model, it is used to:

[0182] The extended statement is queried using the natural language model to obtain query results and a matching score between the query results and the extended statement.

[0183] According to one or more embodiments of this disclosure, the expanded terms of the initial term include one or more of the following:

[0184] Synonyms of the initial term, pinyin of the initial term, and standard format terms corresponding to the initial term.

[0185] According to one or more embodiments of this disclosure, when the analysis unit obtains the term set corresponding to the initial input statement to be processed, it is used to:

[0186] The analyzer of the search engine is invoked to obtain the set of terms corresponding to the initial input statement to be processed. The analyzer is a component in the search engine used to process text data. The analyzer includes one or more of the following analyzers: word segmenter, synonym analyzer, pinyin analyzer, and multilingual text analyzer.

[0187] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0188] The memory stores computer-executed instructions;

[0189] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the natural language processing method as described in the first aspect and various possible designs of the first aspect.

[0190] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the natural language processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0191] Fifthly, according to one or more embodiments of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the natural language processing method described in the first aspect and various possible designs of the first aspect.

[0192] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0193] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0194] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A natural language processing method, characterized in that, include: Obtain the term set corresponding to the initial input statement to be processed, wherein the term set includes the initial term included in the initial input statement and the extended term of the initial term; The word order of the words in the word set is restored to obtain the expanded sentence of the initial input sentence; Based on the expanded statement and the natural language processing model, the processing result is obtained.

2. The method according to claim 1, characterized in that, The step of restoring the word order of the words in the word set to obtain the expanded sentence of the initial input sentence includes: Traverse the terms in the term set, obtain the terms with connected position information from the term set, and construct an extended statement, wherein the position information is the position and / or offset value of the term in the initial input statement.

3. The method according to claim 2, characterized in that, The step of traversing the terms in the term set, obtaining the terms with connected positional information from the term set, and constructing the extended sentence includes: The terms in the term set that have not been added with any extended statement are traversed sequentially, and it is determined whether the position information of the current term and the previous term that has been added with the current extended statement is connected. If the positional information is contiguous, the current term is added to the current expanded statement, following the previous term added to the current expanded statement; or If the location information is not connected, skip the current term; At the end of the traversal, the construction of the current extended statement is completed, and the traversal of the terms in the term set that have not been added to any extended statement is restarted to construct new extended statements.

4. The method according to claim 3, characterized in that, The method further includes: An array is created based on the term set, wherein each bit of the array is used to store the state corresponding to each term in the term set, and the state corresponding to the term is used to characterize whether the term has been added to any extended statement; During the traversal, if any word is added with any extended statement, the state corresponding to the word is updated in the array. Accordingly, the step of sequentially traversing the terms in the term set that have not been supplemented with any extended statement includes: Based on the array, the terms in the term set that have not been added with any extended statements are traversed sequentially.

5. The method according to claim 3, characterized in that, The determination of whether the position information of the current term being traversed is connected to the position information of the previous term added to the current expanded statement includes: Determine whether the current term being iterated over is connected to the position and / or offset of the previous term added to the current expansion statement.

6. The method according to claim 5, characterized in that, The determination of whether the current term being traversed is connected to the position and / or offset value of the previous term added to the current expanded statement includes: Determine whether the starting offset value of the current term is connected to the ending offset value of the previous term added to the current expansion statement, and / or determine whether the starting offset value of the current term is the same as the starting offset value of the previous term added to the current expansion statement, and whether the position of the previous term added to the current expansion statement is in an increasing relationship with the position of the current term; If the starting offset of the current term is connected to the ending offset of the previous term added to the current expansion statement, or if the starting offset of the current term is the same as the starting offset of the previous term added to the current expansion statement, and the position of the previous term added to the current expansion statement is in an increasing relationship with the position of the current term, then the position information of the current term is connected to that of the previous term added to the current expansion statement.

7. The method according to claim 1, characterized in that, The process of obtaining the processing result based on the expanded statement and the natural language processing model includes: The extended statement is input into the natural language processing model, which rewrites the extended statement to obtain the rewritten extended statement. Natural language processing is then performed on the rewritten extended statement to obtain the processing result.

8. The method according to claim 1, characterized in that, The process of obtaining the processing result based on the expanded statement and the natural language processing model includes: The extended statement is input into the natural language processing model, the natural language processing model obtains the sentence vector corresponding to the extended statement, and vector processing is performed on the sentence vector to obtain the processing result.

9. The method according to claim 1, characterized in that, The process of obtaining the processing result based on the expanded statement and the natural language processing model includes: The extended statement is queried using the natural language model to obtain query results and a matching score between the query results and the extended statement.

10. The method according to any one of claims 1-9, characterized in that, The expanded terms of the initial term include one or more of the following: Synonyms of the initial term, pinyin of the initial term, and standard format terms corresponding to the initial term.

11. The method according to any one of claims 1-9, characterized in that, The process of obtaining the term set corresponding to the initial input statement to be processed includes: The analyzer of the search engine is invoked to obtain the set of terms corresponding to the initial input statement to be processed. The analyzer is a component in the search engine used to process text data. The analyzer includes one or more of the following analyzers: word segmenter, synonym analyzer, pinyin analyzer, and multilingual text analyzer.

12. A natural language processing device, characterized in that, include: The analysis unit is used to obtain the term set corresponding to the initial input statement to be processed, wherein the term set includes the initial term included in the initial input statement and the extended term of the initial term; The word order restoration unit is used to restore the word order of the word items in the word set to obtain the expanded sentence of the initial input sentence; The model processing unit is used to obtain the processing result based on the extended statement and the natural language processing model.

13. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-11.

Images