Chunk recognition method and apparatus

Abstract

This application discloses a method and apparatus for chunk recognition. The method includes: segmenting the text to be recognized into words to obtain segmented text; determining at least one target language sequence corresponding to the segmented text; the target language sequence consisting of multiple language units; inputting the target language sequence into a pre-trained semantic vector model for semantic vector recognition to obtain a semantic combination vector corresponding to each language unit in the target language sequence; determining the semantic contribution of each word group in the multiple language units relative to the text to be recognized based on the semantic combination vector; and determining the word group corresponding to the semantic contribution that meets the preset semantic contribution conditions as the target chunk corresponding to the text to be recognized based on the semantic contribution of each word group. This technical solution realizes the recognition of chunks in text from the semantic dimension, thereby improving the accuracy of text semantic understanding.

Description

Technical Field

[0001] This application relates to the field of language processing technology, and in particular to a method and apparatus for chunk recognition. Background Technology

[0002] Natural Language Processing (NLP) word segmentation is a fundamental capability, dividing text into segments at the word level. However, in some cases, word-level segmentation may not be necessary. Instead, readers often need to extract chunks of information from the text—the information contained within linguistic units composed of multiple words. Therefore, accurately identifying chunks of information from text has become a pressing issue. Summary of the Invention

[0003] The purpose of this application is to provide a method and apparatus for identifying language blocks in text.

[0004] To solve the above-mentioned technical problems, the embodiments of this application are implemented as follows:

[0005] On one hand, embodiments of this application provide a chunk recognition method, including:

[0006] The text to be identified is segmented into words to obtain the segmented text corresponding to the text to be identified; the segmented text is composed of multiple segmentation units.

[0007] Determine at least one target language sequence corresponding to the segmented text; the target language sequence consists of multiple language units, and the multiple language units include at least one word group obtained by combining multiple adjacent segmented units in the segmented text;

[0008] The target language sequence is input into a pre-trained semantic vector model for semantic vector recognition to obtain a semantic combination vector corresponding to each language unit in the target language sequence; the semantic combination vector is a combination of semantic vectors of multiple semantic elements corresponding to the language unit; the multiple semantic elements corresponding to the language unit are used to represent the semantic information of the language unit;

[0009] Based on the semantic feature combination vector, determine the semantic contribution of each word group in the plurality of language units relative to the text to be identified;

[0010] Based on the semantic contribution of each phrase, the phrases corresponding to the semantic contribution that meet the preset semantic contribution conditions are determined as the target word blocks corresponding to the text to be identified.

[0011] On the other hand, embodiments of this application provide a chunk recognition device, including:

[0012] The word segmentation module is used to perform word segmentation on the text to be identified, and obtain the word segmented text corresponding to the text to be identified; the word segmented text is composed of multiple word segmentation units.

[0013] The first determining module is used to determine at least one target language sequence corresponding to the segmented text; the target language sequence is composed of multiple language units, and the multiple language units include at least one word group obtained by combining multiple adjacent segmented units in the segmented text;

[0014] The semantic vector recognition module is used to input the target language sequence into a pre-trained semantic vector model for semantic vector recognition, and obtain the semantic combination vector corresponding to each language unit in the target language sequence; the semantic combination vector is a combination of the semantic vectors of multiple semantic elements corresponding to the language unit; the multiple semantic elements corresponding to the language unit are used to represent the semantic information of the language unit;

[0015] The second determining module is used to determine the semantic contribution of each word group in the plurality of language units relative to the text to be identified based on the semantic feature combination vector.

[0016] The third determining module is used to determine, based on the semantic contribution of each phrase, the phrase corresponding to the semantic contribution that meets the preset semantic contribution conditions as the target word block corresponding to the text to be identified.

[0017] In another aspect, embodiments of this application provide an electronic device, including a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being used to call and execute the computer program from the memory to implement the above-described chunk recognition method.

[0018] In another aspect, embodiments of this application provide a computer-readable storage medium for storing a computer program that can be executed by a processor to implement the above-described chunk recognition method.

[0019] The technical solution of this application involves segmenting the text to be identified into words to obtain segmented text. At least one target language sequence corresponding to the segmented text is determined. This target language sequence consists of multiple language units, including at least one phrase formed by combining multiple adjacent segmented units in the segmented text. Since the target language sequence includes phrases formed by combining multiple adjacent segmented units in the segmented text, and a phrase is a phrase with semantic information, determining the target language sequence corresponding to the text to be identified provides a strong textual foundation for phrase recognition. Furthermore, the target language sequence is input into a pre-trained semantic vector model for semantic vector recognition, obtaining a semantic combination vector corresponding to each language unit in the target language sequence, i.e., a combination of semantic vectors of multiple semantic elements. Based on the semantic combination vector, the semantic contribution of each phrase in the multiple language units relative to the text to be identified is determined, and thus the phrase corresponding to the semantic contribution that meets the preset semantic contribution conditions is determined as the target phrase corresponding to the text to be identified. Since a semantic feature combination vector is a combination of multiple semantic feature vectors, and semantic features, as the smallest granularity in natural language processing, can more accurately represent the semantic information of each language unit, by identifying the semantic feature combination vector corresponding to each language unit in the target language sequence, and determining the target word chunk in the text to be identified based on the semantic feature combination vector, the effect of identifying word chunks in the text from the semantic feature dimension is achieved. This not only accurately identifies word chunks in the text, but also the identified word chunks have rich semantic information, which is conducive to the machine to better understand and analyze the semantics of the text, and avoids the situation where inaccurate word chunk identification in the text leads to errors in the understanding of the entire text. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in one or more embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic flowchart of a chunk recognition method according to an embodiment of this application;

[0022] Figure 2 This is a schematic flowchart illustrating a training method for a semantic feature annotation model according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram illustrating a training method for a semantic feature annotation model according to an embodiment of this application.

[0024] Figure 4 This is a schematic flowchart of a chunk recognition method according to another embodiment of this application;

[0025] Figure 5 This is a schematic block diagram of a language block recognition device according to an embodiment of this application;

[0026] Figure 6 This is a schematic block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] This application provides a method and apparatus for identifying language blocks in text.

[0028] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0029] NLP word segmentation is a fundamental capability, dividing text at the word level. However, in some cases, this word level may not be necessary. Instead, readers often need to extract chunks of information from the text—the information units composed of multiple words. Simply recognizing individual words only allows for understanding at the word level, failing to grasp the meaning of these chunks, leading to inaccurate semantic understanding and ultimately, a biased understanding of the entire text. In an exemplary scenario, such as in the financial sector, "business transaction history" is a chunk of words, a specialized term with specific meaning in the financial field. Analyzing only at the word level only reveals the individual semantics of "business" and "transaction history." However, accurately identifying the chunk "business transaction history" allows for the analysis of its semantic meaning at the chunk level (e.g., business details), thus leading to a more accurate understanding of the entire text, including the chunk "business transaction history." Therefore, accurately identifying chunks of information in text is a crucial aspect of semantic understanding.

[0030] This application determines the target language sequence corresponding to the text to be identified. This target language sequence includes word groups obtained by combining multiple adjacent word segmentation units in the segmented text, thus providing a strong textual foundation for the identification of language chunks in the text to be identified. The target language sequence is then input into a pre-trained semantic vector model for semantic vector identification, resulting in a semantic combination vector corresponding to each language unit in the target language sequence, i.e., a combination of semantic vectors of multiple semantic elements. Based on the semantic combination vector, the semantic contribution of each word group in the multiple language units relative to the text to be identified is determined, and then the word group corresponding to the semantic contribution that meets the preset semantic contribution conditions is determined as the target language chunk corresponding to the text to be identified. Since a semantic feature combination vector is a combination of multiple semantic feature vectors, and semantic features, as the smallest granularity in natural language processing, can more accurately represent the semantic information of each language unit, this method identifies the semantic feature combination vector corresponding to each language unit in the target language sequence and determines the target chunk in the text based on the semantic feature combination vector. This achieves the effect of identifying chunks in the text from the semantic feature dimension. It not only accurately identifies chunks in the text but also ensures that the identified chunks possess rich semantic information, thereby facilitating better semantic understanding and analysis of the text by the machine and avoiding errors in the overall text understanding caused by inaccurate chunk identification. The chunk identification method provided in this application is described in detail below.

[0031] The language block recognition method provided in this application can be executed by an electronic device or by software installed in an electronic device. Specifically, the electronic device can be a terminal device or a server device. The terminal device can include smartphones, laptops, smart wearable devices, vehicle terminals, etc., and the server device can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0032] Figure 1 This is a schematic flowchart of a chunk recognition method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0033] S102, perform word segmentation on the text to be identified to obtain the word segmented text corresponding to the text to be identified. The word segmented text is composed of multiple word segmentation units.

[0034] In this step, existing word segmentation techniques can be used to segment the text to be recognized. This embodiment does not limit the specific word segmentation technique used, such as dictionary-based word segmentation algorithms, semantic analysis-based word segmentation algorithms, etc. After the text to be recognized is segmented, the resulting segmented text consists of multiple segmentation units, each segmentation unit corresponding to a word.

[0035] S104, determine at least one target language sequence corresponding to the segmented text, the target language sequence is composed of multiple language units, the multiple language units include at least one word group obtained by combining multiple adjacent segmented units in the segmented text.

[0036] The target language sequence corresponding to the segmented text is the same as the target language sequence corresponding to the text to be identified. The difference between the target language sequence and the segmented text is that the segmented text only includes words, while the target language sequence includes words and at least one phrase.

[0037] For example, the segmented text is "early repayment refers to making repayment 5 days before the repayment date," where spaces are used to separate different segmented units. The corresponding target language sequence could include "early repayment refers to making repayment 5 days before the repayment date." It can be seen that the difference between the target language sequence and the segmented text is that in the segmented text, "early" and "repayment" are separate words, while in the target language sequence, "early repayment" is a phrase.

[0038] S106, Input the target language sequence into the pre-trained semantic vector model to perform semantic vector recognition, and obtain the semantic combination vector corresponding to each language unit in the target language sequence. The semantic combination vector is the combination of the semantic vectors of multiple semantic elements corresponding to the language unit. The multiple semantic elements corresponding to the language unit are used to represent the semantic information of the language unit.

[0039] Each language unit corresponds to multiple semantic features, which clearly and completely represent the semantic information of the language unit. For example, the semantic features included in the language unit "Man" are: +[human]+[male]+[adult], and the semantic features corresponding to the language unit "Woman" are: +[human]-[male]+[adult]. The symbol "+" before a semantic feature indicates that the language unit contains that semantic feature, meaning it possesses the semantic information represented by that feature; the symbol "-" indicates that the language unit does not contain that semantic feature, meaning it does not possess the semantic information represented by that feature.

[0040] The input data of the semantic feature vector model is a language sequence, and the output data is a semantic feature combination vector corresponding to each language unit in the language sequence. The semantic feature vector model includes the mapping relationship between multiple semantic features and semantic feature vectors. The multiple semantic features in the semantic feature vector model can include first-class semantic features extracted from a pre-set dictionary (also known as explicit semantic features), or second-class semantic features extracted after analyzing multiple corpora in a pre-set corpus (also known as implicit semantic features). How to construct the mapping relationship between multiple semantic features and semantic feature vectors, and how to use the semantic feature vector model for semantic feature vector recognition, will be described in detail in the following embodiments, and will not be elaborated here.

[0041] S108, based on the semantic feature combination vector, determine the semantic contribution of each word group in multiple language units relative to the text to be identified.

[0042] S110, based on the semantic contribution of each phrase, determine the phrases corresponding to the semantic contribution that meet the preset semantic contribution conditions as the target text blocks corresponding to the text to be identified.

[0043] The preset semantic contribution conditions may include at least one of the following conditions a1 and a2:

[0044] Condition a1: The semantic contribution is greater than or equal to a preset threshold. For example, if the preset threshold is 0.8, after determining the semantic contribution of each word group in multiple language units, the word groups with semantic contributions greater than or equal to 0.8 are selected, and the word groups corresponding to the selected semantic contribution are the target word groups.

[0045] Condition a2: The semantic contribution is among the top N semantic contributions, where N is an integer greater than or equal to 1. All semantic contributions refer to the semantic contribution of each word group in the target language sequence relative to the text to be identified. For example, if N = 2, after determining the semantic contribution of each word group in multiple language units, the semantic contributions of each word group are sorted from largest to smallest. The top two semantic contributions are then selected based on this sorting, and the word groups corresponding to these selected semantic contributions are the target language chunks.

[0046] In this embodiment, the text to be identified can be a single sentence or an article composed of multiple sentences. If the text to be identified is an article composed of multiple sentences, it can be split into multiple sentences, and then the technical solution of steps S102-S110 described above can be executed for each sentence to identify the target language block in each sentence.

[0047] The technical solution of this application involves segmenting the text to be identified into words to obtain segmented text. At least one target language sequence corresponding to the segmented text is determined. This target language sequence consists of multiple language units, including at least one phrase formed by combining multiple adjacent segmented units in the segmented text. Since the target language sequence includes phrases formed by combining multiple adjacent segmented units in the segmented text, and a phrase is a phrase with semantic information, determining the target language sequence corresponding to the text to be identified provides a strong textual foundation for phrase recognition. Furthermore, the target language sequence is input into a pre-trained semantic vector model for semantic vector recognition, obtaining a semantic combination vector corresponding to each language unit in the target language sequence, i.e., a combination of semantic vectors of multiple semantic elements. Based on the semantic combination vector, the semantic contribution of each phrase in the multiple language units relative to the text to be identified is determined, and thus the phrase corresponding to the semantic contribution that meets the preset semantic contribution conditions is determined as the target phrase corresponding to the text to be identified. Since a semantic feature combination vector is a combination of multiple semantic feature vectors, and semantic features, as the smallest granularity in natural language processing, can more accurately represent the semantic information of each language unit, by identifying the semantic feature combination vector corresponding to each language unit in the target language sequence, and determining the target word chunk in the text to be identified based on the semantic feature combination vector, the effect of identifying word chunks in the text from the semantic feature dimension is achieved. This not only accurately identifies word chunks in the text, but also the identified word chunks have rich semantic information, which is conducive to the machine to better understand and analyze the semantics of the text, and avoids the situation where inaccurate word chunk identification in the text leads to errors in the understanding of the entire text.

[0048] In one embodiment, the target language sequence conforms to preset grammatical rules. These preset grammatical rules can be understood as grammatical rules that conform to normal human language habits. For example, each language unit in the target language sequence has a unique part of speech, adjacent words or phrases have different parts of speech, and so on. Therefore, at least one target language sequence corresponding to the segmented text can be determined through grammatical discrimination. Optionally, at least one target language sequence corresponding to the segmented text can be determined through the following steps A1-A3:

[0049] Step A1: Determine multiple word segmentation unit groups to be combined in the segmented text. Each word segmentation unit group includes multiple adjacent word segmentation units.

[0050] Step A2 involves combining multiple word segments in each word segmentation unit group, and then integrating the resulting word groups with other uncombined word segments in the word segmentation text to obtain multiple candidate language sequences corresponding to the word segmentation text.

[0051] Optionally, a combination rule for word segmentation units can be pre-defined. This rule may include at least one of the following: the number of word segmentation units in each group and the number of combinations corresponding to the same candidate language sequence. For example, the combination rule could be: each group contains 2 word segmentation units, and the number of combinations corresponding to the same candidate language sequence is 1. Then, when determining multiple groups of word segmentation units to be combined in the segmented text, each pair of adjacent word segmentation units in the segmented text can be defined as a separate group. Furthermore, since the number of combinations corresponding to the same candidate language sequence is 1, for the same segmented text, if the two combined word segmentation units are different, different candidate language sequences will be obtained. That is, different groups of word segmentation units correspond to different candidate language sequences. Thus, by combining each pair of adjacent word segmentation units in the segmented text, multiple candidate language sequences can be obtained.

[0052] For example, the segmented text is: "Early repayment refers to making repayment 5 days before the due date".

[0053] By combining every two adjacent word segments in the above segmented text, the following candidate language sequences can be obtained:

[0054] Combining "early repayment" and "repayment" yields the candidate language sequence "early repayment refers to making repayment 5 days before the due date".

[0055] Combining "refers to" and "at" yields the candidate language sequence "early repayment refers to making repayment 5 days before the repayment date".

[0056] Of course, the above rules for combination are merely examples. In practical applications, to ensure the identification of more comprehensive segments in the text to be recognized, the number of segmented units to be combined can be expanded, rather than being limited to just two or three. For example, for the same segmented text, after combining according to the rule of "each segmented unit group includes two segmented units," it can be further combined according to the rule of "each segmented unit group includes three segmented units," and then further combined according to the rule of "each segmented unit group includes four segmented units," and so on, until all potentially combined adjacent segmented units in the segmented text have been combined. Through this combination method, all possible candidate language sequences corresponding to the segmented text can be extracted as much as possible.

[0057] Step A3: Select at least one candidate language sequence that conforms to the preset grammatical rules from multiple candidate language sequences as the target language sequence. The multiple language units in the target language sequence include combined word groups and other uncombined word units.

[0058] The parts of speech (PPS) can include nouns, verbs, adjectives, numerals, classifiers, adverbs, pronouns, prepositions, conjunctions, auxiliary words, and modal particles. Existing PPS recognition algorithms can be used to identify the PPS of each language unit. Optionally, the candidate language sequence is input into a grammar recognizer to identify the PPS of each language unit and perform PPS tagging, thereby outputting a sequence of candidate PPS identifiers corresponding to the candidate language sequence. The grammar recognizer pre-deploys any existing PPS recognition tool to identify the PPS of each language unit.

[0059] When selecting at least one candidate language sequence that conforms to preset grammatical rules, the following steps can be performed for each candidate language sequence: Part-of-speech tagging (POS) is performed on each language unit in the candidate language sequence to obtain a candidate POS tag sequence corresponding to the candidate language sequence. If the candidate POS tag sequence conforms to the preset grammatical rules, then the candidate language sequence corresponding to the candidate POS tag sequence is determined to be the target language sequence. Otherwise, if the candidate POS tag sequence does not conform to the preset grammatical rules, then the candidate language sequence corresponding to the candidate POS tag sequence is determined not to be the target language sequence.

[0060] Taking the example that each language unit has a unique part of speech according to the preset grammatical rules, the segmented text is "early repayment refers to making repayment 5 days before the repayment date".

[0061] For a candidate language sequence "early repayment refers to repayment made 5 days before the repayment date" corresponding to the segmented text, part-of-speech tagging (POS) yields the candidate POS tag sequence "vv prep n loc quantv v", which conforms to the preset grammatical rules. Here, v represents a verb, prep represents a preposition, n represents a noun, loc represents a preposition, and quant represents a quantifier.

[0062] For the other candidate language sequence corresponding to the segmented text, "early repayment refers to repayment made 5 days before the repayment date," part-of-speech tagging (POS) results in a candidate POS tag sequence of "null." The reason for the candidate POS tag sequence being "null" is that the part of speech for "repayment refers to" in the candidate language sequence cannot be identified, meaning it does not have a unique part of speech. Therefore, this candidate POS tag sequence does not conform to the preset grammatical rules.

[0063] In one embodiment, after filtering out candidate part-of-speech tag sequences that conform to preset grammatical rules, the candidate language sequences corresponding to these sequences are determined as the target language sequences. Optionally, a part-of-speech tagging algorithm is used to perform part-of-speech tagging on the candidate language sequences. The output of the algorithm includes candidate part-of-speech tag sequences that conform to preset grammatical rules and their corresponding candidate language sequences. Based on the output, the candidate language sequence corresponding to each candidate part-of-speech tag sequence that conforms to the preset grammatical rules can be directly determined. If the output of the algorithm only includes candidate part-of-speech tag sequences that conform to preset grammatical rules, the candidate language sequences corresponding to these sequences can be determined through steps B1-B4.

[0064] Step B1: If the candidate part-of-speech tag sequence conforms to the preset grammar rules, then align the candidate part-of-speech tag sequence with the initial part-of-speech tag sequence corresponding to the segmented text according to the preset alignment rules.

[0065] The preset alignment rules include: the number of aligned part-of-speech tags in the candidate part-of-speech tag sequence and the initial part-of-speech tag sequence is the largest. The initial part-of-speech tag sequence corresponding to the segmented text is obtained by performing part-of-speech tagging on each segmented unit in the segmented text. Optionally, the segmented text is input into a syntax recognizer to identify the part of speech of each segmented unit and perform part-of-speech tagging, thereby outputting the initial part-of-speech tag sequence corresponding to the segmented text. The syntax recognizer pre-deploys any existing part-of-speech tagging tool to identify the part of speech of each segmented unit.

[0066] Step B2: Based on the alignment results, determine the multiple first part-of-speech tags that are combined in the initial part-of-speech tag sequence.

[0067] In this process, multiple first part-of-speech tags in the initial part-of-speech tag sequence are combined, and together with other part-of-speech tags that are not combined in the initial part-of-speech tag sequence, they constitute the candidate part-of-speech tag sequence.

[0068] Step B3: Determine the first segmentation unit group in the segmented text based on multiple first part-of-speech identifiers.

[0069] The first segmentation unit group is one of multiple segmentation unit groups corresponding to the segmented text. Each segmentation unit group includes multiple adjacent segmentation units.

[0070] Step B4: Based on the first word segmentation unit group, determine the candidate language sequence corresponding to the candidate part-of-speech tag sequence.

[0071] For example, the initial part-of-speech tag sequence corresponding to the segmented text is "adv vv prep n loc quant vv". A candidate part-of-speech tag sequence corresponding to the segmented text is "vv prep n loc quant vv". Following the preset alignment rules, the following alignment result can be obtained:

[0072]

[0073] As can be seen from the alignment results above, only the first part-of-speech tag in the aligned candidate part-of-speech tag sequence and the initial part-of-speech tag sequence is not aligned. This indicates that "adv v" in the initial part-of-speech tag sequence corresponding to the segmented text has been combined into "v". Therefore, it can be determined that "adv v" in the initial part-of-speech tag sequence is a combination of multiple first part-of-speech tags. Based on the multiple combined first part-of-speech tags "adv v" and the segmented text "early repayment refers to repayment made 5 days before the repayment date", it can be determined that the first segmentation unit group is "early repayment". The segmentation units "early" and "repayment" in the first segmentation unit group are combined to obtain "early repayment". Therefore, the candidate language sequence that includes the combined "early repayment" among the multiple candidate language sequences is the candidate language sequence corresponding to the candidate part-of-speech tag sequence.

[0074] In one embodiment, when determining the semantic feature combination vector corresponding to each language unit in a target language sequence, the target language sequence can be input into a pre-trained semantic feature vector model. The semantic feature vector model then identifies and analyzes the semantic feature vectors of each language unit. The semantic feature vector model includes the mapping relationship between multiple semantic features and semantic feature vectors. Since language units include words and phrases, the semantic feature vector model can employ different methods to identify and analyze the semantic feature vectors of different language units.

[0075] Optionally, for each language unit in the target language sequence, if the language unit is a phrase, the phrase is split into multiple words. Then, for each word obtained after splitting, the semantic vector of each semantic feature corresponding to the word is determined according to the mapping relationship in the semantic feature vector model. Then, the first semantic feature combination vector corresponding to the word is determined according to the semantic feature vector of each semantic feature corresponding to the word. Furthermore, the first semantic feature combination vectors corresponding to each word in the phrase are concatenated to obtain the second semantic feature combination vector corresponding to the phrase. If the language unit is a word, the first semantic feature combination vector corresponding to the word is determined according to the mapping relationship in the semantic feature vector model. The first semantic feature combination vector corresponding to the word is the combination of the semantic feature vectors corresponding to each semantic feature included in the word.

[0076] When determining the first semantic feature combination vector corresponding to a word, we can first determine the semantic features included in the word, then compare the mapping relationship between the semantic features included in the word and the semantic feature vector model to determine the semantic feature vector corresponding to each semantic feature included in the word, and then concatenate the semantic feature vectors corresponding to each semantic feature included in the word to obtain the first semantic feature combination vector corresponding to the word.

[0077] For example, the semantic feature vector model includes the following semantic features: [human], [male], and [adult]. The brackets "[]" are only used to distinguish different semantic features and have no actual meaning. Language units include words such as "Man" and "Woman." The semantic features included in the word "Man" are: +[human] + [male] + [adult]; the semantic features included in the word "Woman" are: +[human] - [male] + [adult]. The semantic symbols preceding the semantic features are used to indicate whether the word possesses the semantic information corresponding to the semantic feature. A plus sign "+" indicates that the word possesses the semantic information corresponding to the semantic feature, and a minus sign "-" indicates that the word does not possess the semantic information corresponding to the semantic feature. When determining the first semantic feature combination vector corresponding to the word "Man," each semantic feature [human], [male], and [adult] included in the word "Man" is compared with the mapping relationship to determine the semantic feature vectors corresponding to the semantic features [human], [male], and [adult] respectively. Then, combining the semantic symbols preceding each semantic element [human], [male], and [adult], the semantic element vectors corresponding to [human], [male], and [adult] are concatenated to obtain the first semantic element combination vector for the word "Man". An optional concatenation method is to concatenate the semantic element vectors corresponding to the semantic elements marked with a plus sign "+" among the semantic elements corresponding to the word to obtain the first semantic element combination vector for the word.

[0078] In this embodiment, since the semantic vector model is used to maintain the mapping relationship between multiple semantic elements and semantic vectors, and the smallest granularity of words and phrases is the semantic element, the semantic vector model can split words or phrases into the smallest granularity from the semantic element dimension, and then determine the semantic element combination vector corresponding to each language unit from the smallest granularity of the semantic element, so that the semantic element combination vector can cover the semantic information of all semantic elements in the language unit, thereby more accurately representing the semantics of the language unit.

[0079] In one embodiment, the preset semantic contribution condition includes at least one of the following conditions a1 and a2:

[0080] Condition a1: The semantic contribution is greater than or equal to a preset threshold. For example, if the preset threshold is 0.8, after determining the semantic contribution of each word group in multiple language units, the word groups with semantic contributions greater than or equal to 0.8 are selected, and the word groups corresponding to the selected semantic contribution are the target word groups.

[0081] Condition a2: The semantic contribution is among the top N semantic contributions, where N is an integer greater than or equal to 1. All semantic contributions refer to the semantic contribution of each word group in the target language sequence relative to the text to be identified. For example, if N = 2, after determining the semantic contribution of each word group in multiple language units, the semantic contributions of each word group are sorted from largest to smallest. The top two semantic contributions are then selected based on this sorting, and the word groups corresponding to these selected semantic contributions are the target language chunks.

[0082] When determining the semantic contribution of each word group in multiple language units relative to the text to be identified based on the semantic feature combination vector, there are multiple ways to calculate the semantic contribution of each word group. Two of them are listed below as examples.

[0083] Optionally, for each phrase in multiple language units, firstly, the first proportion of the semantic feature combination vector corresponding to the phrase in the total semantic feature combination vector corresponding to the target language sequence is calculated; secondly, based on the first proportion, the semantic contribution of the phrase is determined. Here, the total semantic feature combination vector is the sum of the semantic feature combination vectors corresponding to each language unit in the target language sequence. If the first proportion is taken as the semantic contribution of the phrase, then the semantic contribution of the phrase relative to the text to be recognized can be expressed as follows:

[0084] Semantic contribution = (Semantic feature combination vector corresponding to the phrase / Total semantic feature combination vector)

[0085] The total semantic feature combination vector is equal to the sum of the semantic feature combination vectors corresponding to each language unit in the target language sequence.

[0086] Optionally, the semantic contribution can also be calculated by combining the semantic feature weights corresponding to each word group. Specifically, for each word group in multiple language units, firstly, based on the first semantic feature weights corresponding to the word group, the first weighted value of the semantic feature combination vector corresponding to the word group is determined; and secondly, based on the first semantic feature weights corresponding to each language unit in the target language sequence, the second weighted value of the total semantic feature combination vector is determined. Secondly, the second proportion of the first weighted value in the second weighted value is calculated. Thirdly, based on the second proportion, the semantic contribution of the word group is determined. Here, the total semantic feature combination vector is the sum of the semantic feature combination vectors corresponding to each language unit in the target language sequence. If the second proportion is used as the semantic contribution of the word group, the semantic contribution of the word group relative to the text to be identified can be expressed as follows:

[0087]

[0088] Where Y represents the semantic contribution of phrase [A] in the target language sequence, A represents the semantic feature combination vector corresponding to phrase [A], and a represents the weight of the first semantic feature corresponding to phrase [A]. B and C are the semantic feature combination vectors corresponding to language units [B] and [C] in the target language sequence, respectively, and b and c are the weights of the first semantic features corresponding to language units [B] and [C], respectively. The second weighted value of the total semantic feature combination vector is the sum of the weighted values ​​of all language units in the target language sequence. The weighted value of each language unit is equal to the product of the semantic feature combination vector corresponding to the language unit and its corresponding weight of the first semantic feature. It should be noted that language units [B] and [C] can be phrases or words.

[0089] Since a chunk is an informational unit of language composed of multiple words, meaning that a chunk itself belongs to a phrase, we can calculate only the semantic contribution of the phrase in the target language sequence, while the semantic contribution of the words themselves does not need to be calculated, thus saving unnecessary computational work.

[0090] In this embodiment, by determining the word groups in the target language sequence that meet the preset semantic contribution conditions as the target language blocks corresponding to the text to be identified, since the word groups that meet the preset semantic contribution conditions have high semantic importance in the target language sequence, the identified target language blocks can have rich semantic information, which is conducive to the machine to better understand and analyze the semantics of the text, and avoids the situation where the inaccurate identification of language blocks in the text leads to the error in the understanding of the entire text.

[0091] In one embodiment, the primary semantic feature weights corresponding to each language unit can be preset by a user (such as a language expert or a user with a high level of language understanding). The primary semantic feature weights are related to the importance of the semantic information of the language unit; for example, a higher primary semantic feature weight indicates a higher level of importance of the semantic information of the language unit. The primary semantic feature weights can be values ​​within the range of (0, 1). For example, for language units with high semantic information importance, a primary semantic feature weight close to 1 can be set; for language units with low semantic information importance, a primary semantic feature weight close to 0 can be set.

[0092] In one embodiment, the weight of the first semantic feature corresponding to each language unit can be determined according to the following steps C1-C2:

[0093] Step C1: For each language unit in the target language sequence, determine the weight of the second semantic feature corresponding to each semantic feature included in the language unit according to a preset spatial dimension table. The spatial dimension table includes the mapping relationship between semantic features, semantic feature weights, and the spatial dimension information corresponding to the semantic features. The spatial dimension information includes the distribution information of semantic features in at least one of the grammatical, categorical, and connotative dimensions.

[0094] The distribution information in the grammatical dimension refers to the grammatical category of the semantic element, which may include nouns, verbs, adjectives, numerals, classifiers, adverbs, pronouns, prepositions, conjunctions, auxiliary words, or modal particles. The distribution information in the category dimension refers to the class to which the semantic element belongs; for example, the semantic element "cow" belongs to the animal category. The distribution information in the connotation dimension refers to the characteristic description information of the semantic element; for example, the distribution information of the semantic element "dog" in the connotation dimension includes: the morphological characteristics of a dog, its fur characteristics, etc. The distribution information in each dimension can be set by the user (such as a language expert or a user with a high level of language understanding).

[0095] The semantic features in the spatial dimension table can be obtained from the semantic feature vector model. Optionally, the semantic features in the spatial dimension table are the same as those in the semantic feature vector model. The semantic feature weight of each semantic feature in the spatial dimension table is related to the importance of the semantic information corresponding to the semantic feature. Optionally, the higher the importance of the semantic information corresponding to the semantic feature, the higher the semantic feature weight can be set for that semantic feature; conversely, the lower the importance of the semantic information corresponding to the semantic feature, the lower the semantic feature weight can be set for that semantic feature. In the spatial dimension table, each semantic feature has a unique semantic feature weight.

[0096] In the spatial dimension table, the spatial dimension information corresponding to a semantic feature includes distribution information across multiple dimensions. In this case, the semantic feature weight can be determined based on the dimension weights corresponding to each dimension and the sub-semantic feature weights corresponding to the distribution information of the semantic feature across each dimension. The dimension weights corresponding to each dimension are preset. The sub-semantic feature weights corresponding to the distribution information of the semantic feature across each dimension are related to the importance of the semantic information corresponding to the distribution information of the semantic feature across each dimension. Optionally, the higher the importance of the semantic information corresponding to the distribution information of the semantic feature across a dimension, the higher the sub-semantic feature weight can be set for the semantic feature in that dimension; conversely, the lower the importance of the semantic information corresponding to the distribution information of the semantic feature across a dimension, the lower the sub-semantic feature weight can be set for the semantic feature in that dimension.

[0097] Optionally, the semantic feature weights are the weighted sum of the sub-semantic feature weights corresponding to the distribution information of the semantic feature in each dimension and the dimension weights corresponding to each dimension. If the dimensions include syntactic, categorical, and connotative dimensions, the semantic feature weights can be calculated using the following formula:

[0098] P = P 语法 *L语法 +P 范畴 *L 范畴 +P 内涵 *L 内涵

[0099] Where P is the semantic weight corresponding to the semantic element, P 语法 P 范畴 and P 内涵 L represents the sub-semantic weights corresponding to the distribution information of semantic features in the grammatical, categorical, and connotative dimensions, respectively. 语法 L 范畴 and L 内涵 These represent the dimensional weights corresponding to the grammatical dimension, categorical dimension, and intensional dimension, respectively.

[0100] Step C2: Determine the weight of the first semantic element corresponding to the language unit based on the weight of the second semantic element corresponding to each semantic element included in the language unit.

[0101] Optionally, the weight of the second semantic feature corresponding to a language unit is equal to the sum of the weights of the first semantic features corresponding to each semantic feature included in the language unit.

[0102] In this embodiment, the spatial dimension information can be in vector form, i.e., a spatial dimension vector. If the spatial dimension information corresponding to a semantic feature includes distribution information across multiple dimensions, then the spatial dimension vector corresponding to the semantic feature can be the sum of distribution vectors across multiple dimensions. For example, if the spatial dimension information corresponding to a semantic feature includes distribution information on the syntactic dimension and distribution information on the connotative dimension, then the spatial dimension vector corresponding to the semantic feature is the sum of the distribution vectors on the syntactic dimension and the distribution vectors on the connotative dimension. Alternatively, the spatial dimension vector corresponding to a semantic feature can also be a weighted sum of distribution vectors on each dimension, with each dimension corresponding to its respective dimension weight. If the dimensions include syntactic dimension, category dimension, and connotative dimension, then the spatial dimension vector corresponding to the semantic feature can be determined using the following formula:

[0103] T = T 语法 *L 语法 +T 范畴 *L 范畴 +T 内涵 *L 内涵

[0104] Where T is the spatial dimension vector corresponding to the semantic element, T 语法 T 范畴 and T 内涵 These represent the distribution vectors of semantic features in the grammatical, categorical, and connotative dimensions, respectively. 语法 L 范畴 and L 内涵 These represent the dimensional weights corresponding to the grammatical dimension, categorical dimension, and intensional dimension, respectively.

[0105] The distribution vector of semantic features in a certain dimension is the information vector corresponding to the distribution information of semantic features in that dimension. After determining the distribution information of semantic features in a dimension, the information vector corresponding to the distribution information can be determined through the conversion relationship between language and language vectors. The conversion relationship between language and language vectors is existing technology and will not be elaborated here.

[0106] In this embodiment, the semantic weights are determined by the spatial dimension table corresponding to the semantic features, and then the semantic weights corresponding to the language units are determined. Since the spatial dimension table includes the mapping relationship between semantic features, semantic weights and spatial dimension information corresponding to the semantic features, and the semantic weights corresponding to the semantic features can characterize the importance of the semantic information corresponding to the semantic features to a certain extent, the semantic weights corresponding to the language units can characterize the importance of the semantic information corresponding to the language units to a certain extent, which is beneficial for filtering the language blocks in the text to be identified from the semantic feature level.

[0107] In one embodiment, the semantic vector model can be constructed through the following steps D1-D4:

[0108] Step D1: Extract the first type of semantic element from the preset dictionary.

[0109] The preset dictionary can be any existing dictionary or one or more. It includes multiple words and phrases, along with semantic definitions for each word and phrase. By analyzing the semantic definitions in the preset dictionary, the first type of semantic features, i.e., explicit semantic features, can be extracted.

[0110] Step D2 involves analyzing multiple corpora in a pre-defined corpus to obtain the second type of semantic elements corresponding to each corpus.

[0111] Among them, since the second type of semantic features can only be obtained by analyzing multiple corpora, that is, they cannot be directly extracted from the pre-set dictionary, the second type of semantic features are also called latent semantic features.

[0112] Step D3: Determine the semantic vectors corresponding to the first and second semantic elements, respectively.

[0113] Specifically, the semantic vector corresponding to a semantic feature can be determined through the conversion relationship between language and language vectors. The conversion relationship between language and language vectors is existing technology and will not be elaborated here.

[0114] Step D4: Establish the mapping relationship between the first type of semantic feature and its corresponding semantic feature vector, and the mapping relationship between the second type of semantic feature and its corresponding semantic feature vector, to obtain the semantic feature vector model.

[0115] In this embodiment, the preset corpus includes sentence corpus and word corpus. When performing step D2, for the sentence corpus and the word corpus, corresponding methods can be used to extract the second type of sememes.

[0116] For the sentence corpus in the preset corpus, the second type of sememes in the sentence corpus can be analyzed through the following method:

[0117] First, determine the word to be replaced in the sentence corpus. The word to be replaced can be selected by the user or automatically by the computer. When the computer automatically selects the word to be replaced, it can select a word with semantically related words in the sentence corpus as the word to be replaced.

[0118] Second, use the first semantically related word corresponding to the word to be replaced to replace the word to be replaced, and obtain the comparison sentence corresponding to the sentence corpus.

[0119] [[ID=1e12]]Among them, the first semantically related word corresponding to the word to be replaced may include the synonym and / or antonym corresponding to the word to be replaced. For example, for the word "river" in the sentence corpus, its first semantically related word can be "lake". Therefore, "river" can be used as a word to be replaced, and "lake" can be used as the first semantically related word of the word to be replaced "river". Assuming that each sentence corpus only includes one word to be replaced, after replacing the word to be replaced in the sentence corpus, only one word in the sentence corpus and the comparison sentence is different, that is, the word to be replaced and the first semantically related word.

[0120] Third, input the sentence corpus group into the trained sememe annotation model for sememe annotation, and obtain the second type of sememes corresponding to the word to be replaced and the first semantically related word respectively. Among them, the sentence corpus group includes the sentence corpus and the comparison sentence. The sememe annotation model is used to perform sememe annotation on the sentence corpus and the comparison sentence. The essence of sememe annotation is: by comparing the sentence corpus and the corresponding comparison sentence, determine the different words in the sentence corpus and the comparison sentence, and then analyze the semantic information of the different words. This semantic information includes the same semantic information and different semantic information between the different words. Furthermore, annotate the semantic information corresponding to the different words, and the annotated semantic information is the first type of sememe.

[0121] [[ID=1ec18]]For example, the different words in the sentence corpus and the comparison sentence include: "Man" in the sentence corpus and "Woman" in the comparison sentence. By analyzing "Man" and "Woman", it can be determined that the sememes corresponding to "Man" are: +[human]+[male]+[adult]; the sememes corresponding to "Woman" are: +[human]-[male]+[adult].

[0122] For the word corpus in the preset corpus, the second type of sememes in the word corpus can be analyzed through the following method:

[0123] First, determine the second semantically related words corresponding to the word corpus. The second semantically related words corresponding to the word corpus may include synonyms and / or antonyms of the word corpus. For example, for the word corpus "river", its second semantically related word may be "lake".

[0124] Secondly, input the word corpus group into the trained sememe annotation model for sememe annotation to obtain the second type of sememes corresponding to the word corpus and the second semantically related words respectively. The word corpus group includes the word corpus and the second semantically related words. The sememe annotation model is used to perform sememe annotation on the word corpus and the second semantically related words. The essence of sememe annotation is to analyze the semantic information respectively possessed by the word corpus and the second semantically related words. This semantic information includes the same semantic information and different semantic information between the word corpus and the second semantically related words. Furthermore, annotate the semantic information corresponding to the word corpus and the second semantically related words respectively, and the annotated semantic information is the second type of sememes.

[0125] In one embodiment, the sememe annotation model can be trained according to the steps S201 - S203 as shown in Figure 2 Figure.

[0126] S201, obtain multiple sample corpus groups. The sample corpus group includes a sample sentence corpus group and a sample word corpus group. The sample sentence corpus group includes the sample sentence corpus and its corresponding sample comparison sentence, and the sample comparison sentence is obtained by replacing the word to be replaced in the sample sentence corpus with a semantically related word; the sample word corpus group includes the sample word corpus and its corresponding sample semantically related word.

[0127] Among them, the word to be replaced in the sample sentence corpus can be selected by the user or automatically selected by the computer. When the computer automatically selects the word to be replaced, it can select the word with a semantically related word in the sentence corpus as the word to be replaced. The semantically related word can be a synonym and / or an antonym.

[0128] S202, determine the label information corresponding to each sample corpus group; among them, the label information corresponding to the sample sentence corpus group includes: the sememe information corresponding to the word to be replaced and its corresponding semantically related word respectively; the label information corresponding to the sample word corpus group includes: the sememe information corresponding to the sample word corpus and its corresponding sample semantically related word respectively.

[0129] Among them, the sememe information corresponding to the sample sentence corpus group includes: the same semantic information and different semantic information between the word to be replaced and the corresponding semantically related word. That is, the sememe information can reflect the distinguishing features between two different words (i.e., the word to be replaced and the corresponding semantically related word).

[0130] S203, input multiple sample corpus groups and the label information corresponding to each sample corpus group into the semantic feature annotation model to be trained for model training, and obtain the trained semantic feature annotation model.

[0131] Figure 3 This is a schematic diagram illustrating the principle of a training method for a semantic feature annotation model according to an embodiment of this application. Figure 3 As shown, in training the semantic feature annotation model, two tasks are first required. One task is to construct multiple sample sentence corpora, i.e., to obtain multiple sample sentence corpora and construct the corresponding sample comparison sentences for each sample sentence corpus. The other task is to construct multiple sample word corpora, i.e., to obtain multiple sample word corpora and construct the corresponding sample semantic related words for each sample word corpus. Then, label information is added to the sample sentence corpora and sample word corpora. The label information of the sample sentence corpora can represent the common features (i.e., the same semantic information) and distinguishing features (i.e., different semantic information) between the words to be replaced and the semantic related words in the sample sentence corpora. The label information of the sample word corpora can represent the common features (i.e., the same semantic information) and distinguishing features (i.e., different semantic information) between the sample word corpora and the sample semantic related words. Then, each sample sentence corpus and each sample word corpus is input into the semantic feature annotation model to be trained. The semantic feature annotation model to be trained performs feature learning based on each sample sentence corpus, each sample word corpus, and the label information, thereby obtaining the trained semantic feature annotation model.

[0132] Figure 4 This is a schematic flowchart illustrating a chunk recognition method according to another embodiment of this application. Figure 4 As shown, the chunk recognition method includes the following steps:

[0133] S401, perform word segmentation on the text to be identified to obtain the word segmented text corresponding to the text to be identified. The word segmented text is composed of multiple word segmentation units.

[0134] In this step, existing word segmentation techniques can be used to segment the text to be recognized. This embodiment does not limit the specific word segmentation technique used, such as dictionary-based word segmentation algorithms, semantic analysis-based word segmentation algorithms, etc. After the text to be recognized is segmented, the resulting segmented text consists of multiple segmentation units, each segmentation unit corresponding to a word.

[0135] S402, determine multiple word segmentation unit groups to be combined in the segmented text, each word segmentation unit group including multiple adjacent word segmentation units.

[0136] S403, combine multiple word segmentation units in each word segmentation unit group, and integrate the combined phrases with other uncombined word segmentation units in the word segmentation text to obtain multiple candidate language sequences corresponding to the word segmentation text.

[0137] The combination rules for word segmentation units can be pre-set. These rules may include at least one of the following: the number of word segmentation units in each group and the number of combinations corresponding to the same candidate language sequence. For example, the combination rules could be: each group contains 2 word segmentation units, and the number of combinations corresponding to the same candidate language sequence is 1. When determining multiple groups of word segmentation units to be combined in the segmented text, each pair of adjacent word segmentation units can be designated as a separate group. Since the number of combinations corresponding to the same candidate language sequence is 1, different word segmentation units will result in different candidate language sequences for the same segmented text. That is, different groups of word segmentation units correspond to different candidate language sequences. Thus, by combining each pair of adjacent word segmentation units in the segmented text, multiple candidate language sequences can be obtained.

[0138] S404, perform part-of-speech tagging on each language unit in each candidate language sequence to obtain the candidate part-of-speech tagging sequence corresponding to each candidate language sequence.

[0139] In this step, existing part-of-speech (POS) recognition algorithms can be used to identify the part of speech of each language unit. Optionally, the candidate language sequence is input into a grammar recognizer to identify the part of speech of each language unit and perform POS tagging, thereby outputting a candidate POS tag sequence corresponding to the candidate language sequence. The grammar recognizer pre-deploys any existing POS recognition tool to identify the part of speech of each language unit.

[0140] S405, select candidate part-of-speech tag sequences that conform to preset grammatical rules from multiple candidate part-of-speech tag sequences, and determine the candidate language sequence corresponding to the candidate part-of-speech tag sequence that conforms to the preset grammatical rules as the target language sequence.

[0141] Among them, the preset grammar rules can be understood as grammar rules that conform to people's normal language habits. For example, each language unit in the target language sequence has a unique part of speech, adjacent words or adjacent phrases have different parts of speech, and so on.

[0142] S406, Input the target language sequence into the pre-trained semantic vector model to perform semantic vector recognition, and obtain the semantic combination vector corresponding to each language unit in the target language sequence.

[0143] In this model, the semantic feature combination vector corresponding to a language unit is a combination of semantic feature vectors of multiple semantic features corresponding to the language unit. These multiple semantic features are used to represent the semantic information of the language unit. The semantic feature vector model includes the mapping relationship between multiple semantic features and semantic feature vectors. The construction method of the semantic feature vector model has been described in detail in the above embodiments and will not be repeated here.

[0144] S407, based on the semantic feature combination vector, calculate the semantic contribution of each word group in the target language sequence relative to the text to be identified.

[0145] The method for calculating the semantic contribution of a phrase relative to the text to be identified has been described in detail in the above embodiments and will not be repeated here.

[0146] S408, based on the semantic contribution of each phrase relative to the text to be identified, determine the phrases with semantic contributions greater than or equal to a preset threshold as target phrases.

[0147] The following example, using the text to be recognized as “early repayment refers to making repayment 5 days before the repayment date”, illustrates the chunk recognition method provided in this application.

[0148] First, the text to be identified is segmented into words to obtain the segmented text "early repayment refers to making repayment 5 days before the repayment date".

[0149] Then, according to the preset combination rules, multiple adjacent word segmentation units (i.e., word segmentation unit groups) in the segmented text are combined to obtain the following candidate language sequences: "early repayment means making repayment 5 days before the repayment date", "early repayment means making repayment 5 days before the repayment date", "early repayment means making repayment 5 days before the repayment date", and so on. This is just an example listing a few candidate language sequences. In actual applications, since segmented text usually includes many word segmentation units, a larger number of candidate language sequences can be obtained by combining different word segmentation units.

[0150] Then, part-of-speech tagging is performed on each language unit in the candidate language sequences obtained in the previous step, resulting in candidate part-of-speech tag sequences for each candidate language sequence. Specifically, the candidate part-of-speech tag sequence for the candidate language sequence "early repayment refers to repayment made 5 days before the repayment date" is "vv prep n loc quant vv"; the candidate part-of-speech tag sequence for the candidate language sequence "early repayment refers to repayment made 5 days before the repayment date" is "null"; the candidate part-of-speech tag sequence for the candidate language sequence "early repayment refers to repayment made 5 days before the repayment date" is also "null". The reason for the candidate part-of-speech tag sequence being "null" is that the part of speech for "repayment refers to" or "refers to at" in the candidate language sequence cannot be identified, i.e., it does not have a unique part of speech, therefore the candidate part-of-speech tag sequence does not conform to the preset grammatical rules. By performing this step, the target language sequence "early repayment refers to repayment made 5 days before the repayment date" is obtained.

[0151] Then, the target language sequence "early repayment refers to making repayment 5 days before the repayment date" is input into the semantic vector model, and the semantic combination vector corresponding to each language unit in the target language sequence is output.

[0152] Then, based on the semantic feature combination vector, the semantic contribution of each word group in the target language sequence relative to the text to be identified is calculated. Based on the semantic contribution of each word group relative to the text to be identified, word groups with semantic contributions greater than or equal to a preset threshold are identified as target phrases. By performing this step, the target phrase "early repayment" can be obtained.

[0153] As can be seen, the technical solution of this application involves segmenting the text to be identified into words to obtain the segmented text corresponding to the text to be identified; determining at least one target language sequence corresponding to the segmented text, wherein the target language sequence consists of multiple language units, and the multiple language units include at least one phrase obtained by combining multiple adjacent segmented units in the segmented text. Since the target language sequence includes phrases obtained by combining multiple adjacent segmented units in the segmented text, and a phrase is a phrase with semantic information, determining the target language sequence corresponding to the text to be identified can provide a strong text foundation for the identification of phrases in the text to be identified. Furthermore, the target language sequence is input into a pre-trained semantic vector model for semantic vector identification to obtain the semantic combination vector corresponding to each language unit in the target language sequence, that is, the combination of semantic vectors of multiple semantic elements; the semantic contribution of each phrase in the multiple language units relative to the text to be identified is determined based on the semantic combination vector, and then the phrase corresponding to the semantic contribution that meets the preset semantic contribution conditions is determined as the target phrase corresponding to the text to be identified. Since a semantic feature combination vector is a combination of multiple semantic feature vectors, and semantic features, as the smallest granularity in natural language processing, can more accurately represent the semantic information of each language unit, by identifying the semantic feature combination vector corresponding to each language unit in the target language sequence, and determining the target word chunk in the text to be identified based on the semantic feature combination vector, the effect of identifying word chunks in the text from the semantic feature dimension is achieved. This not only accurately identifies word chunks in the text, but also the identified word chunks have rich semantic information, which is conducive to the machine to better understand and analyze the semantics of the text, and avoids the situation where inaccurate word chunk identification in the text leads to errors in the understanding of the entire text.

[0154] In summary, specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.

[0155] The above are the chunk recognition methods provided in the embodiments of this application. Based on the same idea, the embodiments of this application also provide a chunk recognition device.

[0156] Figure 5 This is a schematic block diagram of a language block recognition device according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0157] Word segmentation module 51 is used to perform word segmentation processing on the text to be identified, and obtain the word segmented text corresponding to the text to be identified; the word segmented text is composed of multiple word segmentation units.

[0158] The first determining module 52 is used to determine at least one target language sequence corresponding to the segmented text; the target language sequence is composed of multiple language units, and the multiple language units include at least one word group obtained by combining multiple adjacent segmented units in the segmented text;

[0159] The semantic vector recognition module 53 is used to input the target language sequence into a pre-trained semantic vector model to perform semantic vector recognition, and obtain the semantic combination vector corresponding to each language unit in the target language sequence; the semantic combination vector is a combination of the semantic vectors of multiple semantic elements corresponding to the language unit; the multiple semantic elements corresponding to the language unit are used to represent the semantic information of the language unit;

[0160] The second determining module 54 is used to determine the semantic contribution of each word group in the plurality of language units relative to the text to be identified based on the semantic feature combination vector.

[0161] The third determining module 55 is used to determine the word group corresponding to the semantic contribution degree that meets the preset semantic contribution conditions as the target word block corresponding to the text to be identified, based on the semantic contribution degree of each word group.

[0162] In one embodiment, the first determining module 52 includes:

[0163] The first determining unit is used to determine multiple word segmentation unit groups to be combined in the segmented text, each word segmentation unit group including multiple adjacent word segmentation units;

[0164] A combination unit is used to combine multiple adjacent word segmentation units in the word segmentation text, and integrate the combined word group with other uncombined word segmentation units in the word segmentation text to obtain multiple candidate language sequences corresponding to the word segmentation text.

[0165] The filtering unit is used to filter at least one candidate language sequence that conforms to preset grammatical rules from the plurality of candidate language sequences as the target language sequence; the plurality of language units in the target language sequence include the word groups and the other word segmentation units.

[0166] In one embodiment, the filtering unit is specifically used for:

[0167] For each candidate language sequence, part-of-speech tagging is performed on each language unit in the candidate language sequence to obtain the candidate part-of-speech tagging sequence corresponding to the candidate language sequence;

[0168] If the candidate part-of-speech tag sequence conforms to the preset grammar rules, then the candidate language sequence corresponding to the candidate part-of-speech tag sequence is determined to be the target language sequence.

[0169] In one embodiment, the filtering unit is specifically used for:

[0170] If the candidate part-of-speech tag sequence conforms to the preset grammar rules, then the candidate part-of-speech tag sequence and the initial part-of-speech tag sequence corresponding to the segmented text are aligned according to the preset alignment rules; the preset alignment rules include: the number of aligned part-of-speech tags in the candidate part-of-speech tag sequence and the initial part-of-speech tag sequence is the largest; the initial part-of-speech tag sequence is obtained by performing part-of-speech tagging on each segmented unit in the segmented text;

[0171] Based on the alignment results, determine the multiple first part-of-speech tags that are combined in the initial part-of-speech tag sequence;

[0172] Based on the multiple first part-of-speech identifiers, the first word segmentation unit group in the segmented text is determined;

[0173] Based on the first word segmentation unit group, determine the candidate language sequence corresponding to the candidate part-of-speech tag sequence.

[0174] In one embodiment, the semantic vector model includes a mapping relationship between multiple semantic elements and semantic vectors; the semantic vector recognition module 53 is specifically used for:

[0175] The target language sequence is input into the semantic vector model to perform the following:

[0176] For each language unit in the target language sequence, if the language unit is a phrase, the phrase is split into multiple words; for each word obtained from the splitting, the semantic vector of each semantic feature corresponding to the word is determined according to the mapping relationship, and the first semantic feature combination vector corresponding to the word is determined according to the semantic feature vector of each semantic feature corresponding to the word; the first semantic feature combination vectors corresponding to each word are concatenated to obtain the second semantic feature combination vector corresponding to the phrase.

[0177] If the language unit is a word, then the first semantic element combination vector corresponding to the word is determined according to the mapping relationship.

[0178] In one embodiment, the preset semantic contribution condition includes: the semantic contribution degree is greater than or equal to a preset threshold.

[0179] In one embodiment, the second determining module 54 includes:

[0180] The first calculation unit is used to calculate, for each word group, the first proportion of the semantic feature combination vector corresponding to the word group in the total semantic feature combination vector corresponding to the target language sequence; wherein, the total semantic feature combination vector is the sum of the semantic feature combination vectors corresponding to each language unit in the target language sequence;

[0181] The second determining unit is used to determine the semantic contribution degree corresponding to the phrase based on the first proportion.

[0182] In one embodiment, the second determining module 54 includes:

[0183] The third determining unit is configured to, for each word group, determine a first weighted value of the semantic feature combination vector corresponding to the word group based on the first semantic feature weight corresponding to the word group; and determine a second weighted value of the total semantic feature combination vector based on the first semantic feature weight corresponding to each language unit in the target language sequence; wherein the total semantic feature combination vector is the sum of the semantic feature combination vectors corresponding to each language unit in the target language sequence;

[0184] The second calculation unit is used to calculate the second proportion of the first weighted value in the second weighted value;

[0185] The fourth determining unit is used to determine the semantic contribution degree corresponding to the phrase based on the second proportion.

[0186] In one embodiment, the apparatus further includes:

[0187] The fourth determining module is used to determine the second semantic feature weight corresponding to each semantic feature included in the language unit for each language unit in the target language sequence, according to a preset spatial dimension table; the spatial dimension table includes the mapping relationship between semantic features, semantic feature weights and spatial dimension information corresponding to the semantic features; the spatial dimension information includes the distribution information of the semantic features in at least one of the grammatical dimension, category dimension and connotation dimension.

[0188] The fifth determining module is used to determine the weight of the first semantic element corresponding to the language unit based on the weight of the second semantic element corresponding to each semantic element included in the language unit.

[0189] In one embodiment, the apparatus further includes:

[0190] The semantic feature extraction module is used to extract the first type of semantic features from a preset dictionary;

[0191] The corpus analysis module is used to analyze multiple corpora in a preset corpus to obtain the second type of semantic features corresponding to each corpus.

[0192] The sixth determining module is used to determine the semantic vectors corresponding to the first semantic element and the second semantic element, respectively;

[0193] A module is established to establish the mapping relationship between the first type of semantic element and its corresponding semantic element vector, and the mapping relationship between the second type of semantic element and its corresponding semantic element vector, so as to obtain the semantic element vector model.

[0194] The apparatus of this application performs word segmentation on the text to be identified to obtain the segmented text corresponding to the text to be identified; it then determines at least one target language sequence corresponding to the segmented text, the target language sequence being composed of multiple language units, including at least one phrase obtained by combining multiple adjacent segmented units in the segmented text. Since the target language sequence includes phrases obtained by combining multiple adjacent segmented units in the segmented text, and a phrase is a phrase with semantic information, determining the target language sequence corresponding to the text to be identified provides a strong textual foundation for phrase recognition in the text to be identified. Furthermore, the target language sequence is input into a pre-trained semantic feature vector model for semantic feature vector recognition, obtaining a semantic feature combination vector corresponding to each language unit in the target language sequence, i.e., a combination of semantic feature vectors of multiple semantic features; based on the semantic feature combination vector, the semantic contribution of each phrase that meets the multiple language units relative to the text to be identified is determined, and thus the phrase corresponding to the semantic contribution under the preset semantic contribution condition is determined as the target phrase corresponding to the text to be identified. Since a semantic feature combination vector is a combination of multiple semantic feature vectors, and semantic features, as the smallest granularity in natural language processing, can more accurately represent the semantic information of each language unit, by identifying the semantic feature combination vector corresponding to each language unit in the target language sequence, and determining the target word chunk in the text to be identified based on the semantic feature combination vector, the effect of identifying word chunks in the text from the semantic feature dimension is achieved. This not only accurately identifies word chunks in the text, but also the identified word chunks have rich semantic information, which is conducive to the machine to better understand and analyze the semantics of the text, and avoids the situation where inaccurate word chunk identification in the text leads to errors in the understanding of the entire text.

[0195] Those skilled in the art will understand that Figure 5 The chunk recognition device in the text can be used to implement the chunk recognition method described above. The detailed description should be similar to that in the method section above. To avoid being too complicated, it will not be repeated here.

[0196] Following the same line of thought, embodiments of this application also provide an electronic device, such as... Figure 6As shown. Electronic devices can vary considerably due to differences in configuration or performance, and may include one or more processors 601 and memory 602. Memory 602 may store one or more application programs or data. Memory 602 may be temporary or persistent storage. The application programs stored in memory 602 may include one or more modules (not shown), each module may include a series of computer-executable instructions for the electronic device. Furthermore, processor 601 may be configured to communicate with memory 602 and execute the series of computer-executable instructions in memory 602 on the electronic device. The electronic device may also include one or more power supplies 603, one or more wired or wireless network interfaces 604, one or more input / output interfaces 605, and one or more keyboards 606.

[0197] Specifically, in this embodiment, the electronic device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the electronic device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0198] The text to be identified is segmented into words to obtain the segmented text corresponding to the text to be identified; the segmented text is composed of multiple segmentation units.

[0199] Determine at least one target language sequence corresponding to the segmented text; the target language sequence consists of multiple language units, and the multiple language units include at least one word group obtained by combining multiple adjacent segmented units in the segmented text;

[0200] The target language sequence is input into a pre-trained semantic vector model for semantic vector recognition to obtain a semantic combination vector corresponding to each language unit in the target language sequence; the semantic combination vector is a combination of semantic vectors of multiple semantic elements corresponding to the language unit; the multiple semantic elements corresponding to the language unit are used to represent the semantic information of the language unit;

[0201] Based on the semantic feature combination vector, determine the semantic contribution of each word group in the plurality of language units relative to the text to be identified;

[0202] Based on the semantic contribution of each phrase, the phrases corresponding to the semantic contribution that meet the preset semantic contribution conditions are determined as the target word blocks corresponding to the text to be identified.

[0203] The technical solution of this application involves segmenting the text to be identified into words to obtain segmented text. At least one target language sequence corresponding to the segmented text is determined. This target language sequence consists of multiple language units, including at least one phrase formed by combining multiple adjacent segmented units in the segmented text. Since the target language sequence includes phrases formed by combining multiple adjacent segmented units in the segmented text, and a phrase is a phrase with semantic information, determining the target language sequence corresponding to the text to be identified provides a strong textual foundation for phrase recognition. Furthermore, the target language sequence is input into a pre-trained semantic vector model for semantic vector recognition, obtaining a semantic combination vector corresponding to each language unit in the target language sequence, i.e., a combination of semantic vectors of multiple semantic elements. Based on the semantic combination vector, the semantic contribution of each phrase in the multiple language units relative to the text to be identified is determined, and thus the phrase corresponding to the semantic contribution that meets the preset semantic contribution conditions is determined as the target phrase corresponding to the text to be identified. Since a semantic feature combination vector is a combination of multiple semantic feature vectors, and semantic features, as the smallest granularity in natural language processing, can more accurately represent the semantic information of each language unit, by identifying the semantic feature combination vector corresponding to each language unit in the target language sequence, and determining the target word chunk in the text to be identified based on the semantic feature combination vector, the effect of identifying word chunks in the text from the semantic feature dimension is achieved. This not only accurately identifies word chunks in the text, but also the identified word chunks have rich semantic information, which is conducive to the machine to better understand and analyze the semantics of the text, and avoids the situation where inaccurate word chunk identification in the text leads to errors in the understanding of the entire text.

[0204] This application also proposes a computer-readable storage medium that stores one or more computer programs, each including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform various processes of the above-described chunk recognition method embodiments, specifically for executing:

[0205] The text to be identified is segmented into words to obtain the segmented text corresponding to the text to be identified; the segmented text is composed of multiple segmentation units.

[0206] Determine at least one target language sequence corresponding to the segmented text; the target language sequence consists of multiple language units, and the multiple language units include at least one word group obtained by combining multiple adjacent segmented units in the segmented text;

[0207] The target language sequence is input into a pre-trained semantic vector model for semantic vector recognition to obtain a semantic combination vector corresponding to each language unit in the target language sequence; the semantic combination vector is a combination of semantic vectors of multiple semantic elements corresponding to the language unit; the multiple semantic elements corresponding to the language unit are used to represent the semantic information of the language unit;

[0208] Based on the semantic feature combination vector, determine the semantic contribution of each word group in the plurality of language units relative to the text to be identified;

[0209] Based on the semantic contribution of each phrase, the phrases corresponding to the semantic contribution that meet the preset semantic contribution conditions are determined as the target word blocks corresponding to the text to be identified.

[0210] The technical solution of this application involves segmenting the text to be identified into words to obtain segmented text. At least one target language sequence corresponding to the segmented text is determined. This target language sequence consists of multiple language units, including at least one phrase formed by combining multiple adjacent segmented units in the segmented text. Since the target language sequence includes phrases formed by combining multiple adjacent segmented units in the segmented text, and a phrase is a phrase with semantic information, determining the target language sequence corresponding to the text to be identified provides a strong textual foundation for phrase recognition. Furthermore, the target language sequence is input into a pre-trained semantic vector model for semantic vector recognition, obtaining a semantic combination vector corresponding to each language unit in the target language sequence, i.e., a combination of semantic vectors of multiple semantic elements. Based on the semantic combination vector, the semantic contribution of each phrase in the multiple language units relative to the text to be identified is determined, and thus, phrases with semantic contributions meeting preset semantic contribution conditions are identified as the target phrases corresponding to the text to be identified. Since a semantic feature combination vector is a combination of multiple semantic feature vectors, and semantic features, as the smallest granularity in natural language processing, can more accurately represent the semantic information of each language unit, by identifying the semantic feature combination vector corresponding to each language unit in the target language sequence, and determining the target word chunk in the text to be identified based on the semantic feature combination vector, the effect of identifying word chunks in the text from the semantic feature dimension is achieved. This not only accurately identifies word chunks in the text, but also the identified word chunks have rich semantic information, which is conducive to the machine to better understand and analyze the semantics of the text, and avoids the situation where inaccurate word chunk identification in the text leads to errors in the understanding of the entire text.

[0211] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0212] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0213] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0214] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0215] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0216] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0217] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0218] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0219] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0220] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0221] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0222] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0223] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for identifying language blocks, characterized in that, include: The text to be identified is segmented into words to obtain the segmented text corresponding to the text to be identified; The segmented text is composed of multiple segmentation units; Determine at least one target language sequence corresponding to the segmented text; the target language sequence consists of multiple language units, and the multiple language units include at least one word group obtained by combining multiple adjacent segmented units in the segmented text; The target language sequence is input into a pre-trained semantic vector model for semantic vector recognition to obtain a semantic combination vector corresponding to each language unit in the target language sequence; the semantic combination vector is a combination of semantic vectors of multiple semantic elements corresponding to the language unit; the multiple semantic elements corresponding to the language unit are used to represent the semantic information of the language unit; Based on the semantic feature combination vector, the semantic contribution of each word group in the plurality of language units relative to the text to be identified is determined. The semantic contribution of the word group relative to the text to be identified is determined based on the ratio between the semantic feature combination vector corresponding to the word group and the total semantic feature combination vector corresponding to the target language sequence, or based on the ratio between the first weighted value of the semantic feature combination vector and the second weighted value of the total semantic feature combination vector. The total semantic feature combination vector is the sum of the semantic feature combination vectors corresponding to each language unit in the target language sequence. Based on the semantic contribution of each phrase, the phrases corresponding to the semantic contribution that meet the preset semantic contribution conditions are determined as the target word blocks corresponding to the text to be identified.

2. The method according to claim 1, characterized in that, Determining at least one target language sequence corresponding to the segmented text includes: Determine multiple word segmentation unit groups to be combined in the segmented text, each word segmentation unit group including multiple adjacent word segmentation units; The word segmentation units in each of the word segmentation unit groups are combined, and the resulting word groups are integrated with other uncombined word segmentation units in the word segmentation text to obtain multiple candidate language sequences corresponding to the word segmentation text. At least one candidate language sequence that conforms to preset grammatical rules is selected from the plurality of candidate language sequences as the target language sequence; the plurality of language units in the target language sequence include the phrases and the other word segmentation units.

3. The method according to claim 2, characterized in that, The step of selecting at least one candidate language sequence that conforms to preset grammatical rules from the plurality of candidate language sequences as the target language sequence includes: For each candidate language sequence, part-of-speech tagging is performed on each language unit in the candidate language sequence to obtain the candidate part-of-speech tagging sequence corresponding to the candidate language sequence; If the candidate part-of-speech tag sequence conforms to the preset grammar rules, then the candidate language sequence corresponding to the candidate part-of-speech tag sequence is determined to be the target language sequence.

4. The method according to claim 3, characterized in that, The step of determining the candidate language sequence corresponding to the candidate part-of-speech tag sequence as the target language sequence if the candidate part-of-speech tag sequence conforms to the preset grammar rule includes: If the candidate part-of-speech tag sequence conforms to the preset grammar rules, then the candidate part-of-speech tag sequence and the initial part-of-speech tag sequence corresponding to the segmented text are aligned according to the preset alignment rules; the preset alignment rules include: the number of aligned part-of-speech tags in the candidate part-of-speech tag sequence and the initial part-of-speech tag sequence is the largest; the initial part-of-speech tag sequence is obtained by performing part-of-speech tagging on each segmented unit in the segmented text; Based on the alignment results, determine the multiple first part-of-speech tags that are combined in the initial part-of-speech tag sequence; Based on the multiple first part-of-speech identifiers, the first word segmentation unit group in the segmented text is determined; Based on the first word segmentation unit group, determine the candidate language sequence corresponding to the candidate part-of-speech tag sequence.

5. The method according to claim 1, characterized in that, The semantic vector model includes the mapping relationship between multiple semantic elements and semantic vectors; The step of inputting the target language sequence into a pre-trained semantic vector model for semantic vector recognition to obtain a semantic combination vector corresponding to each language unit in the target language sequence includes: The target language sequence is input into the semantic vector model for execution via the semantic vector model: For each language unit in the target language sequence, if the language unit is a phrase, the phrase is split into multiple words; for each word obtained from the splitting, the semantic vector of each semantic feature corresponding to the word is determined according to the mapping relationship, and the first semantic feature combination vector corresponding to the word is determined according to the semantic feature vector of each semantic feature corresponding to the word; the first semantic feature combination vectors corresponding to each word are concatenated to obtain the second semantic feature combination vector corresponding to the phrase. If the language unit is a word, then the first semantic element combination vector corresponding to the word is determined according to the mapping relationship.

6. The method according to claim 1, characterized in that, The step of determining the semantic contribution of each word group in the plurality of language units relative to the text to be identified based on the semantic feature combination vector includes: For each word group, calculate the first proportion of the semantic feature combination vector corresponding to the word group in the total semantic feature combination vector corresponding to the target language sequence; Based on the first proportion, the semantic contribution of the phrase is determined.

7. The method according to claim 1, characterized in that, The step of determining the semantic contribution of each word group in the plurality of language units relative to the text to be identified based on the semantic feature combination vector includes: For each word group, a first weighted value of the semantic feature combination vector corresponding to the word group is determined based on the weight of the first semantic feature corresponding to the word group; and a second weighted value of the total semantic feature combination vector is determined based on the weight of the first semantic feature corresponding to each language unit in the target language sequence. Calculate the second proportion of the first weighted value in the second weighted value; Based on the second proportion, the semantic contribution of the phrase is determined.

8. The method according to claim 7, characterized in that, The method further includes: For each language unit in the target language sequence, the weight of the second semantic feature corresponding to each semantic feature included in the language unit is determined according to a preset spatial dimension table; the spatial dimension table includes the mapping relationship between semantic features, semantic feature weights, and spatial dimension information corresponding to the semantic features; the spatial dimension information includes the distribution information of the semantic features in at least one of the grammatical dimension, category dimension, and connotation dimension. The weight of the first semantic element corresponding to the language unit is determined based on the weight of the second semantic element corresponding to each semantic element included in the language unit.

9. The method according to claim 5, characterized in that, The method further includes: Extract the first type of semantic element from the preset dictionary; Analyze multiple corpora in the pre-set corpus to obtain the second type of semantic features corresponding to each corpus; Determine the semantic vectors corresponding to the first semantic element and the second semantic element, respectively; Establish the mapping relationship between the first type of semantic element and its corresponding semantic element vector, and the mapping relationship between the second type of semantic element and its corresponding semantic element vector, to obtain the semantic element vector model.

10. A language block recognition device, characterized in that, include: The word segmentation module is used to perform word segmentation on the text to be identified, and obtain the word segmented text corresponding to the text to be identified; the word segmented text is composed of multiple word segmentation units. The first determining module is used to determine at least one target language sequence corresponding to the segmented text; the target language sequence is composed of multiple language units, and the multiple language units include at least one word group obtained by combining multiple adjacent segmented units in the segmented text; The semantic vector recognition module is used to input the target language sequence into a pre-trained semantic vector model for semantic vector recognition, and obtain the semantic combination vector corresponding to each language unit in the target language sequence; the semantic combination vector is a combination of the semantic vectors of multiple semantic elements corresponding to the language unit; the multiple semantic elements corresponding to the language unit are used to represent the semantic information of the language unit; The second determining module is used to determine the semantic contribution of each word group in the plurality of language units relative to the text to be identified based on the semantic feature combination vector. The semantic contribution of the word group relative to the text to be identified is determined based on the ratio between the semantic feature combination vector corresponding to the word group and the total semantic feature combination vector corresponding to the target language sequence, or based on the ratio between the first weighted value of the semantic feature combination vector and the second weighted value of the total semantic feature combination vector. The total semantic feature combination vector is the sum of the semantic feature combination vectors corresponding to each language unit in the target language sequence. The third determining module is used to determine, based on the semantic contribution of each phrase, the phrase corresponding to the semantic contribution that meets the preset semantic contribution conditions as the target word block corresponding to the text to be identified.

11. An electronic device, characterized in that, The method includes a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being configured to call and execute the computer program from the memory to implement the chunk recognition method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The storage medium is used to store a computer program that can be executed by a processor to implement the chunk recognition method as described in any one of claims 1-9.

Images