Method for establishing Arabic knowledge base and related device

Abstract

The invention provides an Arabic knowledge base establishment method and a related device, which are applied to the field of Arabic knowledge libraries, and the method comprises the following steps: firstly, preliminarily establishing an Arabic initial knowledge base, the initial knowledge base comprising a plurality of word forms, and further comprising root words and root letters corresponding to each of the plurality of word forms; the initial knowledge base is expanded, finally, the Arabic knowledge base is obtained, and the Arabic knowledge base comprises the word forms in the initial knowledge base, the root words corresponding to the word forms and the root letters corresponding to the word forms and further comprises a plurality of expanded word forms and the root words and the root letters of each expanded word form. The Arabic knowledge base can be used for restoring the word form, and the established Arabic knowledge base can be used for restoring the word form, so that the use requirement of a user on Arabic is met.

Description

Technical Field

[0001] The present application relates to the field of Arabic knowledge, and in particular to a method for establishing an Arabic knowledge base and related devices. Background Art

[0002] The complex morphological changes and rich grammatical structure of Arabic make it challenging to perform natural language processing on it. Compared with some other languages, there are relatively few Arabic corpora, and tools and resources for Arabic processing are also relatively scarce.

[0003] Due to the complexity of Arabic, a large-scale Arabic knowledge base is needed to support natural language processing tasks such as machine translation, text classification, and information retrieval. However, there is currently no large-scale Arabic knowledge base available. Summary of the invention

[0004] The present application provides an Arabic knowledge base establishment method and related devices. The established Arabic knowledge base can be used to restore word forms to meet the user's needs for using Arabic.

[0005] In a first aspect, the present application provides a method for establishing an Arabic knowledge base, comprising: establishing an initial Arabic knowledge base, the initial Arabic knowledge base comprising multiple entries, each of the multiple entries comprising a word form, a root word of the word form, and a root letter of the word form; expanding the initial Arabic knowledge base based on a corpus to obtain an Arabic knowledge base, the Arabic knowledge base comprising multiple entries and multiple extended entries, each of the multiple extended entries comprising an extended word form, a root word of the extended word form, and a root letter of the extended word form, the multiple extended word forms in the multiple extended entries comprising: one or more of a word form having the same root word as a word form in the initial Arabic knowledge base, a word form having the same root letter, a word form having the same semantics, and a word form having an associated form, wherein the associated form refers to a form having partially the same form.

[0006] The present application provides a method for establishing an Arabic knowledge base, wherein the established Arabic knowledge base includes word forms in an initial knowledge base and extended word forms obtained by expanding the word forms in the initial knowledge base, and the Arabic knowledge base also includes root words of word forms, root letters of word forms, and root words of extended word forms and root letters of extended word forms. The established Arabic knowledge base can restore word forms, for example, when a user inputs a word form in the Arabic knowledge base, the root word and root letter of the word form can be obtained, and for another example, when a user inputs a sentence in the Arabic knowledge base, the Arabic knowledge base can restore the sentence and obtain the root words and root letters of each word form in the sentence.

[0007] It should be noted that the entries and extended entries in this scheme refer to corresponding relationships. For example, an entry includes a word form, a root word of a word form, and a root letter of a word form. Then this entry refers to the corresponding relationship between the word form, the root word, and the root letter, that is, the root word and the root letter are the root word and root letter of the word form; an extended entry includes an extended word form, the root word of the extended word form, and the root letter of the extended word form. Then this extended entry refers to the corresponding relationship between the extended word form, the extended root word, and the extended root letter, that is, the root word and the root letter are the root word and root letter of the extended word form.

[0008] Based on the first aspect, in a possible implementation, the method further includes: when the extended word form in the extended entry is a word form having the same semantics as a word form in the Arabic initial knowledge base or a word form having an associated morphology, determining the word form in the Arabic initial knowledge base that has the highest morphological similarity to the extended word form, then the root letter and root word of the extended word form are the same as the root letter and root word of the word form with the highest similarity.

[0009] Based on the first aspect, in a possible implementation, before obtaining the Arabic knowledge base, the method also includes: receiving a first operation from the user, the first operation being an operation in which the user checks whether the extended word form, the root word of the extended word form, and the root letter of the extended word form in each extended entry are correct; in the event that at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form is incorrect, receiving a second operation from the user, the second operation being an operation in which the user corrects at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form.

[0010] It is understandable that due to the complexity of Arabic, the expanded word form, the root word of the expanded word form, and the root letter of the expanded word form obtained by expanding the known word form may be incorrect. Therefore, the user is required to confirm the extended word form, the root word of the extended word form, and the root letter of the extended word form. When at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form is incorrect, the user corrects it through a second operation.

[0011] Based on the first aspect, in a possible implementation, the second operation includes any one of the following forms: 1) the user directly inputs at least one of the correct extended word form, the root word of the correct extended word form, and the root letter of the correct extended word form; 2) the user inputs a regular expression, and the regular expression is used to generate the correct extended word form, the root word of the correct extended word form, and at least one of the root letter of the correct extended word form; 3) the user selects at least one of the correct extended word form, the root word of the correct extended word form, and the root letter of the correct extended word form in a specified selection box, and the specified selection box includes multiple options; 4) the user deletes the extended entry.

[0012] Based on the first aspect, in a possible implementation, multiple options are generated according to a preset mechanism, which includes one or more of the morphological change mechanism between root letters and root words, the morphological change mechanism between root words and word forms, and the morphological change mechanism in the process of adding or deleting affixes to word forms.

[0013] The extended word forms or the root words of the extended word forms or the root letters of the extended word forms in the multiple options are generated by the computing device according to a preset mechanism. For example, when determining the root words or root letters of the extended word forms, after determining the root words of the extended word forms according to the word forms with the highest degree of morphological similarity with the word forms in the initial knowledge base, the root letters can be determined based on the morphological change mechanism between the root letters and the root words. For another example, in the process of generating the extended word forms, when adding or deleting affixes to the extended word forms, the extended word forms can be determined based on the morphological change mechanism in the process of adding or deleting affixes.

[0014] Based on the first aspect, in a possible implementation, each entry in the Arabic initial knowledge base further includes an example sentence of the word form and the part of speech of the word form, and each extended entry further includes an example sentence of the extended word form and the part of speech of the extended word form.

[0015] In a second aspect, the present application provides an Arabic knowledge base establishment device, comprising:

[0016] An initial knowledge base building module is used to build an Arabic initial knowledge base, the Arabic initial knowledge base includes a plurality of entries, each of the plurality of entries includes a word form, a root word of the word form and a root letter of the word form;

[0017] The expansion module is used to expand the Arabic initial knowledge base based on the corpus to obtain the Arabic knowledge base, the Arabic knowledge base includes multiple entries and multiple extended entries, each of the multiple extended entries includes an extended word form, a root word of the extended word form, and a root letter of the extended word form, and the multiple extended word forms in the multiple extended entries include: one or more of a word form with the same root word as the word form in the Arabic initial knowledge base, a word form with the same root letter, a word form with the same semantics, and a word form with an associated form, wherein the associated form refers to a form with partially identical forms.

[0018] Based on the second aspect, in a possible implementation, the extension module is used to:

[0019] When the extended word form in the extended entry is a word form having the same semantics as a word form in the Arabic initial knowledge base or a word form having an associated morphology, the word form in the Arabic initial knowledge base having the highest morphological similarity to the extended word form is determined, and the root letter and root word of the extended word form are the same as the root letter and root word of the word form having the highest similarity.

[0020] Based on the second aspect, in a possible implementation, the device further includes:

[0021] An interactive module, used for receiving a first operation of a user, wherein the first operation is an operation of the user checking whether the extended word form, the root word of the extended word form, and the root letter of the extended word form in each extended entry are correct;

[0022] The interactive module is also used to receive a second operation from the user when at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form is incorrect, and the second operation is an operation in which the user corrects at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form.

[0023] Based on the second aspect, in a possible implementation, the second operation includes any one of the following forms:

[0024] 1) The user directly inputs at least one of the correct extended word form, the root word of the correct extended word form, and the root letter of the correct extended word form;

[0025] 2) A user inputs a regular expression, where the regular expression is used to generate at least one of a correct extended word form, a root word of the correct extended word form, and a root letter of the correct extended word form;

[0026] 3) The user selects at least one of a correct extended word form, a correct root word of the extended word form, and a correct root letter of the extended word form in a designated selection box, wherein the designated selection box includes a plurality of options;

[0027] 4) User deletes the extension entry.

[0028] Based on the second aspect, in a possible implementation, multiple options are generated according to a preset mechanism, which includes one or more of the morphological change mechanism between root letters and root words, the morphological change mechanism between root words and word forms, and the morphological change mechanism in the process of adding or deleting affixes to word forms.

[0029] Based on the second aspect, in a possible implementation, each entry in the Arabic initial knowledge base further includes an example sentence of the word form and the part of speech of the word form, and each extended entry further includes an example sentence of the extended word form and the part of speech of the extended word form.

[0030] The functional modules of the second aspect are used to implement the method of the above-mentioned first aspect and any possible implementation manner of the first aspect.

[0031] In a third aspect, the present application provides a computing device cluster, comprising at least one computing device, wherein the at least one computing device comprises a memory and a processor, and the processor in the at least one computing device is used to execute instructions stored in the memory in the at least one computing device to implement the method described in the first aspect and any possible implementation method of the first aspect.

[0032] In a fourth aspect, the present application provides a computer storage medium including program instructions. When the program instructions are executed on a computing device cluster, the computing device cluster executes the method described in the first aspect and any possible implementation of the first aspect.

[0033] In a fifth aspect, the present application provides a computer program product, including program instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute the method described in the first aspect or any possible implementation of the first aspect. The computer program product may be a software installation package, and when it is necessary to use the method provided by any possible design of the first aspect, the computer program product may be downloaded and executed on the computing device cluster to implement the method described in the first aspect and any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a process for establishing an Arabic knowledge base provided in this application;

[0035] Figure 2 A flowchart of another method for establishing an Arabic knowledge base provided in this application;

[0036] Figure 3 An example diagram provided for this application;

[0037] Figure 4 An example interface diagram provided for this application;

[0038] Figure 5 An example interface diagram provided for this application;

[0039] Figure 6 Another example interface diagram provided for this application;

[0040] Figure 7 A schematic diagram of the structure of an Arabic knowledge base establishment device provided in this application;

[0041] Figure 8 A schematic diagram of the structure of a computing device provided for this application;

[0042] Fig. 9 A schematic diagram of the structure of a computing device cluster provided for this application;

[0043] Fig.10 A schematic diagram of the structure of another computing device cluster provided in this application. DETAILED DESCRIPTION

[0044] Before introducing the embodiments of the present application, the technical terms involved in the present application are first introduced.

[0045] Root letters are the basic elements that make up a word. Each word is composed of three or four root letters. These root letters remain unchanged in different forms of the word and determine the basic meaning of the word. Root letters are usually composed of consonants, and vowels in Arabic are used to fill in the gaps between root letters.

[0046] A root word is the most basic word form consisting of root letters. It is usually a noun, verb or adjective and has a basic meaning. The root word is unchanging in Arabic, and other forms of the word are formed by inflecting and changing the root word. That is, the root word is the original form of a word without any affixes or any changes in the basic form.

[0047] Word form / actual word form (word) refers to the various inflections and changes of the root word in Arabic, including singular and plural, past and present tense, active and passive, etc. By making these changes to the root word, different word forms are formed. Generally speaking, the various word forms of a word are restored to its original form, that is, restored to the root word. Word form reduction refers to the process of restoring the word form to the root word.

[0048] Among them, the entry refers to the specific form of a word under different grammatical rules and variations. The entry in Arabic consists of a root word and a series of affixes. The root is the basic form of a word form and the core part of the word form. Different word forms can be formed by adding prefixes, suffixes or infixes to the root. Prefixes, suffixes or infixes are used to indicate different grammatical features of the word, such as tense, voice, person, number, etc. For example, in Arabic, the root word (ktb) means "writing". Based on this root, multiple entry forms can be formed, such as:

[0049] (kitāb): book (singular noun)

[0050] (kutub): books (plural noun)

[0051] (kataba): wrote (past tense verb)

[0052] (yaktubu): writing (present tense verb)

[0053] (sayaktubu): will write (future tense verb)

[0054] (maktab): table (noun related to writing)

[0055] (maktabah): library

[0056] (maktūb): written, written (adjective)

[0057] (maktabiyy): my office (noun)

[0058] This is just some of (ktb) is an example of a word formed from a root. There are many more words in Arabic that are formed based on this root. By adding prefixes, suffixes and infixes to the root word, different tenses, persons, numbers and voices can be formed, thus expanding the richness and expressiveness of the vocabulary.

[0059] Before introducing the embodiment of the method of the present application, the method of establishing the Arabic knowledge base in the present application scheme is briefly described. Figure 1 , Figure 1 A flowchart of a method for establishing an Arabic knowledge base provided in this application.

[0060] First, an initial Arabic knowledge base is established based on Arabic dictionaries, Arabic textbooks, etc. The initial knowledge base includes multiple word forms, root words and root letters of each word form; then, the word forms in the initial knowledge base are expanded based on Arabic synonym dictionaries, Arabic announcements, official newspapers, literary materials and other corpora to obtain a larger number of word forms; but in the process of expansion, some inaccurate word forms are inevitably introduced, or the root words and root letters determined when determining the root words and root letters of the extended word forms are incorrect. Therefore, user interaction needs to be introduced to judge whether the extended word forms, the root words and root letters of the extended word forms are correct by the users, and the incorrect ones can be modified in time; finally, an Arabic knowledge base is obtained.

[0061] When a user inputs a word form in the Arabic knowledge base, the user can query the root word and root letter corresponding to the word form through the Arabic knowledge base, and can also query some example sentences about the word form, etc. Alternatively, the user can input a sentence in the Arabic knowledge base and query the text after the restoration of each word form in the sentence.

[0062] See also Figure 2 , Figure 2 A flowchart of a method for establishing an Arabic knowledge base provided in the present application, the method can be executed by a computing device, and the method includes but is not limited to the description below.

[0063] S101. Establish an Arabic initial knowledge base (first knowledge base), wherein the Arabic initial knowledge base includes a plurality of entries, each of the plurality of entries including a word form, a root word of the word form, and a root letter of the word form.

[0064] The Arabic initial knowledge base includes a plurality of entries, each of which includes a word form, a root word of the word form, and a root letter of the word form. Optionally, each entry in the Arabic initial knowledge base may also include an example sentence of the word form and a part of speech of the word form. The example sentence of the word form refers to an exemplary sentence composed of the word form, and the part of speech of the word form may be, for example, a noun, a verb, an adjective, etc.

[0065] The Arabic initial knowledge base can be generated based on the contents recorded in Arabic dictionaries, textbooks, etc. The dictionary can be an electronic dictionary or a paper dictionary. For example, based on the contents recorded in Arabic dictionaries and textbooks, the correspondence between all root letters, the root words corresponding to all root letters, and the affixes that can be used for each root letter is obtained, and then the correspondence between all word forms and root letters is obtained, thereby forming multiple entries, each of which includes a word form, the root word of the word form, and the root letter of the word form. Optionally, examples of a part of the word form can also be obtained from the examples recorded in Arabic dictionaries and textbooks, and each entry can also include examples of the word form.

[0066] In one example, each entry can be represented by a tuple, for example, each entry includes a word form, a root word of the word form, a root letter of the word form, an example sentence of the word form, and a part of speech of the word form, where each piece of information is a grammar, and the entry can be represented by a quintuple.

[0067] Optionally, each entry may also include frequency, which consists of three bigrams, which are used to record the number of brothers of the root letter, the number of times the root letter appears, the number of brothers of the root word, the number of times the root word appears, the number of brothers of the word form, and the number of times the word form appears. There are 28 root letters in Arabic, so the number of brothers of the root letter is a fixed value of 28. The number of brothers of a root word refers to how many different root words can be combined with the root letters that constitute the root word. For example, see Figure 3 The schematic diagram shown, Figure 3 The root letters shown in can be combined into root word 1, root word 2 and root word 3. According to root word 1, actual word (word form) 1 and actual word (word form) 2 can be obtained. Figure 3In the above, the number of brothers of root word 1 refers to the number of root words that can be combined according to the root letters that constitute root word 1. The brothers of root word 1 include root word 1, root word 2 and root word 3. The number of brothers of root word 1 (including root word 1 itself) is 3. Similarly, the number of brothers of root word 2 or root word 3 is also 3. The number of brothers of a word form refers to the number of actual word forms that can be formed by the root words of the word form. For example, Figure 3 In the figure, the number of brothers of word form 1 refers to the number of actual word forms that can be formed by the root word 1 that constitutes word form 1. The brothers of word form 1 include word form 1 and word form 2. The number of brothers of word form 1 (including word form 1 itself) is 2. The number of times the root letter appears, the number of times the root word appears, and the number of times the word form appears are not fixed values, but change in real time. In the Arabic initial knowledge base, they have an initial value, which is the number of times they appear in the corpus (Arabic dictionary, textbooks, etc.) used to establish the Arabic initial knowledge base. Their values ​​may increase as the corpus increases.

[0068] Figure 3 The schematic diagram shown includes multiple entries. For example, an entry may be: root letter, root word 1, actual word (word form) 1, wherein root word 1 is composed of root letters, and word form 1 is synthesized based on root word 1, or in other words, the root word of word form 1 is root word 1, and the root letter of word form 1 is Figure 3 Root letters shown; Figure 3 The entries also include: root letter, root word1, actual word (word form)2. Figure 3 This is just an example. In actual applications, the root letters can be other root letters, each root letter can form a greater or lesser number of root words, and each root word can be synthesized into a greater or lesser number of word forms.

[0069] It should be noted that the entries and extended entries in this scheme refer to corresponding relationships. For example, an entry includes a word form, a root word of a word form, and a root letter of a word form. Then this entry refers to the corresponding relationship between the word form, the root word, and the root letter, that is, the root word and the root letter are the root word and root letter of the word form; an extended entry includes an extended word form, the root word of the extended word form, and the root letter of the extended word form. Then this extended entry refers to the corresponding relationship between the extended word form, the extended root word, and the extended root letter, that is, the root word and the root letter are the root word and root letter of the extended word form.

[0070] S102. Expand the initial Arabic knowledge base based on the corpus to obtain a second knowledge base, wherein the second knowledge base includes multiple entries and a first extended entry set, each extended entry in the first extended entry set includes an extended word form, a root word of the extended word form, and a root letter of the extended word form, and the multiple extended word forms in the first extended entry set include: one or more of word forms having the same root word as the word forms in the initial Arabic knowledge base, word forms having the same root letters, word forms having the same semantics, and word forms having associated morphology.

[0071] The Arabic initial knowledge base is expanded based on the corpus to obtain a second knowledge base, wherein the corpus includes one or more of an Arabic synonym dictionary, an Arabic pre-trained model, newspapers, announcements, literary materials, etc. The second knowledge base includes multiple entries in the initial knowledge base and a first extended entry set, wherein each extended entry in the first extended entry set includes an extended word form, a root word of the extended word form, and a root letter of the extended word form. The following describes how to expand the initial knowledge base.

[0072] The Arabic synonym dictionary includes word forms with the same or similar semantics. Based on the Arabic synonym dictionary, word forms with the same or similar semantics as word forms in the Arabic initial knowledge base can be obtained through regular expressions, wherein the regular expressions are used to obtain word forms with the same or similar semantics as word forms in the Arabic initial knowledge base.

[0073] The Arabic pre-training model is based on some extended word forms trained on an Arabic corpus (e.g., an Arabic dictionary, a synonym dictionary, and online materials). The word forms that meet the target constraints can be obtained by inputting target constraints into the Arabic pre-training model. For example, the target constraints include word forms with the same root words or root letters as the word forms in the Arabic initial knowledge base, and the obtained word forms include word forms with the same root words or root letters as the word forms in the Arabic initial knowledge base. The word form is a part of the extended word form in the first extended item set, and the root letters and / or root words of this part of the extended word form are determined. The extended word form, the root word of the extended word form, and the root letter of the extended word form constitute an extended item.

[0074] The training samples can also be annotated by natural language annotation, and then the training samples are trained based on the annotation to obtain a trained model. For example, the annotation of the training samples includes the annotation of conjunctions "or", "also called", "also called", etc., the annotation of the training samples also includes the annotation of punctuation marks "", ","", "-", etc., and the annotation of the training samples also includes the annotation of word forms with associated morphology. These conjunctions or punctuation marks are used to indicate that the adjacent previous and next word forms have the same or similar semantics. Word forms with associated morphology refer to two or more word forms with partially identical morphology, including two or more word forms with the same prefix, the same suffix, or the same middle affix of two or more word forms, and also including two or more word forms with more than a threshold number of consecutive letters in the Arabic writing method being the same, wherein the threshold number can be freely set. Morphology includes letters, the order of letters, and dots and horizontal lines above, below, or beside the letters. In Arabic, dots and horizontal lines above, below, or beside letters are usually called (harakat) or (tashkil), these symbols are used to indicate pronunciation and grammatical information to help readers pronounce the text correctly.

[0075] It is also possible to annotate the word forms with the same or coreference relationship in the training samples. For example, "bus, public bus, is an important means of transportation for people's daily travel", in which bus and public bus are word forms with the same or coreference relationship. According to the trained model, the word forms with the same or similar semantics and the word forms with the same associated morphology in the Arabic corpus (for example, Arabic dictionary, synonym dictionary, network data) are mined to obtain the extended word forms.

[0076] The initial knowledge base can also be expanded based on official Arabic newspapers, announcements, and literary materials, for example, word forms that appear in official newspapers, announcements, and literary materials but not in the initial knowledge base can be added to the initial knowledge base. Example sentences of a certain word form that appears in official newspapers, announcements, and literary materials can also be added to the example sentences of the corresponding word form in the initial knowledge base, and frequency information in a multi-tuple can be updated based on a certain word form in official newspapers, announcements, and literary materials.

[0077] The initial knowledge base is expanded by one or more of the above-mentioned methods to obtain an extended word form. For extended word forms whose root words and root letters are not determined, for example, extended word forms with the same semantics as the word forms in the Arabic initial knowledge base or extended word forms with associated morphology, the root words and root letters of these extended word forms can be determined according to the similarity. Specifically, for a certain extended word form, the similarity of the extended word form and each word form whose root words and root letters have been determined is calculated, and the target word form with the highest similarity to the extended word form is determined, and the root word and root letter of the target word form are used as the root word and root letter of the extended word form, wherein each word form whose root words and root letters have been determined includes each word form in the initial knowledge base, and the word form in the extended word form that has the same root word or the same root letter as the word form in the initial knowledge base. The present application does not specifically limit the method for calculating similarity.

[0078] After the extended word form, the root word of the extended word form and the root letter of the extended word form are determined, the extended word form, the root word of the extended word form and the root letter of the extended word form form an extended entry. Optionally, the extended entry may also include one or more of the example sentence of the extended word form, the part of speech of the extended word form and the frequency, wherein the frequency is continuously updated.

[0079] It should be noted that the extended word form, the root word of the extended word form, and the root letter of the extended word form in each extended entry in the second knowledge base are not necessarily all correct. In some extended entries, one or more of the extended word form, the root word of the extended word form, and the root letter of the extended word form are incorrect. Therefore, the extended entries need to be manually checked and corrected.

[0080] S103, receiving a first operation of the user, where the first operation is an operation of the user checking whether the extended word form, the root word of the extended word form, and the root letter of the extended word form in each extended entry are correct.

[0081] The user checks the extended word form, the root word of the extended word form, and the root letter of the extended word form in each extended entry by clicking, sliding, etc., and confirms / checks whether the extended word form, the root word of the extended word form, and the root letter of the extended word form in each extended entry are correct. Accordingly, the computing device receives the user's first operation, which is an operation for the user to check whether the extended word form, the root word of the extended word form, and the root letter of the extended word form in each extended entry are confirmed.

[0082] For example, see Figure 4 As shown, Figure 4 This is an example interface diagram provided for this application. Figure 4 In the example, the text to be restored is a sentence composed of extended word forms. Each word form in the example sentence is restored to obtain each root word. The text composed of each root word is Figure 4The restored text in the example sentence is restored one by one, wherein the multiple word forms in the example sentence are restored separately, the to-be-restored term refers to which word form in the example sentence needs to be restored next, and the triggering rules include what the rules are for restoring the to-be-restored word forms in the to-be-restored term. Figure 4 The interface diagram shown is merely an example and does not constitute a limitation to the present application.

[0083] In the right Figure 4 During the process of restoring the text to be restored or after the restoration of the text to be restored is completed, the user can check whether the restored text is correct (the first operation).

[0084] S104. When at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form is incorrect, a second operation of the user is received to obtain an Arabic knowledge base, wherein the second operation is an operation of the user correcting at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form.

[0085] At least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form is incorrect, including: the extended word form is incorrect; the extended word form and the root word of the extended word form are correct but the root letter of the extended word form is incorrect; the extended word form and the root letter of the extended word form are correct but the root word of the extended word form is incorrect; only the extended word form is correct but both the root word and the root letter of the extended word form are incorrect.

[0086] In the case that at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form is incorrect, the computing device receives a second operation of the user, and the second operation is an operation of the user to correct at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form, and the Arabic knowledge base is obtained after the correction. The correction operation is introduced below.

[0087] In one implementation, when at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form is incorrect, the user can directly modify the incorrect extended word form, and / or the incorrect root word, and / or the incorrect root letter based on prior knowledge to obtain the correct extended entry.

[0088] In one implementation, when at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form is incorrect, the user can enter a regular expression, and the computing device generates the correct word form, and / or the correct root word, and / or the correct root letter based on the regular expression to obtain the correct extended entry. For example, if the root word is incorrect, the user can enter a regular expression in the box where the root word is located, and the computing device generates the correct root word based on the regular expression. If the root letter is incorrect, the user can enter a regular expression in the box where the root letter is located, and the computing device generates the correct root letter based on the regular expression.

[0089] In one implementation, on the interactive interface of the computing device, there is a selection box at the position of the root word or root letter of the extended word form. When at least one of the root word of the extended word form and the root letter of the extended word form is incorrect, the user can select the correct root word and / or root letter in the selection box corresponding to the root word and / or root letter, wherein the selection box includes multiple options. For example, the selection box corresponding to the root word includes multiple root word options. The user can see the multiple root word options by clicking the selection box, and select a root word option from them to obtain the correct root word.

[0090] Among them, multiple options are generated according to preset mechanisms, which include one or more of the morphological change mechanism between root letters and root words, the morphological change mechanism between root words and word forms, and the morphological change mechanism in the process of adding or deleting affixes in word forms, wherein the morphological change includes the change of letters, the change of adding or deleting points above or below letters. For example, the commonly used morphological change mechanisms between root letters and root words are shown in Table 1.

[0091] Table 1

[0092]

[0093]

[0094] For example, the commonly used morphological change mechanisms between root words and word forms are shown in Table 2.

[0095] Table 2

[0096]

[0097] For example, in the process of adding and deleting affixes to word forms, the commonly used morphological change mechanisms are shown in Table 3.

[0098] Table 3

[0099]

[0100] Table 1 lists the correspondence between the root letter and the morphology after the root letter is combined with the root word. From Table 1, it can be obtained that the root letter is obtained according to the synthesized root word based on the morphological change mechanism; Table 2 lists the correspondence between the root word and the morphology after the root word generates the word form. From Table 2, it can be obtained that the root word corresponding to the word form is obtained according to the word form based on the morphological change mechanism; Table 3 lists the correspondence between the affix and the morphology after the affix is ​​combined with the word form. From Table 3, it can be obtained that the original affix is ​​obtained according to the affix in the word form based on the morphological change mechanism.

[0101] The order of multiple options can be determined by historical behavior or similar changes, or by weighted historical behavior and similar changes, where the weights of historical behavior and similar changes can be set by the user. Historical behavior refers to the change operations that the user has modified, and similar changes refer to similar changes that the user has modified in historical operations.

[0102] For example, see Figure 5 As shown, Figure 5 A schematic diagram of an interface provided for this application. Figure 5 In the correction, the correction includes root word change correction, actual word form change correction and affix change correction. Root word change correction refers to the correction of the root word of the restored word. Figure 5 Several root words are given for selection. Users can select the correct root word from the given root words. The actual word form correction refers to the correction of the restored word, and the affix correction refers to the correction of the affix of the restored word. If the root word of the restored word is incorrect, users can select the correct one from the given root words. If the root word of the restored word is correct, no processing is done. Similarly, users can correct word forms and affixes. After the correction is completed, you can click submit.

[0103] See also Figure 6 , Figure 6 Another interface schematic diagram provided for this application. Figure 6 In the correction, the correction includes root word correction and root letter correction. The user can directly enter the correct root word in the box where the root word is located. The user can click the drop-down box where the root letter is located to select the correct root letter. The drop-down box where the root letter is located provides multiple options. Alternatively, the user can directly enter the root word and root letter in the two boxes of the processing rule (which can contain regular expressions) respectively, or enter the regular expressions for generating the root word and root letter in the two boxes of the processing rule respectively, and generate the root word and root letter through the regular expression. After the correction is completed, you can click submit.

[0104] In one implementation, when the extended word form is incorrect, for example, the extended word form does not exist, the extended entry where the extended word form is located can be directly deleted.

[0105] S105. When the extended word form, the root word of the extended word form, and the root letter of the extended word form are all correct, receive a third operation from the user to obtain an Arabic knowledge base, wherein the third operation is an operation to confirm that the extended word form, the root word of the extended word form, and the root letter of the extended word form in each extended entry are correct.

[0106] If the extended word form, the root word of the extended word form, and the root letter of the extended word form are all correct, a third operation of the user is received, and the third operation is an operation for the user to confirm that the extended word form, the root word of the extended word form, and the root letter of the extended word form in each extended item are correct. That is, if the user checks that the information in each extended item is correct, the user clicks to confirm, and finally an Arabic knowledge base is generated.

[0107] The generated Arabic knowledge base can be used for various natural language processing tasks.

[0108] After the Arabic knowledge base is generated, the user can input a word form, and the computing device searches the established Arabic knowledge base for the root word, root letter, and example sentence corresponding to the word form, and displays them on the interactive interface for the user to view.

[0109] It can be seen that the present application provides a method for establishing an Arabic knowledge base, wherein the established Arabic knowledge base includes word forms in the initial knowledge base and extended word forms obtained by expanding the word forms in the initial knowledge base, and the Arabic knowledge base also includes root words of word forms, root letters of word forms, and root words of extended word forms and root letters of extended word forms. The established Arabic knowledge base can restore word forms, for example, when a user enters a word form in the Arabic knowledge base, the root word and root letter of the word form can be obtained, and for another example, when a user enters a sentence in the Arabic knowledge base, the Arabic knowledge base can restore the sentence and obtain the root words and root letters of each word form in the sentence.

[0110] In this application, the process of word form restoration is actually a table lookup process. Due to the complex grammatical structure of Arabic, it is impossible to restore the word form by training, inference, and prediction of a neural network model. Therefore, this application scheme is expanded according to the word form in the initial knowledge base, and the extended word form, the root word of the extended word form, and the root letter of the extended word form are manually confirmed and corrected, and finally an Arabic knowledge base is obtained. The user can use the Arabic knowledge base to restore the word form, and query the root word of a certain word form from the Arabic knowledge base in the form of a table lookup. The established Arabic knowledge base can be used for Arabic learners to learn, and can also be used for natural language processing.

[0111] The method embodiments of the present application are introduced above, and the device embodiments corresponding to the method embodiments are introduced below.

[0112] See also Figure 7 , Figure 7 The present application provides a schematic diagram of the structure of an Arabic knowledge base establishment device 700, the device 700 comprising:

[0113] An initial knowledge base building module 710 is used to build an Arabic initial knowledge base, wherein the Arabic initial knowledge base includes a plurality of entries, each of which includes a word form, a root word of the word form, and a root letter of the word form;

[0114] The expansion module 720 is used to expand the Arabic initial knowledge base based on the corpus to obtain an Arabic knowledge base, wherein the Arabic knowledge base includes multiple entries and multiple expansion entries, each of the multiple expansion entries includes an expansion word form, a root word of the expansion word form, and a root letter of the expansion word form, and the multiple expansion word forms in the multiple expansion entries include: one or more of a word form having the same root word as a word form in the Arabic initial knowledge base, a word form having the same root letter, a word form having the same semantics, and a word form having an associated form, wherein the associated form refers to a form having partially the same form.

[0115] In a possible implementation, the expansion module 720 is used to:

[0116] When the extended word form in the extended entry is a word form having the same semantics as a word form in the Arabic initial knowledge base or a word form having an associated morphology, the word form in the Arabic initial knowledge base having the highest morphological similarity to the extended word form is determined, and the root letter and root word of the extended word form are the same as the root letter and root word of the word form having the highest similarity.

[0117] In a possible implementation, the apparatus 700 further includes:

[0118] Interaction module 730, used to receive a first operation of a user, where the first operation is an operation of the user checking whether the extended word form, the root word of the extended word form, and the root letter of the extended word form in each extended entry are correct;

[0119] The interaction module 730 is also used to receive a second operation from the user when at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form is incorrect, and the second operation is an operation in which the user corrects at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form.

[0120] In a possible implementation, the second operation includes any of the following forms:

[0121] 1) The user directly inputs at least one of the correct extended word form, the root word of the correct extended word form, and the root letter of the correct extended word form;

[0122] 2) A user inputs a regular expression, where the regular expression is used to generate at least one of a correct extended word form, a root word of the correct extended word form, and a root letter of the correct extended word form;

[0123] 3) The user selects at least one of a correct extended word form, a correct root word of the extended word form, and a correct root letter of the extended word form in a designated selection box, wherein the designated selection box includes a plurality of options;

[0124] 4) User deletes the extension entry.

[0125] In a possible implementation, multiple options are generated based on a preset mechanism, which includes one or more of a morphological change mechanism between root letters and root words, a morphological change mechanism between root words and word forms, and a morphological change mechanism in the process of adding or deleting affixes to word forms.

[0126] In a possible implementation, each entry in the Arabic initial knowledge base further includes an example sentence of the word form and the part of speech of the word form, and each extended entry further includes an example sentence of the extended word form and the part of speech of the extended word form.

[0127] Among them, the initial knowledge base building module 710, the expansion module 720 and the interaction module 730 can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the expansion module 720 is introduced below by taking the expansion module 720 as an example. Similarly, the implementation of the initial knowledge base building module 710 and the interaction module 730 can refer to the implementation of the expansion module 720.

[0128] The device 700 can be deployed on a computing device. As an example of a software functional unit, the extension module 720 may include code running on a computing device. Among them, the computing device may be a computing device in a cloud service, and the computing device may be, for example, a bare metal server, a virtual machine, a container, etc. Further, the computing device may be one or more. For example, the extension module 720 may include code running on multiple computing devices. It should be noted that the multiple computing devices used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple computing devices used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Among them, generally a region may include multiple availability zones AZ.

[0129] Similarly, multiple computing devices used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set in each VPC to achieve interconnection between VPCs through the communication gateway.

[0130] As an example of a hardware functional unit, the expansion module 720 may include at least one computing device. Alternatively, the expansion module 720 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0131] The multiple computing devices included in the extension module 720 can be distributed in the same region or in different regions. The multiple computing devices included in the extension module 720 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the extension module 720 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0132] It should be noted that, in other embodiments, the expansion module 720 can be used to execute any step in a method for establishing an Arabic knowledge base, and the initial knowledge base establishment module 710 and the interactive module 730 can both be used to execute any step in a method for establishing an Arabic knowledge base. The steps that the initial knowledge base establishment module 710, the expansion module 720, and the interactive module 730 are responsible for implementing can be specified as needed. The initial knowledge base establishment module 710, the expansion module 720, and the interactive module 730 respectively implement different steps in a method for establishing an Arabic knowledge base to achieve all the functions of the device 700.

[0133] See also Figure 8 , Figure 8The present application provides a schematic diagram of a computing device 800, which may be, for example, a bare metal server, a virtual machine, a container, etc. The computing device 800 may be configured as an apparatus 700, and the computing device 800 includes: a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate with each other through the bus 802. It should be understood that the present application does not limit the number of processors and memories in the computing device 800.

[0134] The bus 802 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus 802 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 802 may include a path for transmitting information between various components of the computing device 800 (eg, the memory 806, the processor 804, and the communication interface 808).

[0135] The processor 804 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0136] The memory 806 may include a volatile memory, such as a random access memory (RAM). The processor 804 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0137] The memory 806 stores executable program codes, and the processor 804 executes the executable program codes to respectively implement the functions of the aforementioned initial knowledge base building module 710, the expansion module 720, and the interaction module 730, thereby implementing an Arabic knowledge base building method. That is, the memory 806 stores instructions for executing an Arabic knowledge base building method.

[0138] The communication interface 808 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or communication networks.

[0139] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a desktop computer, a laptop computer, or other devices.

[0140] like Fig. 9 As shown, Fig. 9 The schematic diagram of the structure of a computing device cluster provided by the present application includes at least one computing device 800. The memory 806 in one or more computing devices 800 in the computing device cluster may store the same instructions for executing a method for establishing an Arabic knowledge base.

[0141] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing a method for establishing an Arabic knowledge base. In other words, the combination of one or more computing devices 800 can be used to jointly execute instructions for an Arabic knowledge base establishment method.

[0142] It should be noted that the memory 806 in different computing devices 800 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the computing device 800. That is, the instructions stored in the memory 806 in different computing devices 800 may implement the functions of one or more modules among the initial knowledge base building module 710, the expansion module 720 and the interaction module 730.

[0143] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.10 A possible implementation is shown. Fig.10As shown, two computing devices 800A and 800B are connected via a network. Specifically, they are connected to the network via the communication interface in each computing device. In this type of possible implementation, the memory 806 in the computing device 800A stores instructions for executing the functions of the initial knowledge base establishment module 710. At the same time, the memory 806 in the computing device 800B stores instructions for the functions of the expansion module 720 and the interaction module 730. Among them, the initial knowledge base establishment module 710 in the computing device 800A is used to establish an Arabic initial knowledge base, the expansion module 720 in the computing device 800B is used to expand the initial knowledge base, and the interaction module 730 is used for the user to confirm and correct the extended word form, the root word and the root letter of the extended word form.

[0144] It should be understood that Fig.10 The functions of the computing device 800A shown in the figure may also be completed by multiple computing devices 800, or the computing device cluster may include multiple computing devices with the same functions as the computing device 800A. Similarly, the functions of the computing device 800B may also be completed by multiple computing devices 800, or the computing device cluster may include multiple computing devices with the same functions as the computing device 800B.

[0145] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig. 9 and Fig.10 The connection mode of the computing device cluster. Differently, the memory 806 in one or more computing devices 800 in the computing device cluster may store different instructions for executing a method for establishing an Arabic knowledge base. In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing a method for establishing an Arabic knowledge base. In other words, a combination of one or more computing devices 800 can jointly execute instructions for executing a method for establishing an Arabic knowledge base.

[0146] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes a method for establishing an Arabic knowledge base.

[0147] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct a computing device or a computing device cluster to execute a method for establishing an Arabic knowledge base.

[0148] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for establishing an Arabic knowledge base, characterized in that: include: Establishing an Arabic initial knowledge base, wherein the Arabic initial knowledge base includes a plurality of entries, each of the plurality of entries includes a word form, a root word of the word form, and a root letter of the word form; The Arabic initial knowledge base is expanded based on a corpus to obtain an Arabic knowledge base, wherein the Arabic knowledge base includes the multiple entries and multiple extended entries, each of the multiple extended entries includes an extended word form, a root word of the extended word form, and a root letter of the extended word form, and the multiple extended word forms in the multiple extended entries include: one or more of a word form having the same root word as a word form in the Arabic initial knowledge base, a word form having the same root letter, a word form having the same semantics, and a word form having an associated morphology, wherein the associated morphology refers to a morphology having partially the same structure as a word form in the Arabic initial knowledge base.

2. The method according to claim 1, characterized in that The method further comprises: When the extended word form in the extended entry is a word form having the same semantics as a word form in the initial Arabic knowledge base or a word form having an associated morphology, the word form in the initial Arabic knowledge base having the highest morphological similarity to the extended word form is determined, and the root letter and root word of the extended word form are the same as the root letter and root word of the word form with the highest similarity.

3. The method according to claim 1 or 2, characterized in that: Before obtaining the Arabic knowledge base, the method further includes: receiving a first operation of a user, wherein the first operation is an operation of the user checking whether the extended word form, the root word of the extended word form, and the root letter of the extended word form in each extended entry are correct; In the case that at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form is incorrect, a second operation of the user is received, and the second operation is an operation in which the user corrects at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form.

4. The method according to claim 3, characterized in that The second operation includes any of the following forms: 1) The user directly inputs at least one of the correct extended word form, the root word of the correct extended word form, and the root letter of the correct extended word form; 2) the user inputs a regular expression, wherein the regular expression is used to generate at least one of the correct extended word form, the root word of the correct extended word form, and the root letter of the correct extended word form; 3) the user selects at least one of the correct extended word form, the root word of the correct extended word form, and the root letter of the correct extended word form in a designated selection box, wherein the designated selection box includes a plurality of options; 4) The user deletes the extended entry.

5. The method according to claim 4, characterized in that The multiple options are generated according to a preset mechanism, which includes one or more of the morphological change mechanism between root letters and root words, the morphological change mechanism between root words and word forms, and the morphological change mechanism in the process of adding or deleting affixes to word forms.

6. The method according to any one of claims 1 to 5, characterized in that: Each entry in the Arabic initial knowledge base further includes an example sentence of the word form and the part of speech of the word form, and each extended entry further includes an example sentence of the extended word form and the part of speech of the extended word form.

7. An Arabic knowledge base building device, characterized in that: include: An initial knowledge base building module, used for building an Arabic initial knowledge base, wherein the Arabic initial knowledge base includes a plurality of entries, each of the plurality of entries includes a word form, a root word of the word form and a root letter of the word form; An expansion module is used to expand the Arabic initial knowledge base based on a corpus to obtain an Arabic knowledge base, wherein the Arabic knowledge base includes the multiple entries and multiple expansion entries, each of the multiple expansion entries includes an expansion word form, a root word of the expansion word form, and a root letter of the expansion word form, and the multiple expansion word forms in the multiple expansion entries include: a word form having the same root word as a word form in the Arabic initial knowledge base, a word form having the same root letter, a word form having the same semantics, and a word form having an associated form, wherein the associated form refers to a form having partially the same form.

8. The device according to claim 7, characterized in that The expansion module is used to: When the extended word form in the extended entry is a word form having the same semantics as a word form in the initial Arabic knowledge base or a word form having an associated morphology, the word form in the initial Arabic knowledge base having the highest morphological similarity to the extended word form is determined, and the root letter and root word of the extended word form are the same as the root letter and root word of the word form with the highest similarity.

9. The device according to claim 7 or 8, characterized in that The device also includes: An interactive module, configured to receive a first operation of a user, wherein the first operation is an operation of the user checking whether the extended word form, the root word of the extended word form, and the root letter of the extended word form in each extended entry are correct; The interaction module is also used to receive a second operation from the user when at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form is incorrect, and the second operation is an operation for the user to correct at least one of the extended word form, the root word of the extended word form, and the root letter of the extended word form.

10. The device according to claim 9, characterized in that The second operation includes any of the following forms: 1) The user directly inputs at least one of the correct extended word form, the root word of the correct extended word form, and the root letter of the correct extended word form; 2) the user inputs a regular expression, wherein the regular expression is used to generate at least one of the correct extended word form, the root word of the correct extended word form, and the root letter of the correct extended word form; 3) the user selects at least one of the correct extended word form, the root word of the correct extended word form, and the root letter of the correct extended word form in a designated selection box, wherein the designated selection box includes a plurality of options; 4) The user deletes the extended entry.

11. The device according to claim 10, characterized in that The multiple options are generated according to a preset mechanism, which includes one or more of the morphological change mechanism between root letters and root words, the morphological change mechanism between root words and word forms, and the morphological change mechanism in the process of adding or deleting affixes to word forms.

12. The device according to any one of claims 7 to 11, characterized in that Each entry in the Arabic initial knowledge base further includes an example sentence of the word form and the part of speech of the word form, and each extended entry further includes an example sentence of the extended word form and the part of speech of the extended word form.

13. A computing device cluster, characterized in that: The method comprises at least one computing device, wherein the at least one computing device comprises a memory and a processor, wherein the processor in the at least one computing device is used to execute instructions stored in the memory in the at least one computing device to implement the method according to any one of claims 1 to 6.

14. A computer storage medium, characterized in that: The method comprises program instructions. When the program instructions are executed on a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 6.

Images