Corpus construction method, device, electronic device and storage medium

Abstract

The present invention provides a corpus construction method, device, electronic device and storage medium, wherein the method includes: determining a target discipline; selecting related disciplines related to the target discipline from each candidate discipline based on the intersection of the target discipline and each candidate discipline in the discipline hierarchy, and / or the similarity of the meanings of nouns in the target discipline and each candidate discipline; and constructing a corpus based on the corpus of the target discipline and the corpus of the related disciplines. The method, device, electronic device and storage medium provided by the present invention select related disciplines related to the target discipline from the perspective of discipline classification and / or the meaning of common nouns, thereby ensuring the objectivity and accuracy of the selection of related disciplines, thereby constructing a corpus covering the corpus of the target discipline and its related disciplines, and its comprehensiveness can meet the needs of subsequent applications of the corpus such as machine translation.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a corpus construction method, device, electronic device and storage medium. Background Art

[0002] In the training and optimization process of machine learning, a large-scale corpus is one of the indispensable source data. Here, a corpus refers to a reasonably sampled electronic text library that contains language materials that have actually appeared in actual use.

[0003] At present, the construction of a corpus for any subject can usually be divided into two ways: one is to collect only the electronic texts of the subject, and the other is to collect both the electronic texts of the subject and the electronic texts of subjects related to the subject.

[0004] Taking into account the frequent cross-references between disciplines, collecting electronic texts of only one discipline will inevitably lead to a lack of content in the corpus constructed thereby; and when collecting electronic texts of disciplines related to this discipline, since most of the related disciplines are artificially defined, the selection of related disciplines is not objective, so it is difficult to ensure that the corpus constructed thereby is emotionally neutral, and it cannot meet the training requirements of machine learning. Summary of the invention

[0005] The present invention provides a corpus construction method, device, electronic device and storage medium, which are used to solve the defects of poor objectivity and low accuracy of corpus construction in the prior art.

[0006] The present invention provides a corpus construction method, comprising:

[0007] Identify target disciplines;

[0008] Based on the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure, and / or the similarity of the meanings of nouns in the target discipline and each candidate discipline, selecting related disciplines related to the target discipline from each candidate discipline;

[0009] A corpus is constructed based on the corpus of the target subject and the corpus of the related subjects.

[0010] According to a corpus construction method provided by the present invention, the intersection of the target subject and each candidate subject in the subject hierarchy structure is determined based on the following steps:

[0011] Based on the subject hierarchy structure, determining the intersection level of the target subject and each candidate subject;

[0012] Based on the intersection level between the target discipline and each candidate discipline, the intersection situation between the target discipline and each candidate discipline in the discipline hierarchy structure is determined.

[0013] According to a corpus construction method provided by the present invention, the step of determining the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure based on the intersection level of the target discipline and each candidate discipline includes:

[0014] Based on the hierarchical weights corresponding to the intersection levels of the target discipline and each candidate discipline, determining the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure;

[0015] The level weights are determined based on the number of disciplines contained in each level in the discipline hierarchy.

[0016] According to a corpus construction method provided by the present invention, the similarity of the meanings of the nouns in the target discipline and each candidate discipline is determined based on the following steps:

[0017] Based on the noun set of the target discipline and the noun sets of each candidate discipline, determining the common nouns of the target discipline and each candidate discipline respectively;

[0018] Based on the distribution of the meanings of the common nouns in the target discipline and the candidate disciplines, the similarity of the meanings of the nouns in the target discipline and the candidate disciplines is determined.

[0019] According to a corpus construction method provided by the present invention, the method of determining the similarity of the meanings of the common nouns in the target discipline and the candidate disciplines based on the distribution of the meanings of the common nouns in the target discipline and the candidate disciplines respectively includes:

[0020] Based on the distribution of the meanings of the common nouns in the target discipline and the candidate disciplines, and the relationship between the common meanings of the common nouns in the target discipline and the candidate disciplines, determining the similarity of the meanings of the nouns in the target discipline and the candidate disciplines;

[0021] The common meaning is the most frequent meaning of the common noun in the corresponding subject.

[0022] According to a corpus construction method provided by the present invention, based on the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure, and / or the similarity of the meanings of nouns in the target discipline and each candidate discipline, selecting related disciplines related to the target discipline from each candidate discipline includes:

[0023] Determine the subject score of each candidate subject based on the corpus size of each candidate subject, the intersection of the target subject and each candidate subject in the subject hierarchy, and / or the similarity of the meaning of nouns in the target subject and each candidate subject;

[0024] Based on the subject score of each candidate subject, an associated subject related to the target subject is selected from each candidate subject.

[0025] According to a corpus construction method provided by the present invention, the corpus is constructed based on the corpus of the target subject and the corpus of the related subject, comprising:

[0026] Determining the associated corpus volume of the associated subject based on the subject score of the associated subject;

[0027] A corpus is constructed based on the corpus of the target subject and the corpus of the associated subjects under the associated corpus volume.

[0028] According to a corpus construction method provided by the present invention, the corpus is constructed based on the corpus of the target subject and the corpus of the associated subject under the associated corpus volume, comprising:

[0029] Storing the corpus of the target subject into a corpus;

[0030] With the goal of making the sum of sentiment scores neutral, based on the sentiment scores of each corpus in the related disciplines, corpora of related disciplines within the related corpus volume are selected and stored in the corpus.

[0031] The present invention also provides a corpus construction device, comprising:

[0032] The subject determination unit is used to determine the target subject;

[0033] A correlation determination unit, configured to select a correlation subject related to the target subject from each candidate subject based on the intersection of the target subject and each candidate subject in the subject hierarchy structure and / or the similarity of the meanings of nouns in the target subject and each candidate subject respectively;

[0034] A construction unit is used to construct a corpus based on the corpus of the target subject and the corpus of the related subject.

[0035] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned corpus construction methods is implemented.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the corpus construction method described in any one of the above is implemented.

[0037] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned corpus construction methods.

[0038] The corpus construction method, device, electronic device and storage medium provided by the present invention start from the subject classification level and / or the common noun meaning level, select related subjects related to the target subject, ensure the objectivity and accuracy of the selection of related subjects, and thus construct a corpus covering the corpus of the target subject and its related subjects, and its comprehensiveness can meet the needs of subsequent corpus applications such as machine translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flowchart of the corpus construction method provided by the present invention;

[0041] Figure 2 It is a schematic diagram of the subject hierarchy structure provided by the present invention;

[0042] Figure 3 It is a flow chart of a method for determining the intersection of a target discipline and each candidate discipline in a discipline hierarchy structure provided by the present invention;

[0043] Figure 4 It is a flow chart of a method for determining similarities between the meanings of nouns in a target discipline and each candidate discipline provided by the present invention;

[0044] Figure 5 is a flow chart of step 120 in the corpus construction method provided by the present invention;

[0045] Figure 6 is a flow chart of step 130 in the corpus construction method provided by the present invention;

[0046] Figure 7 It is a structural schematic diagram of the corpus construction device provided by the present invention;

[0047] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The construction of a corpus is usually aimed at a specific subject. From the perspective of subject category, the construction of a corpus can usually be divided into two ways. One is to collect only the electronic texts of the subject. This method can ensure the purity of the corpus and can better ensure that the corpus has no emotional tendency, but the defect is also obvious, that is, it does not contain information from cross-disciplinary subjects; the other is to collect both the electronic texts of the subject and the electronic texts of related subjects. Although this method has been extended to related subjects, the current method is relatively arbitrary in specific expansion, lacks specific corpus collection principles, does not consider the correlation between related subjects, and it is difficult to ensure that the corpus as a whole is emotionally neutral.

[0050] In view of the above problems, an embodiment of the present invention provides a corpus construction method. Figure 1 It is a flowchart of the corpus construction method provided by the present invention, such as Figure 1 As shown, the method includes:

[0051] Step 110, determine the target subject.

[0052] Specifically, the target discipline is the discipline for which the corpus to be constructed is aimed, such as civil aviation, law and economics, etc. The target discipline can be one discipline or multiple disciplines. It can be understood that when there are multiple target disciplines, each target discipline can construct its own independent corpus, and the method for constructing the corresponding corpus for each target discipline is the same. For the sake of ease of expression, the following is explained using a target discipline as an example.

[0053] Step 120, based on the intersection of the target discipline and each candidate discipline in the discipline hierarchy, and / or the similarity of the meanings of nouns in the target discipline and each candidate discipline, select related disciplines related to the target discipline from each candidate discipline.

[0054] Specifically, there may be multiple candidate disciplines for the target discipline. The candidate disciplines here are disciplines that may have a disciplinary association with the target discipline. The scope of the candidate disciplines may be all disciplines, or may be disciplines that have cross-references with the target discipline, or may be disciplines that belong to the same discipline category as the target discipline, which is not specifically limited in the embodiment of the present invention.

[0055] After determining the target discipline and its candidate disciplines, related disciplines related to the target discipline can be selected from the candidate disciplines, so that the corpus can be constructed by combining the corpus of the target discipline and the related disciplines to ensure the comprehensiveness and objectivity of the corpus construction. Specifically, in the process of selecting related disciplines, the correlation between the target discipline and the candidate disciplines can be measured from the intersection of the target discipline and the candidate disciplines in the discipline hierarchy, and / or the similarity of the meanings of nouns in the target discipline and the candidate disciplines.

[0056] Among them, the subject hierarchy structure is a tree structure that includes various subjects according to the subject classification method. Each subject in the subject hierarchy structure is divided according to the classification, for example Figure 2 It is a schematic diagram of the subject hierarchy structure provided by the present invention, such as Figure 2 As shown, civil aviation belongs to transportation engineering in the discipline hierarchy, and civil aviation can also be subdivided into commercial aviation, industrial aviation, agricultural aviation, aviation research and exploration, aviation sports and other disciplines. In addition, disciplines at the same level as transportation engineering include road and railway engineering, traffic information engineering and control, etc. Transportation engineering can also include road and airport engineering, traffic engineering, urban rail and railway engineering, transportation management engineering and other disciplines at the same level as civil aviation.

[0057] The intersection of the target discipline and the candidate discipline in the discipline hierarchy structure can reflect at which specific level the target discipline and the candidate discipline intersect in the discipline hierarchy structure, that is, to which discipline level the target discipline and the candidate discipline belong in the discipline hierarchy structure. It can be understood that the lower the level at which the target discipline and the candidate discipline intersect in the discipline hierarchy structure, the stronger the correlation between the two at the discipline classification level; the higher the level at which the target discipline and the candidate discipline intersect in the discipline hierarchy structure, the weaker the correlation between the two at the discipline classification level; and the situation where the target discipline and the candidate discipline do not intersect in the discipline hierarchy structure can indicate that there is no correlation between the two at the discipline classification level. For example, Figure 2 In the data, civil aviation and transportation management engineering intersect at the third level of transportation engineering, while civil aviation and fluid mechanics intersect at the first level of engineering. The relationship between civil aviation and transportation management engineering is obviously higher than that between civil aviation and fluid mechanics.

[0058] In addition, in the scenario of multidisciplinary cross-reference, the same noun may be applied to different disciplines, and the meaning of the same noun in different disciplines may be the same or different. For example, turbulence means "turbulence" or "bumping" in civil aviation, and "shock" in medicine; bus means "bus" in civil aviation, "bus" in computers, and "bus" in other disciplines. It is understandable that the stronger the correlation between the two disciplines, the weaker the correlation between the two disciplines, if the meanings of the same noun in two disciplines are mostly the same or similar; and the weaker the correlation between the two disciplines, if the meanings of the same noun in two disciplines are mostly unrelated or even opposite.

[0059] When selecting related subjects, we can consider only the intersection of the subject hierarchy structure and select candidate subjects with higher correlation at the subject classification level as related subjects, or we can consider only the similarity of the meanings of nouns in the target subject and the candidate subjects and select candidate subjects with stronger similarity at the level of common noun meaning as related subjects. We can also consider both of the above and score candidate subjects at the subject classification level and the level of common noun meaning, so as to select candidate subjects with higher scores as related subjects. The related subjects selected in this way are more objective and accurate than those manually specified, and can better meet the needs of subsequent corpus applications such as machine translation.

[0060] Step 130: construct a corpus based on the corpus of the target subject and the corpus of the related subjects.

[0061] Specifically, after determining the target discipline and the related disciplines related to the target discipline, the corpus of these two disciplines can be combined to build a corpus for the target discipline. Here, the corpus of each discipline can be papers, textbooks, news reports, analysis reports, etc., which can be obtained by downloading from the Internet or using computer crawlers.

[0062] The corpus constructed in this way can not only cover the corpus of the target discipline itself, but also cover the corpus of disciplines that have more cross-references with the target discipline, ensuring the comprehensive coverage of the corpus.

[0063] The method provided in the embodiment of the present invention starts from the subject classification level and / or the common noun meaning level, selects related subjects related to the target subject, ensures the objectivity and accuracy of the selection of related subjects, and thus constructs a corpus covering the target subject and its related subject corpus, and its comprehensiveness can meet the needs of subsequent applications of the corpus such as machine translation.

[0064] Based on the above embodiments, Figure 2The discipline hierarchy structure shown includes five discipline levels, among which the 1st to 3rd levels can be divided with reference to the currently recognized discipline classification method or the departments of universities and research institutes. For example, the 1st level can be a discipline category, the 2nd level can be a first-level discipline, and the 3rd level can be a second-level discipline; the 4th and 5th levels can be divided by field personnel based on professional knowledge, and the embodiments of the present invention do not make specific limitations on this.

[0065] In particular, since each corpus has its own emphasis, for example, the same textbook is unlikely to focus on both "commercial aviation" and "aerospace scientific research and exploration", each corpus should correspond to the fifth level in the subject hierarchy. The precise division of the corpus helps to ensure the accuracy of corpus construction.

[0066] Based on any of the above embodiments, Figure 3 : is a flow chart of a method for determining the intersection of a target discipline and each candidate discipline in a discipline hierarchy structure provided by the present invention, such as Figure 3 As shown, the intersection of the target discipline applied in step 120 and each candidate discipline in the discipline hierarchy structure can be determined based on the following steps:

[0067] Step 310: Based on the subject hierarchy structure, determine the intersection level between the target subject and each candidate subject.

[0068] Specifically, the discipline hierarchy structure shows the classification relationship between disciplines at different levels in the form of a tree structure. For the target discipline and any candidate discipline, the levels of the target discipline and the candidate discipline can be located in the discipline hierarchy structure, as well as the discipline to which the discipline belongs in the previous level. By tracing upward level by level, it can be determined whether the target discipline and the candidate discipline have the same upper-level discipline, that is, whether the target discipline and the candidate discipline have an intersection. If the target discipline and the candidate discipline have the same upper-level discipline, the level of the upper-level discipline in the discipline hierarchy structure can be determined as the intersection level of the target discipline and the candidate discipline.

[0069] For example, Figure 2 In the figure, assuming that civil aviation is the target discipline and transportation management engineering is the candidate discipline, the two disciplines have the same parent discipline, namely transportation engineering, which belongs to the third level, that is, the intersection level of civil aviation and transportation management engineering is the third level; assuming that civil aviation is the target discipline and fluid mechanics is the candidate discipline, the two disciplines have the same parent discipline, namely engineering, which belongs to the first level, that is, the intersection level of civil aviation and fluid mechanics is the first level.

[0070] Step 320, based on the intersection level of the target discipline and each candidate discipline, determine the intersection status of the target discipline and each candidate discipline in the discipline hierarchy structure.

[0071] Specifically, the intersection level of the target discipline and the candidate disciplines can reflect the correlation between the target discipline and the candidate disciplines from the discipline classification level. The higher the level, the weaker the correlation, and the lower the level, the stronger the correlation. For example, the correlation between civil aviation and transportation management engineering is obviously higher than the correlation between civil aviation and fluid mechanics. Therefore, based on the intersection level of the target discipline and the candidate disciplines, the intersection of the target discipline and the candidate disciplines in the discipline hierarchy structure can be determined. For example, the intersection level can be directly stored in the intersection situation, or the intersection situation can be determined based on the strength of the correlation reflected by the intersection level. The embodiment of the present invention does not specifically limit this.

[0072] Based on any of the above embodiments, step 320 includes:

[0073] Based on the hierarchical weights corresponding to the intersection levels of the target discipline and each candidate discipline, determining the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure;

[0074] The level weights are determined based on the number of elements each level contains in the subject hierarchy.

[0075] Specifically, considering that different intersection levels in the disciplinary hierarchy have different impacts on the size of the correlation between disciplines, when determining the intersection situation, personalized hierarchical weights for the intersection level can be added to improve the reliability of the intersection situation in reflecting the correlation.

[0076] Here, the hierarchical weight of each level can be determined based on the number of disciplines included in the discipline hierarchy structure of each level. It can be understood that, for a certain level, the more disciplines included in the level, the finer the granularity of the discipline classification in the level, and the stronger the correlation between the disciplines intersecting at this level at the discipline classification level.

[0077] Furthermore, the intersection of the target discipline and the candidate discipline in the discipline hierarchy structure can be expressed in the form of hierarchy weight. The size of the hierarchy weight corresponds to the correlation at the discipline classification level. The larger the hierarchy weight, the stronger the correlation, and the smaller the hierarchy weight, the smaller the correlation.

[0078] For example, in Figure 2 In the discipline hierarchy structure shown, assuming that the target discipline is a discipline at the fourth level, the intersection of the target discipline and the candidate discipline in the discipline hierarchy structure is s 交汇情况 (target subject, candidate subject) can be expressed as follows:

[0079]

[0080] Among them, n i is the number of disciplines at the i-th level in the discipline hierarchy.

[0081] Understandably, Figure 2 In the discipline hierarchy structure shown, when the target discipline is a discipline at the 4th level, the intersection level cannot be the 4th and 5th levels, so the above formula does not consider the case where the disciplines at the 4th and 5th levels are the same.

[0082] Based on any of the above embodiments, Figure 4 : is a flow chart of a method for determining the similarity of the meanings of a term in a target discipline and each candidate discipline provided by the present invention, such as Figure 4 As shown, the similarity of the meanings of the nouns used in step 120 in the target discipline and each candidate discipline can be determined based on the following steps:

[0083] Step 410, based on the noun set of the target discipline and the noun sets of each candidate discipline, determining the common nouns of the target discipline and each candidate discipline respectively;

[0084] Step 420, based on the distribution of the meanings of the common nouns in the target discipline and the candidate disciplines, determine the similarity of the meanings of the nouns in the target discipline and the candidate disciplines.

[0085] Specifically, each subject may correspond to a noun set. The noun set of a subject may include commonly used nouns of the subject. For example, the frequency of occurrence of each noun in the corpus of the subject may be counted, and the frequently appearing nouns may be stored in the noun set. The noun set of a subject may also include professional nouns of the subject. For example, the professional nouns of the subject may be extracted from the professional noun explanation part in the corpus of the subject and stored in the noun set. The embodiments of the present invention do not make specific limitations on this.

[0086] For a target discipline and any candidate discipline, the intersection of the noun set of the target discipline and the noun set of the candidate discipline can be taken as the common noun of the target discipline and the candidate discipline.

[0087] For common nouns, we can determine the noun meanings that the common nouns themselves can have, and then count the frequencies of the common nouns appearing in each noun meaning in the two disciplines, thereby forming the meaning distribution of the common nouns in the two disciplines. By comparing the meaning distributions of the common nouns in the two disciplines, the similarities of the meanings of the common nouns in the two disciplines can be determined. For example, the meaning distribution of a common noun in the two disciplines can be represented in the form of vectors, and then the similarities of the meanings of the common noun in the two disciplines can be obtained by calculating the similarities of the two vectors. By counting the similarities of the meanings of all common nouns in the two disciplines, we can obtain the similarities of the meanings of the nouns in the target discipline and each candidate discipline.

[0088] Based on any of the above embodiments, step 420 includes:

[0089] Based on the distribution of the meanings of the common nouns in the target discipline and the candidate disciplines, and the relationship between the common meanings of the common nouns in the target discipline and the candidate disciplines, determining the similarity of the meanings of the nouns in the target discipline and the candidate disciplines;

[0090] The common meaning is the most frequent meaning of the common noun in the corresponding subject.

[0091] Specifically, when measuring the similarity of the meanings of nouns in the target discipline and each candidate discipline, in addition to referring to the distribution of the meanings of common nouns in the target discipline and each candidate discipline, you can also refer to the relationship between the common meanings of common nouns in the target discipline and each candidate discipline.

[0092] Here, in view of the fact that a common noun may have multiple meanings in a subject, the frequency of occurrence of each meaning of the noun in the corpus of the subject can be counted, and the meaning with the highest frequency can be selected as the common meaning of the noun.

[0093] The relationship between the common meanings of common nouns in the target discipline and a candidate discipline can be four types: synonyms, near-synonyms, antonyms, and unrelated words. Different relationships will affect the relevance between the target discipline and the candidate discipline.

[0094] Based on this, the similarities between the meanings of nouns in the target discipline and candidate disciplines are 相似情况 (target subject, candidate subject) can be expressed as follows:

[0095]

[0096]

[0097] Where W = A∩B, A = {a i} is the noun set of the target subject, B = {b i} is a noun set of candidate disciplines. δ, ε, μ, and τ can be set based on business experience.

[0098] represents the meaning distribution of the common noun w in the target subject. Assume that the common noun w appears n times in the corpus of the target subject and has m different meanings. Appearance i times, obviously n1+…+n m =n, so we have,

[0099]

[0100] Similarly, we can get It should be noted that the various meanings of the common noun w here can be translated by calling a machine translation engine.

[0101] In the formula, the dot product of two vectors is calculated as When the two are aligned, the former is denoted as m 目标学科 dimensional vector, the latter is m 候选学科 dimensional vector, where m 共有 If the dimensions are the same, both should be transformed into m 目标学科 +m 候选学科 -m 共有 dimensional vector and then dot product.

[0102] In addition, when judging whether the relationship between two commonly used meanings is synonyms, near-synonyms, antonyms, or unrelated words, the synonyms, near-synonyms, and antonyms can be judged by searching the corresponding dictionary. If the two commonly used meanings are not synonyms, near-synonyms, or antonyms, then the relationship between the two is determined to be unrelated words.

[0103] Based on any of the above embodiments, Figure 5 is a flow chart of step 120 in the corpus construction method provided by the present invention, such as Figure 5 As shown, step 120 includes:

[0104] Step 121, based on the corpus size of each candidate discipline, the intersection of the target discipline and each candidate discipline in the discipline hierarchy, and / or the similarity of the meanings of nouns in the target discipline and each candidate discipline, determine the discipline score of each candidate discipline.

[0105] Step 122, based on the subject score of each candidate subject, select related subjects related to the target subject from each candidate subject.

[0106] Specifically, in the process of selecting related disciplines, one can refer not only to the correlation between disciplines at the discipline classification level and / or the common noun meaning level, but also to the corpus size of the candidate discipline itself.

[0107] The corpus size of a candidate subject can reflect the richness of the corpus materials of the candidate subject. There may be significant differences in the corpus size of different subjects, and the reason for the difference is mostly the different levels of attention received. Considering the corpus size of the candidate subject when selecting related subjects can avoid the corpus of less-focused and less-appearing subjects accounting for too high a proportion in the corpus.

[0108] Specifically, in the selection process, each candidate discipline can be scored based on the intersection of the target discipline and each candidate discipline in the discipline hierarchy, and / or the correlation between disciplines as reflected by the similarity of the meanings of nouns in the target discipline and each candidate discipline, as well as the attention paid to the discipline as reflected by the corpus size of each candidate discipline. In this way, a discipline score for each candidate discipline is obtained. The discipline score here combines the correlation between disciplines and the corpus richness of the discipline itself, and reflects the possibility of the discipline being a related discipline.

[0109] Furthermore, the subject scoring can score the intersection, similarity and corpus size separately, and then add the corresponding scores of the three, or weight the corresponding scores of the three. It can also combine the intersection and / or similarity to perform a correlation score, perform a richness score based on the corpus size, and then add the correlation score and the richness score. The embodiment of the present invention does not make any specific limitations on this.

[0110] For example, the subject rating relation (target subject, candidate subject) can be expressed as follows: relation (target subject, candidate subject)

[0111] =α×s 交汇情况 (target subject, candidate subject)+β×s 相似情况 (target subject, candidate subject)+γ×s 语料规模 (Candidate Subject)

[0112] In the formula, α, β, and γ can be preset based on business experience, s 语料规模 (Candidate subject) refers to the corpus size of the candidate subject.

[0113] After obtaining the subject score of each candidate subject, the first preset number of candidate subjects can be determined as related subjects, for example, the first 5 or 3 candidate subjects ranked from high to low in subject score can be determined as related subjects; candidate subjects with subject scores greater than a preset score threshold can also be determined as related subjects, for example, candidate subjects with subject scores greater than 80 or 70 points can be determined as related subjects.

[0114] Based on any of the above embodiments, the corpus size s of the candidate subject 语料规模 (Candidate subject) can be expressed as follows:

[0115]

[0116] In the formula, subject k represents any candidate subject, wordnum 学科k Indicates the number of words in the material of subject k.

[0117] Here, for the objectivity and accuracy of the corpus size assessment, when counting the number of words in the materials of each subject, it is necessary to ensure that the methods and parameters used to obtain the materials of each subject are the same. For example, crawling news reports of certain media in the same time period, or downloading publicly published papers in the same time period, which is not specifically limited in the embodiment of the present invention.

[0118] Based on any of the above embodiments, Figure 6 is a flow chart of step 130 in the corpus construction method provided by the present invention, such as Figure 6 As shown, step 130 includes:

[0119] Step 131, determining the associated corpus volume of the associated subject based on the subject score of the associated subject;

[0120] Step 132: construct a corpus based on the corpus of the target subject and the corpus of the associated subjects under the associated corpus volume.

[0121] Specifically, after determining the associated disciplines of the target discipline, the proportion of the corpus of the associated disciplines in the corpus can be configured based on the discipline scores of the associated disciplines. Specifically, the amount of corpus of the associated disciplines that can be stored in the corpus, that is, the associated corpus amount, can be determined based on the discipline scores of the associated disciplines. Here, the corpus amount can be the amount of words. For example, when the amount of corpus of all associated disciplines that can be stored in the corpus is a fixed value, the proportion of the associated corpus amount of each associated discipline in the fixed value can be determined based on the discipline scores of each associated discipline. The higher the discipline score, the higher the proportion of the associated corpus in the fixed value, and the lower the discipline score, the smaller the proportion of the associated corpus in the fixed value.

[0122] After determining the associated corpus volume of each related subject, a corpus can be constructed based on the corpus of the target subject and the corpus of the related subjects under the associated corpus volume. For example, all the corpus of the target subject can be stored in the corpus, and the corpus corresponding to the associated corpus volume can be selected from the corpus of each related subject and stored in the corpus.

[0123] The method provided by the embodiment of the present invention determines the amount of related corpus based on the subject scores of the related subjects, thereby ensuring that the corpus is relatively unbiased while ensuring the comprehensive coverage of the corpus.

[0124] Based on any of the above embodiments, the associated corpus volume of the associated discipline can be determined based on the following formula:

[0125]

[0126] In the formula, the total number of related disciplines is N, and related discipline k is one of the N related disciplines. 基准is the amount of target subject corpus stored in the corpus, and RATIO is the ratio of non-target subject corpus to target subject corpus in the final corpus.

[0127] Based on any of the above embodiments, step 132 includes:

[0128] Storing the corpus of the target subject into a corpus;

[0129] With the goal of making the sum of sentiment scores neutral, based on the sentiment scores of each corpus in the related disciplines, corpora of related disciplines within the related corpus volume are selected and stored in the corpus.

[0130] Specifically, in the process of building a corpus, for the corpus of each related subject, it is necessary not only to consider the proportion of the corpus of the related subject, but also to balance the sentiment of the corpus of each related subject stored in the corpus, and try to keep the sentiment neutrality of the corpus. Therefore, in the process of storing the corpus of related subjects in the corpus, it is necessary to store the corpus with the goal of making the sum of the sentiment scores neutral.

[0131] Furthermore, the corpus for the target subject can be directly stored in the corpus. For example, Figure 2 For civil aviation in the corpus, the corpus of each subject in the 5th layer under civil aviation can be stored in the corpus.

[0132] For the corpus of any related subject, it is necessary to first call the sentiment analysis engine to determine the sentiment score of each corpus of the related subject. The sentiment score here can be positive, negative or zero. On this basis, the sum of the sentiment scores is as neutral as possible, that is, the sum of the sentiment scores is as zero as possible. The corpus is put into the database until the corpus of the related subject put into the database reaches the related corpus volume. At this time, the corpus for the related subject is completed.

[0133] Optionally, in order to achieve the goal of presenting a neutral sum of sentiment scores, in the process of storing corpus of any related subject, a greedy algorithm or a dynamic programming algorithm may be called to select and store the corpus, which is not specifically limited in the embodiment of the present invention.

[0134] The method provided in the embodiment of the present invention stores corpus with the goal of making the sum of sentiment scores neutral, so that the corpus can maintain sentiment neutrality, which helps to meet the needs of subsequent corpus applications such as machine translation.

[0135] Based on any of the above embodiments, the corpus construction method includes the following steps:

[0136] First, determine the target discipline and candidate disciplines.

[0137] Subsequently, the intersection level of the target discipline and each candidate discipline in the discipline hierarchy structure is determined, and based on the intersection level of the target discipline and each candidate discipline, the intersection situation of the target discipline and each candidate discipline in the discipline hierarchy structure is determined;

[0138] Based on the distribution of the meanings of the common nouns in the target discipline and the candidate disciplines, and the relationship between the common meanings of the common nouns in the target discipline and the candidate disciplines, the similarity of the meanings of the nouns in the target discipline and the candidate disciplines is determined;

[0139] Next, the subject score of each candidate subject is determined based on the corpus size of each candidate subject, the intersection of the target subject and each candidate subject in the subject hierarchy, and the similarity of the meanings of nouns in the target subject and each candidate subject, thereby selecting related subjects related to the target subject from each candidate subject.

[0140] The corpus of the target subject is stored in the corpus, and based on the subject scores of each related subject, the amount of related corpus of each related subject is determined. Then, with the sum of sentiment scores being neutral as the goal, based on the sentiment scores of each corpus in each related subject, the corpus of each related subject within the amount of related corpus is selected and stored in the corpus, thereby completing the corpus construction.

[0141] Based on any of the above embodiments, Figure 7 is a schematic diagram of the structure of the corpus construction device provided by the present invention, such as Figure 7 As shown, the device comprises:

[0142] A subject determination unit 710 is used to determine a target subject;

[0143] The association determination unit 720 is used to select an associated discipline related to the target discipline from each candidate discipline based on the intersection of the target discipline and each candidate discipline in the discipline hierarchy and / or the similarity of the meanings of nouns in the target discipline and each candidate discipline respectively;

[0144] The construction unit 730 is used to construct a corpus based on the corpus of the target subject and the corpus of the related subject.

[0145] The device provided by the embodiment of the present invention starts from the subject classification level and / or the common noun meaning level, selects related subjects related to the target subject, ensures the objectivity and accuracy of the selection of related subjects, and thus constructs a corpus covering the target subject and its related subject corpus, the comprehensiveness of which can meet the needs of subsequent applications of the corpus such as machine translation.

[0146] Based on any of the above embodiments, the device further includes an intersection determination unit, configured to:

[0147] Based on the subject hierarchy structure, determining the intersection level of the target subject and each candidate subject;

[0148] Based on the intersection level between the target discipline and each candidate discipline, the intersection situation between the target discipline and each candidate discipline in the discipline hierarchy structure is determined.

[0149] Based on any of the above embodiments, the intersection determination unit is specifically used for:

[0150] Based on the hierarchical weights corresponding to the intersection levels of the target discipline and each candidate discipline, determining the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure;

[0151] The level weights are determined based on the number of disciplines contained in each level in the discipline hierarchy.

[0152] Based on any of the above embodiments, the device further includes a similarity determination unit, configured to:

[0153] Based on the noun set of the target discipline and the noun sets of each candidate discipline, determining the common nouns of the target discipline and each candidate discipline respectively;

[0154] Based on the distribution of the meanings of the common nouns in the target discipline and each candidate discipline, the similarity of the meanings of the nouns in the target discipline and each candidate discipline is determined.

[0155] Based on any of the above embodiments, the similarity determination unit is specifically used for:

[0156] Based on the distribution of the meanings of the common nouns in the target discipline and the candidate disciplines, and the relationship between the common meanings of the common nouns in the target discipline and the candidate disciplines, determining the similarity of the meanings of the nouns in the target discipline and the candidate disciplines;

[0157] The common meaning is the most frequent meaning of the common noun in the corresponding subject.

[0158] Based on any of the foregoing embodiments, the association determination unit is specifically configured to:

[0159] Determine the subject score of each candidate subject based on the corpus size of each candidate subject, the intersection of the target subject and each candidate subject in the subject hierarchy, and / or the similarity of the meaning of nouns in the target subject and each candidate subject;

[0160] Based on the subject score of each candidate subject, an associated subject related to the target subject is selected from each candidate subject.

[0161] Based on any of the above embodiments, the construction unit is specifically used for:

[0162] Determining the associated corpus volume of the associated subject based on the subject score of the associated subject;

[0163] A corpus is constructed based on the corpus of the target subject and the corpus of the associated subjects under the associated corpus volume.

[0164] Based on any of the above embodiments, the construction unit is specifically used for:

[0165] Storing the corpus of the target subject into a corpus;

[0166] With the goal of making the sum of sentiment scores neutral, based on the sentiment scores of each corpus in the related disciplines, corpora of related disciplines within the related corpus volume are selected and stored in the corpus.

[0167] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the corpus construction method, which includes: determining the target discipline; based on the intersection of the target discipline and each candidate discipline in the discipline hierarchy, and / or the similarity of the meanings of the nouns in the target discipline and each candidate discipline, selecting the associated discipline related to the target discipline from each candidate discipline; and constructing the corpus based on the corpus of the target discipline and the corpus of the associated discipline.

[0168] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0169] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the corpus construction method provided by the above-mentioned methods, which includes: determining a target discipline; based on the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure, and / or the similarity of the meanings of nouns in the target discipline and each candidate discipline, selecting an associated discipline related to the target discipline from each candidate discipline; and constructing a corpus based on the corpus of the target discipline and the corpus of the associated discipline.

[0170] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the corpus construction method provided by the above-mentioned methods, the method comprising: determining a target discipline; selecting an associated discipline related to the target discipline from each candidate discipline based on the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure, and / or the similarity of the meanings of nouns in the target discipline and each candidate discipline; and constructing a corpus based on the corpus of the target discipline and the corpus of the associated discipline.

[0171] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0172] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A corpus construction method, characterized in that: include: Identify target disciplines; Based on the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure, and / or the similarity of the meanings of nouns in the target discipline and each candidate discipline, selecting related disciplines related to the target discipline from each candidate discipline; Building a corpus based on the corpus of the target subject and the corpus of the related subject; The intersection of the target discipline and each candidate discipline in the discipline hierarchy is determined based on the following steps: Based on the subject hierarchy structure, determining the intersection level of the target subject and each candidate subject; Based on the intersection level of the target discipline and each candidate discipline, determining the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure; The similarity of the meaning of the noun in the target discipline and each candidate discipline is determined based on the following steps: Based on the noun set of the target discipline and the noun sets of each candidate discipline, determining the common nouns of the target discipline and each candidate discipline respectively; Based on the distribution of the meanings of the common nouns in the target discipline and the candidate disciplines, the similarity of the meanings of the nouns in the target discipline and the candidate disciplines is determined.

2. The corpus construction method according to claim 1, characterized in that: The step of determining the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure based on the intersection level of the target discipline and each candidate discipline includes: Based on the hierarchical weights corresponding to the intersection levels of the target discipline and each candidate discipline, determining the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure; The level weights are determined based on the number of disciplines contained in each level in the discipline hierarchy.

3. The corpus construction method according to claim 1, characterized in that: The determining of the similarity of the meanings of the common nouns in the target discipline and the candidate disciplines based on the distribution of the meanings of the common nouns in the target discipline and the candidate disciplines respectively includes: Based on the distribution of the meanings of the common nouns in the target discipline and the candidate disciplines, and the relationship between the common meanings of the common nouns in the target discipline and the candidate disciplines, determining the similarity of the meanings of the nouns in the target discipline and the candidate disciplines; The common meaning is the most frequent meaning of the common noun in the corresponding subject.

4. The corpus construction method according to any one of claims 1 to 3, characterized in that: The selecting of related disciplines related to the target discipline from each candidate discipline based on the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure and / or the similarity of the meanings of nouns in the target discipline and each candidate discipline, includes: Determine the subject score of each candidate subject based on the corpus size of each candidate subject, the intersection of the target subject and each candidate subject in the subject hierarchy, and / or the similarity of the meaning of nouns in the target subject and each candidate subject; Based on the subject score of each candidate subject, an associated subject related to the target subject is selected from each candidate subject.

5. The corpus construction method according to claim 4, characterized in that: The step of constructing a corpus based on the corpus of the target subject and the corpus of the related subject includes: Determining the associated corpus volume of the associated subject based on the subject score of the associated subject; A corpus is constructed based on the corpus of the target subject and the corpus of the associated subjects under the associated corpus volume.

6. The corpus construction method according to claim 5, characterized in that: The step of constructing a corpus based on the corpus of the target subject and the corpus of the associated subject under the associated corpus volume includes: Storing the corpus of the target subject into a corpus; With the goal of making the sum of sentiment scores neutral, based on the sentiment scores of each corpus in the related disciplines, corpora of related disciplines within the related corpus volume are selected and stored in the corpus.

7. A corpus construction device, characterized in that: include: The subject determination unit is used to determine the target subject; A correlation determination unit, configured to select a correlation subject related to the target subject from each candidate subject based on the intersection of the target subject and each candidate subject in the subject hierarchy structure and / or the similarity of the meanings of nouns in the target subject and each candidate subject respectively; A construction unit, used for constructing a corpus based on the corpus of the target subject and the corpus of the related subject; The intersection of the target discipline and each candidate discipline in the discipline hierarchy is determined based on the following steps: Based on the subject hierarchy structure, determining the intersection level of the target subject and each candidate subject; Based on the intersection level of the target discipline and each candidate discipline, determining the intersection of the target discipline and each candidate discipline in the discipline hierarchy structure; The similarity of the meaning of the noun in the target discipline and each candidate discipline is determined based on the following steps: Based on the noun set of the target discipline and the noun sets of each candidate discipline, determining the common nouns of the target discipline and each candidate discipline respectively; Based on the distribution of the meanings of the common nouns in the target discipline and the candidate disciplines, the similarity of the meanings of the nouns in the target discipline and the candidate disciplines is determined.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the corpus construction method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the corpus construction method according to any one of claims 1 to 6 is implemented.

Images