Large-scale corpus processing method and system

Abstract

The invention discloses a large-scale corpus processing method and system. The method comprises the following steps: acquiring associated initial corpus data based on a plurality of subject terms; converting the initial corpus data according to a preset coding format; processing the converted initial corpus data to obtain single effective corpus data and storing the single effective corpus data; performing language classification annotation on the single effective corpus data according to a preset classification standard to obtain corresponding language classification annotation data; based on the language classification annotation data, utilizing a chi-square test method to analyze relevance among the multiple subject terms; and carrying out batch processing on the associated corpus data based on the relevance among the plurality of subject terms. By the adoption of the large-scale corpus processing method and system in the technical scheme, large-scale corpus data obtained through retrieval can be processed in batches on the basis of relevance among the subject terms, the processing speed is increased, and meanwhile the accuracy of the processing result can be guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and information technology, and in particular to a large-scale corpus processing method and system. Background Art

[0002] When searching for a specific word in a large corpus, the resulting corpus data is often quite large. Manual screening of this regular data is a massive, time-consuming, and inefficient task, hindering research progress. Prior art methods for using artificial intelligence to screen data primarily rely on data filtering. However, this method suffers from over-reliance on the quality of the input data, the potential for erroneous deletions, and the inability to effectively guarantee the accuracy of the screening results. Summary of the Invention

[0003] In response to the shortcomings of the existing technology, this application proposes a large-scale corpus processing method and system, which processes large-scale, regular corpus data based on the correlation between retrieval keywords, while improving the processing speed and ensuring the accuracy of the processing results.

[0004] According to a first aspect of the present application, a large-scale corpus processing method is provided, comprising: Obtaining associated initial corpus data based on multiple subject words; Converting the initial corpus data into a predetermined encoding format; Processing the converted initial corpus data to obtain and store single valid corpus data; Performing language classification annotation on the single valid corpus data according to a predetermined classification standard to obtain corresponding language classification annotation data; Based on the language classification annotation data, using the chi-square test method to analyze the correlation between the plurality of subject words; The associated corpus data is batch processed based on the relevance between the multiple subject words.

[0005] In the above method, the multiple keywords have a predetermined correlation.

[0006] In the above method, the step of processing the converted initial corpus data to obtain a single valid corpus data includes: judging whether the words appearing before and after the subject word have research value; if so, judging the corresponding initial corpus data as valid corpus data, and deduplicating the valid corpus data; if not, judging the corresponding initial corpus data as invalid corpus data, and deleting the invalid corpus data.

[0007] In the above method, before performing deduplication processing on the valid corpus data, the method further includes: determining the degree of repetition of the valid corpus data based on characters and semantics.

[0008] The above method further includes: for the single valid corpus data, marking the words appearing before and after the subject word, and counting the frequency of occurrence of the word to obtain corresponding statistical data.

[0009] According to a second aspect of the present application, a large-scale corpus processing system is provided, comprising: An initial corpus data acquisition module is used to acquire associated initial corpus data based on multiple keywords; A format conversion module, configured to convert the initial corpus data into a predetermined encoding format; A single valid corpus data acquisition module is used to process the converted initial corpus data to obtain and store single valid corpus data; A language classification annotation data acquisition module is used to perform language classification annotation on the single valid corpus data according to a predetermined classification standard to obtain corresponding language classification annotation data; A relevance test module, configured to analyze the relevance between the plurality of subject words using a chi-square test method based on the language classification annotation data; The batch processing module performs batch processing on the associated corpus data based on the relevance between the multiple subject words.

[0010] In the above system, the single valid corpus data acquisition module includes a value judgment unit, a corpus deduplication unit and a corpus deletion unit; The value judgment unit is used to judge whether the words appearing before and after the subject word have research value; if so, the corresponding initial corpus data is judged to be valid corpus data; if not, the corresponding initial corpus data is judged to be invalid corpus data; The corpus deduplication unit is used to perform deduplication processing on the valid corpus data; The corpus deletion unit is used to delete the invalid corpus data.

[0011] In the above system, a word tagging module and a frequency statistics module are also included; The word marking module is used to mark the words appearing before and after the subject word for the single valid corpus data, and distinguish the importance levels; The frequency statistics module is used to count the frequencies of occurrence of words before and after the subject word for the single valid corpus data to obtain corresponding statistical data and store the statistics.

[0012] According to a third aspect of the present application, a terminal is provided, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the large-scale corpus processing method described above when running the computer program.

[0013] According to a fourth aspect of the present application, a computer-readable storage medium is provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the terminal where the storage medium is located is controlled to execute the above-mentioned large-scale corpus processing method.

[0014] The technical solution provided by this application has at least the following beneficial effects: multiple subject terms are searched in a corpus to obtain their associated initial corpus data, and the initial corpus data is converted into a format acceptable to the corpus processing system according to a predetermined encoding format, thereby minimizing the risk of data omission. The converted initial corpus data is processed, invalid corpus data with no research value and duplicate valid corpus data are deleted, and only one identical valid corpus data is retained. In this way, the space occupied by the corpus data can be effectively reduced and the efficiency of subsequent analysis and processing can be improved. The retained single valid corpus data is classified and annotated according to predetermined classification standards, and the field is subdivided to make subsequent processing more accurate. Based on the style classification and annotated data, the correlation between multiple subject terms is analyzed using the chi-square test method. The analysis results are more effective and intuitive, and facilitate subsequent batch operations. Using the large-scale corpus processing method and processing system in the above technical solution, large-scale corpus data obtained by retrieval can be batch processed based on the correlation between multiple subject terms, while improving processing speed and ensuring the accuracy of processing results.

[0015] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which: Figure 1 A flowchart of a large-scale corpus processing method provided in an embodiment of the present application; Figure 2 A structural block diagram of a large-scale corpus processing system provided in an embodiment of the present application; Figure 3 This is a structural block diagram of a single valid corpus data acquisition module provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] In the description of an application, "several" means one or more, "more" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed to indicate or imply relative importance, or to implicitly specify the number or order of the technical features indicated.

[0019] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the order in the flowchart.

[0020] In this application, the corpus data with regularity is described as follows: For corpus data retrieved from a search for a keyword, the words that appear before and after the keyword must adhere to the grammatical rules, requirements, and characteristics of the language itself, and must not be disordered. For example, if we study the adverb "gradually" as a keyword, the words that appear before and after "gradually" have certain characteristics and attributes, indicating that the corpus data has regularity.

[0021] like Figure 1 As shown, for the above-mentioned corpus data with regularity, the first embodiment of the present application provides a large-scale corpus processing method, including the following steps: S110 : Acquire associated initial corpus data based on multiple keywords.

[0022] For example, taking the two keywords "gradually" and "gradually" as examples, the two keywords "gradually" and "gradually" are searched separately in professional corpora such as the Peking University CCL corpus. Accordingly, a large amount of relevant initial corpus data will appear. In the initial corpus data, the words appearing before and after "gradually" may be the same as or different from the words appearing before and after "gradually". In this application, multiple keywords must have a predetermined correlation, which can be a semantic correlation, such as synonyms, antonyms, or a hyponym relationship. This predetermined correlation helps to improve the efficiency and accuracy of corpus processing.

[0023] S120: Convert the initial corpus data according to a predetermined encoding format.

[0024] Exemplarily, the predetermined encoding format refers to an encoding format recognized by the large-scale corpus processing system, such as the UTF-8 encoding format. The UTF-8 encoding format is compatible with nearly all Unicode characters, including letters, symbols, and numbers from various languages. This means that when processing multilingual corpus data, UTF-8 encoding can ensure that characters from different languages ​​are correctly represented and stored, preventing garbled characters or missing characters. For example, in a corpus containing multiple languages ​​such as Chinese, English, and French, using UTF-8 encoding ensures that characters from all languages ​​are accurately recorded and processed, providing strong support for cross-language natural language processing tasks. UTF-8 encoding also effectively supports corpus data containing special characters or symbols, such as mathematical formulas, chemical symbols, and musical notation, ensuring that these special characters are not incorrectly or lost during the conversion process, thereby ensuring the integrity and accuracy of the corpus data. Other encoding formats are also possible in this application and are not specifically limited here. The initial corpus data is converted according to a predetermined encoding format to unify its format, eliminate subsequent misjudgments caused by format differences, and ensure the accuracy of subsequent processing.

[0025] S130: Process the converted initial corpus data to obtain and store single valid corpus data.

[0026] Exemplarily, the process of processing the converted initial corpus data mainly includes: judging whether the words appearing before and after the two subject words "gradually" and "gradually" have research value; if there is no research value or there is interference, the corpus data is considered to be invalid corpus data, and it is deleted and the invalid corpus data is transferred to the deletion library. If there is research value, the corpus data is considered to be valid corpus data, and the valid corpus data is deduplicated, and the duplicate valid corpus data is deleted, and the data finally stored is a single valid corpus data. It should be noted that the invalid corpus data that enters the deletion library can be recalled as needed later, while the deleted duplicate valid corpus data does not enter the deletion library and cannot be restored.

[0027] For example, the validity of corpus data is closely related to the research value of the words that appear before and after the subject word. When judging whether a piece of corpus data is valid corpus data, multiple situations need to be considered. If the words that appear before and after the subject word have research value, this piece of corpus data is valid corpus data; if one of the words that appear before and after the subject word has research value, and the other has no research value, this piece of corpus data is still valid corpus data. This is because even if only one word has research value, it can still provide certain semantic or grammatical information for the subject word, which is helpful for studying the relevant content of the subject word. Only when the words that appear before and after the subject word have no research value, this piece of corpus data is invalid corpus data.

[0028] For example, under the premise of a unified encoding format for valid corpus data, before deduplication, the degree of duplication of the valid corpus data must be determined. This determination method includes, but is not limited to: if two valid corpus data come from the same source and are completely identical at the character level, i.e., every character is the same, then they can be determined to be duplicates. For example, "become a trustworthy person" and "become a trustworthy person" come from the same corpus and have exactly the same characters, thus being considered duplicates. If two valid corpus data come from the same source and have a long identical character substring, such as a continuous character substring exceeding a certain length threshold (e.g., 50 characters), they can be considered duplicates. If two valid corpus data come from the same source but have different characters but express the same or similar meaning, the degree of semantic similarity can be used to determine whether they are duplicates. A semantic similarity of 90% or higher is considered highly similar and thus duplicates. For example, "I like to eat apples" and "Apples are my favorite fruit" have similar semantics but different characters, thus being considered duplicates.

[0029] For example, if two valid corpus data have different sources, the duplication degree determination method described above for valid corpus data with the same source can be used. Furthermore, the high similarity in the background information, cited materials, etc. of the two valid corpus data can be used as a reference for duplication. For example, if two valid corpus data both quote content from the same report and the content is highly similar, then the two valid corpus data are considered duplicates.

[0030] For example, under the premise of a unified encoding format for valid corpus data, manual review and judgment can be used, or some tool algorithms can be used to assist in determining whether the valid corpus data is repeated, such as hash algorithms, database deduplication algorithms, string similarity algorithms, semantic similarity algorithms, etc. It is also possible to directly call a pre-trained duplication judgment model, combine the tool algorithm with the manual review and judgment method, and achieve all-round duplication judgment to avoid omissions, further improve the accuracy of duplication judgment, and avoid misjudgment.

[0031] It should be noted that the above-mentioned tool algorithms each have their own characteristics and are suitable for different scenarios and needs. In practical applications, appropriate tool algorithms can be selected based on the characteristics and needs of specific data to assist in determining the duplication of valid corpus data. The specific method used is not specifically limited in this application.

[0032] S140: Perform language classification annotation on the single valid corpus data according to a predetermined classification standard to obtain corresponding language classification annotation data.

[0033] For example, the single valid corpus data obtained after deduplication processing is subjected to genre classification and annotation according to predetermined classification standards, such as the six-category genre classification standard (daily conversation genre, business conversation genre, formal monologue genre, artistic formal genre, practical formal genre, and scientific and elegant genre). For example, the initial corpus data for the two subject terms "gradually" and "gradually" retrieved from the Peking University CCL corpus is processed through steps S110, S120, and S130, and then classified and annotated, resulting in 24,097 and 343 genre classification and annotation data, respectively, as shown in Table 1.

[0034] Table 1

[0035] For example, on the premise of fully understanding the characteristics of these six types of language styles, manual classification and annotation can be performed, or some tool algorithms can be used to assist in language style classification and annotation, such as contrastive learning enhancement algorithm, knowledge graph enhancement algorithm, small sample learning algorithm, multimodal fusion algorithm, etc. It is also possible to directly call the pre-trained classification and annotation model, combine the tool algorithm with manual classification and annotation methods, achieve all-round classification, avoid omissions, and further improve the accuracy of classification and annotation.

[0036] It should be noted that each of the above tools and algorithms has its own characteristics and is suitable for different scenarios and needs. In practical applications, appropriate tools and algorithms can be selected based on the characteristics and needs of specific data to assist in determining the language classification and annotation. The specific method used is not specifically limited in this application.

[0037] S150. Based on the language classification annotation data, the chi-square test method is used to analyze the correlation between multiple subject words.

[0038] For example, based on the language classification annotation data in Table 1, the chi-square test method is used to analyze the correlation between the two subject words "gradually" and "gradually", and the test results are shown in Table 2.

[0039] Table 2

[0040] In Table 2, df represents the degree of freedom, P represents the probability, and ━━ represents that the data amount is too small and has no statistical significance.

[0041] Table 2 shows that the differentiation values ​​for the two key words "gradually" and "gradually" are relatively large in both the artistic and practical formal styles, indicating a significant differentiation between the two. Specifically, the two key words "gradually" and "gradually" are less correlated and more independent in these two styles. In contrast, the differentiation values ​​for the two key words "gradually" and "gradually" are relatively small in the scientific and elegant style, indicating a relatively strong correlation and less independence in this style. In the formal monologue style, the correlation between the two key words "gradually" and "gradually" is even greater.

[0042] For example, the chi-square test is a statistical hypothesis test used to determine whether two categorical or nominal variables are likely to be correlated. In this application, the chi-square test is used to analyze the correlation between the two subject words "gradually" and "gradually". Through statistical inference and effect quantification, the differences in the distribution of the language styles of "gradually" and "gradually" and the strength of their correlation are revealed, providing data support for linguistic research. The specific steps include (1) constructing a contingency table, as shown in Table 3.

[0043] Table 3

[0044] Among them, R i (i=1,2,3…6) represents the total frequency of row i; C j (j=1,2) represents the total frequency of the jth column; N represents the total sample size; O ij Indicates the observation frequency.

[0045] (2) Assumptions: Independent hypothesis H0: The distribution of the subject words "gradually" and "gradually" has nothing to do with the language category; ‌Association hypothesis H1: There is a significant correlation between the distribution of the keywords "gradually" and "gradually".

[0046] (3) Calculate expected frequency Under the independence assumption, calculate the expected frequency E of each cell ij , the formula is: ; Calculate the chi-square value using the formula: ; Determine the degrees of freedom df, the formula is: Degrees of freedom df = (number of rows - 1) × (number of columns - 1) = (6 - 1) × (2 - 1) = 5; Based on the chi-square distribution table or statistical software, find the chi-square critical value corresponding to the degrees of freedom and significance level threshold α, or directly calculate the p-value (which represents the probability of the currently observed data occurring under the premise that the independence hypothesis H0‌ is true).

[0047] It should be noted that in this application, the degree of freedom df of each of the six types of language in Table 2 is calculated separately to analyze the correlation between two subject words. Therefore, the degree of freedom df of the language is 1.

[0048] (4) Determine the correlation: If the calculated chi-square value is greater than the chi-square critical value, or the p-value is less than α, the independence hypothesis H0 is rejected, indicating that there is a significant correlation between the distribution of the keywords "gradually" and "gradually".

[0049] Quantify the strength of the association: Use effect size metrics such as Cramer's V to quantify the strength of the association. The formula for calculating Cramer's V is: ; Where: V ranges from [0,1], the larger the value of V, the stronger the correlation; X 2 represents the chi-square value; N represents the total sample size; K represents the number of rows in the contingency table; r represents the number of columns in the contingency table.

[0050] Residual analysis: We conducted standardized residual analysis (corresponding to the differentiation values ​​in Table 2) to identify the specific language categories that led to significance. The formula for calculating the standardized residual AR is:

[0051] The larger the residual value AR is, the greater the difference is and the smaller the correlation is.

[0052] S160: Batch process the associated corpus data based on the relevance between multiple subject words.

[0053] For example, after obtaining the relevance of the two keywords "gradually" and "gradually", the corpus data related to the two keywords can be batch processed according to the relevance, such as batch deletion or saving.

[0054] In some specific embodiments of the present application, the large-scale corpus processing method also includes: marking the words appearing before and after the subject word in the single valid corpus data obtained after deduplication processing to indicate the importance of the word to the research, and the marking includes three levels: "important", "secondary", and "general".

[0055] In some specific embodiments of the present application, the large-scale corpus processing method further includes: for the single valid corpus data obtained after deduplication processing, counting the frequency of each word appearing before and after each subject word to obtain corresponding statistical data and storing it.

[0056] For example, for the single valid corpus data of the subject words "gradually" and "gradually", the words that appear before "gradually", such as "power" and "dignity", are counted, which appear 150 times and 3 times respectively; the words that appear after "gradually", such as "progress" and "reduce", are counted, which appear 114 times and 4044 times respectively; the words that appear before "gradually", such as "power" and "again", are counted, which appear 1 time and 12 times respectively; the words that appear after "gradually", such as "improvement" and "lose", are counted, which appear 114 times and 8 times respectively; the position and frequency information of the words "power", "dignity", "progress", "reduce", "power", "again", "improvement" and "lose" are stored for subsequent research.

[0057] For example, the frequency of each word appearing before and after each keyword can be counted manually, or with the help of some tool algorithms, such as using jieba, HanLP, LTP or StanfordNLP for word segmentation, followed by data cleaning, database import, SQL query, etc., and the pre-trained statistical model can be directly called to combine the tool algorithm with the manual statistical method to achieve all-round statistics, avoid omissions, and further improve the accuracy of frequency statistics.

[0058] It should be noted that the above-mentioned tool algorithms each have their own characteristics and are suitable for different scenarios and needs. In practical applications, appropriate tool algorithms can be selected to assist in statistics based on the characteristics and needs of specific data. The specific method used is not specifically limited in this application.

[0059] like Figure 2 As shown, based on the above-mentioned large-scale corpus processing method, the second embodiment of the present application also provides a large-scale corpus processing system, including an initial corpus data acquisition module 210, a format conversion module 220, a single valid corpus data acquisition module 230, a style classification annotation data acquisition module 240, a relevance verification module 250 and a batch processing module 260.

[0060] in, An initial corpus data acquisition module 210 is used to acquire associated initial corpus data based on multiple keywords; The format conversion module 220 is used to convert the initial corpus data into a predetermined encoding format; A single valid corpus data acquisition module 230 is used to process the converted initial corpus data to obtain and store single valid corpus data; The language classification annotation data acquisition module 240 is used to perform language classification annotation on a single valid corpus data according to a predetermined classification standard to obtain corresponding language classification annotation data; A relevance test module 250 is used to analyze the relevance between multiple subject words using a chi-square test method based on the language classification annotation data; The batch processing module 260 performs batch processing on the associated corpus data based on the relevance between multiple keywords.

[0061] In some specific embodiments of the present application, Figure 3 As shown, the single valid corpus data acquisition module 230 includes a value judgment unit 231 , a corpus deduplication unit 232 and a corpus deletion unit 233 .

[0062] in, The value judgment unit 231 is used to judge whether the words appearing before and after the subject word have research value; if so, the corresponding initial corpus data is determined to be valid corpus data; if not, the corresponding initial corpus data is determined to be invalid corpus data; A corpus deduplication unit 232 is used to perform deduplication processing on valid corpus data; The corpus deletion unit 233 is used to delete invalid corpus data.

[0063] In some specific embodiments of the present application, Figure 2 As shown, the large-scale corpus processing system further includes a word tagging module 270 and a frequency statistics module 280 .

[0064] in, The word marking module 270 is used to mark the words that appear before and after the subject word for a single valid corpus data, distinguish the importance levels, and indicate the importance of the word to the research. The marking includes three levels: "important", "secondary", and "general".

[0065] The frequency statistics module 280 is used to count the frequencies of occurrence of words before and after a subject word in a single valid corpus data to obtain corresponding statistical data and store the statistics.

[0066] In some specific embodiments of the present application, Figure 2As shown, the large-scale corpus processing system also includes a data display module 290 for displaying single valid corpus data containing the keyword, the labeled data, and related statistical data. The display format can be to provide the single corpus data, labeled data, and related statistical data to the user in PDF or Excel format, or to view it online.

[0067] In some specific embodiments of the present application, the large-scale corpus processing system further includes a user login module for providing a user login interface for entering a user's personal ID and password to facilitate access to the system. A user must first register for their first login, using their personal email address as their personal ID and a custom password as their unique identifier.

[0068] By adopting the processing method and processing system in the above technical solution, large-scale corpus data obtained through retrieval can be batch processed based on the correlation between multiple topics, which can improve the processing speed while ensuring the accuracy of the processing results.

[0069] The large-scale corpus processing method and system in the above technical solution can serve language research. At the same time, it can also provide methods and theoretical basis for the application of artificial intelligence in language research and in the deep mining, analysis and application of large-scale corpus.

[0070] The third embodiment of the present application further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the above-mentioned large-scale corpus processing method when running the computer program.

[0071] Specifically, a processor may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. A processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0072] Specifically, the processor is connected to the memory via a bus. The bus may include pathways for transmitting information. The bus may be a PCI bus or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like.

[0073] The memory may be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an EEPROM, CD-ROM or other optical disk storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0074] Optionally, the memory is used to store the computer program code for executing the solution of the present application, and the processor controls the execution. The processor is used to execute the application program code stored in the memory to implement the actions of the above-mentioned large-scale corpus processing system.

[0075] The fourth aspect of the present application also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed by the processor, the terminal where the storage medium is located is controlled to execute the above-mentioned large-scale corpus processing method.

[0076] Those skilled in the art will appreciate that all or some of the steps and systems disclosed above can be implemented as software, firmware, hardware, or any suitable combination thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVDs) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0077] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A large-scale corpus processing method, characterized in that: include: Obtaining associated initial corpus data based on multiple subject words; Converting the initial corpus data into a predetermined encoding format; Processing the converted initial corpus data to obtain and store single valid corpus data; Performing language classification annotation on the single valid corpus data according to a predetermined classification standard to obtain corresponding language classification annotation data; Based on the language classification annotation data, using the chi-square test method to analyze the correlation between the plurality of subject words; The associated corpus data is batch processed based on the relevance between the multiple subject words.

2. The corpus processing method according to claim 1, characterized in that: The multiple keywords have a predetermined correlation.

3. The corpus processing method according to claim 1, characterized in that: The step of processing the converted initial corpus data to obtain a single valid corpus data includes: Determine whether the words appearing before and after the subject term have research value; If so, the corresponding initial corpus data is determined to be valid corpus data, and duplicate removal is performed on the valid corpus data; If not, the corresponding initial corpus data is determined to be invalid corpus data, and the invalid corpus data is deleted.

4. The corpus processing method according to claim 3, characterized in that: Before performing deduplication processing on the valid corpus data, the method further includes: The degree of repetition of the valid corpus data is determined based on characters and semantics.

5. The large-scale corpus processing system according to claim 1, characterized in that: Also includes: For the single valid corpus data, the words appearing before and after the subject word are marked, and the frequency of occurrence of the word is counted to obtain corresponding statistical data.

6. A large-scale corpus processing system, characterized in that: include: An initial corpus data acquisition module is used to acquire associated initial corpus data based on multiple keywords; A format conversion module, configured to convert the initial corpus data into a predetermined encoding format; A single valid corpus data acquisition module is used to process the converted initial corpus data to obtain and store single valid corpus data; A language classification annotation data acquisition module is used to perform language classification annotation on the single valid corpus data according to a predetermined classification standard to obtain corresponding language classification annotation data; A relevance test module, configured to analyze the relevance between the plurality of subject words using a chi-square test method based on the language classification annotation data; The batch processing module performs batch processing on the associated corpus data based on the relevance between the multiple subject words.

7. The large-scale corpus processing system according to claim 6, characterized in that: The single valid corpus data acquisition module includes a value judgment unit, a corpus deduplication unit and a corpus deletion unit; The value judgment unit is used to judge whether the words appearing before and after the subject word have research value; if so, the corresponding initial corpus data is judged to be valid corpus data; if not, the corresponding initial corpus data is judged to be invalid corpus data; The corpus deduplication unit is used to perform deduplication processing on the valid corpus data; The corpus deletion unit is used to delete the invalid corpus data.

8. The large-scale corpus processing system according to claim 6, characterized in that: It also includes word tagging module and frequency counting module; The word marking module is used to mark the words appearing before and after the subject word for the single valid corpus data, and distinguish the importance levels; The frequency statistics module is used to count the frequencies of occurrence of words before and after the subject word for the single valid corpus data to obtain corresponding statistical data and store the statistics.

9. A terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the large-scale corpus processing method according to any one of claims 1 to 5 is executed.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the terminal where the storage medium is located is controlled to execute the large-scale corpus processing method according to any one of claims 1 to 5.

Images