Keyword weight analysis method and device for text data

Abstract

The invention belongs to the field of data analysis and mining, and discloses a keyword weight analysis method and device for text data, and the method comprises the steps: obtaining a to-be-processed task; determining a target data set and a common data set; performing word segmentation on the target data set and the common data set to obtain a plurality of first keywords of the target data set and a plurality of second keywords of the common data set; calculating a first keyword frequency of a target keyword in the plurality of first keywords; calculating a second keyword frequency of the target keyword; calculating an inverse document frequency of the target keyword in the common data set; and calculating the weight of the target keyword according to the first keyword frequency, the second keyword frequency and the inverse document frequency. According to the method, the weight calculation formula of the TF-IDF is redefined, that is, the number of the communication accounts serves as the weight calculation factor, the distinction degree of the weights of different keywords can be remarkably improved, and then the keyword weight accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the field of data analysis and mining, and in particular relates to a keyword weight analysis method and device for text data. Background Art

[0002] With the popularity of personal terminal devices, instant messaging software is increasingly used in daily work and life. In the process of using instant messaging software for communication, a large amount of text data is generated. For example, for the group chat function of instant messaging software, a large amount of group chat text data is generated during the group chat process. Therefore, the analysis and mining of text data is becoming more and more complicated.

[0003] Currently, the most widely used keyword extraction algorithm is TF IDF (term frequency-inverse document frequency). TF IDF is a typical bag-of-words method that ignores the order, grammar and syntax of words and only regards the text as a collection of several words, each of which is independent. This results in a low accuracy of the extracted keywords and a low accuracy rate of the calculated keyword weights. Summary of the invention

[0004] The purpose of the present invention is to improve the accuracy of keyword weights of text data.

[0005] In a first aspect, an embodiment of the present invention provides a keyword weight analysis method for text data, the method comprising:

[0006] Obtaining a task to be processed; the task to be processed is used to indicate: analyzing the weights corresponding to keywords of target text data of a target communication account;

[0007] Determine a target data set including the target text data, and determine a common data set based on the text data of multiple communication accounts; the multiple communication accounts and the target communication account belong to the same instant messaging software;

[0008] Segmenting the target data set and the common data set to obtain a first total number of a plurality of first keywords in the target data set and a first number of occurrences respectively corresponding to the plurality of first keywords; and a second total number of a plurality of second keywords in the common data set and a second number of occurrences respectively corresponding to the plurality of second keywords;

[0009] For any target keyword among the plurality of first keywords, based on a first number of occurrences corresponding to the target keyword and the first total number, a first keyword frequency of the target keyword is calculated; and based on a second number of occurrences corresponding to the target keyword in the common data set and the second total number, a second keyword frequency of the target keyword is calculated;

[0010] Determine the total number of the plurality of communication accounts in the general data set as the first number of accounts, and the number of the communication accounts whose text data includes the target keyword as the second number of accounts;

[0011] The inverse document frequency of the target keyword in the common data set is calculated according to the first number of accounts and the second number of accounts; and the weight of the target keyword is calculated according to the first keyword frequency, the second keyword frequency and the inverse document frequency.

[0012] Optionally, determining a target data set including the target text data includes:

[0013] Acquire the target text data, and determine the target text data as a target data set;

[0014] or,

[0015] The target text data and associated text data associated with the target text data in other communication accounts are obtained; and the target text data and the associated text data are determined as a target data set.

[0016] Optionally, the acquiring the target text data and associated text data associated with the target text data in other communication accounts; and determining the target text data and the associated text data as a target data set includes:

[0017] Acquire the target text data, and analyze the first content type of the target text data;

[0018] Acquire text data of other communication accounts, and analyze the second content type of the text data of the other communication accounts;

[0019] Calculating the similarity between the first content type and the second content type;

[0020] When the similarity is greater than a preset similarity, the text data of the other communication account is determined as associated text data of the target text data, and the target text data and the associated text data are determined as a target data set.

[0021] Optionally, after segmenting the target data set and the common data set, the method further includes:

[0022] Obtaining a first word segmentation result of the target data set and a second word segmentation result of the common data set;

[0023] Determine whether there are keywords satisfying source text filtering rules in the first word segmentation result and the second word segmentation result respectively; the source text filtering rules include at least one of the following rules: filtering keywords whose source text length is greater than a preset length, filtering keywords whose source text format is a preset format, and filtering keywords whose source text is a link;

[0024] If there are keywords satisfying the source text filtering rule in the first word segmentation result, deleting the keywords satisfying the source text filtering rule from the first word segmentation result;

[0025] When the second word segmentation result contains keywords satisfying the source text filtering rule, the keywords satisfying the source text filtering rule are deleted from the second word segmentation result.

[0026] Optionally, after segmenting the target data set and the common data set, the method further includes:

[0027] Obtaining a first word segmentation result of the target data set and a second word segmentation result of the common data set;

[0028] Determine whether there are keywords satisfying keyword filtering rules in the first word segmentation result and the second word segmentation result respectively; the keyword filtering rules include at least one of the following rules: filtering keywords with consecutive identical characters, filtering keywords with a preset part of speech, and filtering keywords with a preset type of content;

[0029] If there are keywords satisfying the keyword filtering rule in the first word segmentation result, deleting the keywords satisfying the keyword filtering rule from the first word segmentation result;

[0030] When there are keywords satisfying the keyword filtering rule in the second word segmentation result, the keywords satisfying the keyword filtering rule are deleted from the second word segmentation result.

[0031] Optionally, the method further includes:

[0032] After calculating the weight of each first keyword in the target data set, generating a weight result table; the weight result table includes each first keyword and the weight of each first keyword;

[0033] Analyze the weight result table to obtain summary keywords with weights greater than preset weights;

[0034] An account summary of the target communication account is generated using the summary keywords.

[0035] In a second aspect, an embodiment of the present invention provides a device for analyzing keyword weights of text data, the device comprising:

[0036] A task acquisition module is used to acquire tasks to be processed; the tasks to be processed are used to indicate: analyzing the weights corresponding to the keywords of the target text data of the target communication account;

[0037] A data set determination module, used to determine a target data set including the target text data, and determine a common data set based on text data of a plurality of communication accounts; the plurality of communication accounts and the target communication account belong to the same instant messaging software;

[0038] a data segmentation module, configured to segment the target data set and the common data set to obtain a first total number of a plurality of first keywords in the target data set and a first number of occurrences corresponding to the plurality of first keywords respectively; and a second total number of a plurality of second keywords in the common data set and a second number of occurrences corresponding to the plurality of second keywords respectively;

[0039] a keyword frequency calculation module, configured to calculate, for any target keyword among the plurality of first keywords, a first keyword frequency of the target keyword based on a first number of occurrences corresponding to the target keyword and the first total number; and to calculate a second keyword frequency of the target keyword based on a second number of occurrences corresponding to the target keyword in the common data set and the second total number;

[0040] an account number determination module, configured to determine the total number of the plurality of communication accounts in the general data set as the first account number, and the number of the communication accounts whose text data includes the target keyword as the second account number;

[0041] A weight calculation module is used to calculate the inverse document frequency of the target keyword in the common data set according to the first number of accounts and the second number of accounts; and to calculate the weight of the target keyword according to the first keyword frequency, the second keyword frequency and the inverse document frequency.

[0042] Optionally, the data set determination module is specifically used to:

[0043] Acquire the target text data, and determine the target text data as a target data set;

[0044] or,

[0045] The target text data and associated text data associated with the target text data in other communication accounts are obtained; and the target text data and the associated text data are determined as a target data set.

[0046] Optionally, the data set determination module is specifically used to:

[0047] Acquire the target text data, and analyze the first content type of the target text data;

[0048] Acquire text data of other communication accounts, and analyze the second content type of the text data of the other communication accounts;

[0049] Calculating the similarity between the first content type and the second content type;

[0050] When the similarity is greater than a preset similarity, the text data of the other communication account is determined as associated text data of the target text data, and the target text data and the associated text data are determined as a target data set.

[0051] Optionally, the device further includes a keyword filtering module, specifically configured to:

[0052] After segmenting the target data set and the common data set, obtaining a first segmentation result of the target data set and a second segmentation result of the common data set;

[0053] Determine whether there are keywords satisfying source text filtering rules in the first word segmentation result and the second word segmentation result respectively; the source text filtering rules include at least one of the following rules: filtering keywords whose source text length is greater than a preset length, filtering keywords whose source text format is a preset format, and filtering keywords whose source text is a link;

[0054] If there are keywords satisfying the source text filtering rule in the first word segmentation result, deleting the keywords satisfying the source text filtering rule from the first word segmentation result;

[0055] When the second word segmentation result contains keywords satisfying the source text filtering rule, the keywords satisfying the source text filtering rule are deleted from the second word segmentation result.

[0056] Optionally, the keyword filtering module is further specifically used for:

[0057] After segmenting the target data set and the common data set, obtaining a first segmentation result of the target data set and a second segmentation result of the common data set;

[0058] Determine whether there are keywords satisfying keyword filtering rules in the first word segmentation result and the second word segmentation result respectively; the keyword filtering rules include at least one of the following rules: filtering keywords with consecutive identical characters, filtering keywords with a preset part of speech, and filtering keywords with a preset type of content;

[0059] If there are keywords satisfying the keyword filtering rule in the first word segmentation result, deleting the keywords satisfying the keyword filtering rule from the first word segmentation result;

[0060] When there are keywords satisfying the keyword filtering rule in the second word segmentation result, the keywords satisfying the keyword filtering rule are deleted from the second word segmentation result.

[0061] Optionally, the device further includes an account summary generation module, which is specifically used to:

[0062] After calculating the weight of each first keyword in the target data set, generating a weight result table; the weight result table includes each first keyword and the weight of each first keyword;

[0063] Analyze the weight result table to obtain summary keywords with weights greater than preset weights;

[0064] An account summary of the target communication account is generated using the summary keywords.

[0065] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0066] at least one processor;

[0067] a memory for storing the at least one processor-executable instruction;

[0068] The at least one processor is configured to execute the instructions to implement the method described in the first aspect.

[0069] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect.

[0070] In a fifth aspect, an embodiment of the present invention further provides a computer program product, characterized in that it comprises a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0071] In the technical solution of the embodiment of the present application, the keyword weight of the text data of the communication account is asynchronously executed in a task-driven manner, which reduces the dependence on computing power performance compared with the technical solution of traditional word frequency analysis. In addition, a target data set including target text data and a common data set are determined. For any keyword in the target data set, the keyword frequency of the keyword in the target data set and the keyword frequency in the common data set are calculated; and the number of communication accounts is used as a weight calculation factor, that is, the inverse document frequency is calculated by determining the number of communication accounts in which the keyword appears in the common text set and the total number of communication accounts corresponding to the common data set, and finally the weight of the keyword is calculated by the keyword frequency of the keyword in the target data set, the keyword frequency of the keyword in the common data set, and the inverse document frequency. It can be seen that the present invention can significantly improve the discrimination of the weights of different keywords by redefining the weight calculation formula of TF-IDF, that is, taking the number of communication accounts as a weight calculation factor, thereby improving the accuracy of keyword weights. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A schematic diagram of the overall technical solution of an embodiment of the present invention;

[0073] Figure 2 A flowchart of a keyword weight analysis method for text data provided by an embodiment of the present invention;

[0074] Figure 3 A flow chart of a method for determining a target data set provided by an embodiment of the present invention;

[0075] Figure 4 A structural block diagram of a keyword weight analysis device for text data provided by an embodiment of the present invention;

[0076] Figure 5 This is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The present invention will be described in detail below through examples.

[0078] With the popularity of personal terminal devices, instant messaging software is increasingly used in daily work and life. In the process of using instant messaging software for communication, a large amount of text data is generated. For example, for the group chat function of instant messaging software, a large amount of group chat text data is generated during the group chat process. Therefore, the analysis and mining of text data is becoming more and more complicated.

[0079] Currently, the most widely used keyword extraction algorithm is TF IDF (term frequency-inverse document frequency). TF IDF is a typical bag-of-words method that ignores the order, grammar and syntax of words and only regards the text as a collection of several words, each of which is independent. This results in a low accuracy of the extracted keywords and a low accuracy rate of the calculated keyword weights.

[0080] In order to describe the solution clearly, the target data set and the common data set involved in the solution of the present invention are first explained.

[0081] First, the target dataset is introduced.

[0082] In the present invention, text data can be aggregated through a variety of aggregation dimensions. Commonly used aggregation dimensions are: group account dimension and case dimension. For example, when analyzing the keyword weight of the target text data of a target communication account of an instant messaging software, the target data set may include not only the target text data of the target communication account, but also the text data associated with the target text data in other communication accounts. When judging whether the text data of other communication accounts is associated with the target text data, it can be judged through a variety of dimensions such as group account dimension (including group name, region, population category, age group, hobbies, etc.) and case dimension (including whether the theme of the text content is related, etc.).

[0083] Moreover, in the embodiment of the present invention, not only can the collection dimension be used to collect text data, but the size of the target data set can also be further controlled by the collection quantity. For example, the target data set can include text data associated with the target text data in multiple other accounts. After the text data is collected, since the keywords may be used repeatedly many times in the target data set, but the number of times they appear in the ordinary data set will be relatively small, at this time, the difference between the target data set and the ordinary data set will be reflected, so that the calculated keyword weight accuracy is higher.

[0084] Secondly, common data sets are introduced.

[0085] In terms of data selection, try to choose text data that has a low correlation with the text data of the target dataset as the data source for processing and importing.

[0086] In terms of data volume: the size of a normal data set usually needs to be a hundred times that of the target data set. The more text data the target data set has and the richer the content, the more distinguishable the weight calculation results of different keywords will be, and the more obvious the features will be.

[0087] Regarding the data import method: Since the data volume of ordinary data sets is usually large, a distributed processing framework can be used to process the data and generate it into the account word frequency table, which will be explained in the following embodiment.

[0088] Regarding the timing of data import: it can be imported in the form of scripts during the system installation phase. It can also be used as a continuously growing database to continuously expand the normal data set with incremental data.

[0089] Regarding data reusability: Since common data sets have a certain degree of universality, the data of common data sets can be packaged to generate data scripts for use in different system deployments.

[0090] After explaining the target data set and the common data set, the complete technical solution of the embodiment of the present invention will be described below in conjunction with specific application scenarios.

[0091] The technical solution of the embodiment of the present invention obtains the weights of keywords included in the text data by analyzing the text data of instant messaging software, wherein the instant messaging software may include chat software for chatting, including group chat and private chat; it may also include instant messaging software with more text data such as text messages, microblogs, and emails.

[0092] Taking group chat as an example, the weights corresponding to the keywords of the text data generated by the group chat are calculated.

[0093] It should be noted that the information of each communication account of the instant messaging software involved in the present invention (including but not limited to the device information of the communication account, personal information of the communication account, etc.) and related data (including but not limited to text data) are all information authorized by the communication account or authorized by all parties.

[0094] like Figure 1 FIG. 1 is a flow chart of the overall technical solution of an embodiment of the present invention. The process mainly includes the following steps:

[0095] Step 1: Create a word frequency analysis task. Specifically, create an analysis task in group chat units. That is, the word frequency analysis task is used to instruct to analyze the text data of the group chat to determine the weights of the keywords included in the text data of the group chat. The specific task table structure is shown in Table 1:

[0096] Table 1

[0097]

[0098] Step 2: Change the status to "processing". Specifically, scan the word frequency analysis task and change the status of the word frequency analysis task to "processing".

[0099] Specifically, it can be seen from Table 1 that the task table includes the status of pending tasks, and the status of pending tasks includes four categories, namely, pending; processing; processing completed; and processing failed. Before the pending task is executed, the status of the pending task is pending. After the pending task is executed, the status of the pending task is changed from the pending table to processing.

[0100] Step 3, clear historical data. Specifically, in order to support repeated analysis, each time a group chat is analyzed, the data in the account word frequency table obtained from the historical analysis of the text data of the group chat and the data in the weight result table are cleared. The group chat text data is analyzed again, and the results of the second analysis are updated to the account word frequency table and the weight result table. Table 2 shows the structure of the account word frequency table; Table 3 shows the results of the weight result table.

[0101] Table 2

[0102] Data Item type illustrate id String serial number cltId String Business Number gpAct String Group chat account act String Group chat account that sent the keyword wrd String Keywords cnt Long The number of times the keyword appears in the group chat

[0103] Table 3

[0104] Data Item type illustrate id String serial number tskId String Task Number wrd String Keywords score Double Weight score

[0105] Step 4: Batch query data. Specifically, the text data of the group chat is stored in a database. When analyzing the text data of the group chat, the text data can be queried from the database. For example, 5,000 pieces of data can be queried from the database each time as a target data set.

[0106] Step 5: Use the standard word segmenter of Hanlp, a powerful Chinese natural language processing library, to segment the text data of the group chat.

[0107] Step 6: Filter keywords using the following filtering rules. Specifically, according to the word segmentation results, the filtering rules can be flexibly adjusted to exclude some interference items to a certain extent. The present invention mainly provides two filtering rules.

[0108] The first type of filtering rules is source text filtering rules. Specifically, they are divided into the following three categories:

[0109] First, text length filtering: If there is too long text in the word segmentation content, it may be an advertisement, so it needs to be filtered out.

[0110] Second, source text format filtering: If there is text content in the xml, json and other formats in the word segmentation content, it means that it is not a keyword, so it should also be filtered out.

[0111] Third, source text link filtering: If the text content in the word segmentation content is a link, it means that it is not a keyword, so it should also be filtered out.

[0112] The second type of filtering rules is keyword filtering rules. Specifically, they are divided into the following three categories:

[0113] First, filter out consecutive identical characters. If there are consecutive identical characters in the word segmentation content, then the existence of these consecutive identical characters has no practical meaning, so they need to be filtered out.

[0114] Second, preset type content filtering. In actual applications, the preset type content can be set to email, mobile phone number, landline, ID number, etc. These types of content are usually not keywords, so they can be filtered out.

[0115] Third, part of speech filtering. For example, conjunctions, articles, prepositions, and pronouns are not keywords, so they can also be filtered out.

[0116] Step 7: Single batch pre-aggregation results: Specifically, after keyword filtering is performed on the word segmentation results, useful keywords are obtained. At this time, the number of keyword occurrences is accumulated for a single batch of 5,000 data.

[0117] Step 8: Accumulate and insert the account word frequency table. Update the single batch pre-aggregation results to the data in the above account word frequency table.

[0118] Step 9: Prevent the division by 0 problem. Specifically, the useful keywords of the target data set obtained in the above steps can be inserted into the common data set, wherein the number of insertions of each keyword can be 1. The common data set can be text data corresponding to other multiple group chats queried from the database, and the above steps 5 to 8 are also performed on the text data in the common data set.

[0119] 10. Calculate keyword weights. Use the TF-IDF algorithm to calculate the weight of each keyword. The relevant variables for calculating the weights come from the target dataset and the common dataset.

[0120] Specifically, the weight calculation method provided by the embodiment of the present invention is different from the method of calculating word weights by the TF-IDF algorithm in the related art. The technical solution of the present invention realizes the customization of TF-IDF. By adding customized calculation factors, the discrimination of keyword weight calculation can be increased. For example, for the same keyword, the weight in the target data set is very high, while the weight in the ordinary data set is very low. At this time, the weight of this keyword in the target data set can be subtracted from the weight in the ordinary data set, so as to calculate the new weight of this keyword. It can be seen that the new weight calculated in this way is more accurate.

[0121] The following will specifically describe how to calculate the weight of each keyword in the target data set. The following takes calculating the weight of a target keyword as an example for explanation. The target keyword is any one of the multiple keywords in the target data set.

[0122] x is the number of times the target keyword appears in the target data set; y is the number of times the target keyword appears in the common data set; M t is the total number of keywords in the target dataset; M c is the total number of keywords in the general data set. c is the total number of communication accounts corresponding to the common data set. That is to say, if the text data of N communication accounts is taken as the common data set, then A c N; A e is the number of communication accounts in which the target keyword appears in the general data set. That is to say, if the target keyword appears in the text data of n communication accounts among the N communication accounts, then A e is n.

[0123] TF(x) is the frequency of the target keyword in the target data set, and the calculation formula is:

[0124]

[0125] TF(y) is the frequency of the target keyword in the general data set, and the calculation formula is:

[0126]

[0127] IDF is the inverse document frequency of the target keyword in the common data set, and the calculation formula is:

[0128]

[0129] The weight calculation formula for the target keyword is:

[0130]

[0131] From the above description, it can be seen that the present invention can significantly improve the discrimination of weights of different keywords by redefining the weight calculation formula of TF-IDF, that is, taking the number of communication accounts as the weight calculation factor. For example, when the keyword frequencies of two keywords are very close, the two keywords can be calculated to have different weight values ​​by comparing the number of accounts that send the two keywords, which cannot be done in ordinary TF-IDF word frequency analysis technology.

[0132] 11. Write the calculated weights of each keyword into the result table. Save the weight calculation results of a single group chat into the result table for external application display.

[0133] As can be seen from the above description, in the technical solution of the embodiment of the present application, the keyword weight of the text data of the communication account is asynchronously executed in a task-driven manner, which reduces the dependence on computing power performance compared with the technical solution of traditional word frequency analysis. In addition, a target data set including target text data and a common data set are determined. For any keyword in the target data set, the keyword frequency of the keyword in the target data set and the keyword frequency in the common data set are calculated; and the number of communication accounts is used as a weight calculation factor, that is, the inverse document frequency is calculated by determining the number of communication accounts in which the keyword appears in the common text set and the total number of communication accounts corresponding to the common data set, and finally the weight of the keyword is calculated by the keyword frequency of the keyword in the target data set, the keyword frequency of the keyword in the common data set, and the inverse document frequency. It can be seen that the present invention can significantly improve the discrimination of the weights of different keywords by redefining the weight calculation formula of TF-IDF, that is, taking the number of communication accounts as a weight calculation factor, thereby improving the accuracy of keyword weights.

[0134] After the overall technical solution of the embodiment of the present invention is described in detail, a keyword weight analysis method for text data provided by the embodiment of the present invention will be described in detail below.

[0135] like Figure 2 As shown, a keyword weight analysis method for text data provided by an embodiment of the present invention may include the following steps:

[0136] S210, obtaining tasks to be processed.

[0137] The task to be processed is used to indicate: analyzing the weights corresponding to the keywords of the target text data of the target communication account.

[0138] In actual applications, which communication account's text data is to be analyzed, which communication account is used as the target communication account, and a corresponding task to be processed is established. After the task to be processed is established, the task to be processed can be obtained, and the following steps can be started. For example, an analysis task is created in a group chat unit. That is, the word frequency analysis task is used to indicate that the text data of the group chat is analyzed to determine the weights of the keywords included in the text data of the group chat. The task structure table is shown in the above embodiment and will not be repeated here.

[0139] S220, determining a target data set including target text data, and determining a common data set based on the text data of a plurality of communication accounts.

[0140] Among them, the multiple communication accounts and the target communication account belong to the same instant messaging software.

[0141] After creating the task to be analyzed, the target data set and the common data set are determined. The concepts of the target data set and the common data set are also described in the above embodiment and will not be repeated here. The following is an introduction to the method of determining the target data set.

[0142] As an implementation of the embodiment of the present invention, determining the target data set including the target text data may include the following two implementations:

[0143] In one implementation, determining a target data set including target text data may include the following steps:

[0144] Target text data is acquired, and the target text data is determined as a target data set.

[0145] In this implementation, the target text data can be directly used as the target data set. For example, Figure 1 In the illustrated embodiment, when analyzing text data of a group chat, 5000 pieces of data of the group chat can be directly obtained from the database as target text data, and the target text data is determined as the target data set.

[0146] In another embodiment, determining a target data set including target text data may include the following steps:

[0147] The target text data and the associated text data associated with the target text data in other communication accounts are obtained; and the target text data and the associated text data are determined as a target data set.

[0148] In the present invention, text data can be collected through multiple collection dimensions. Commonly used collection dimensions are: group account dimension and case dimension. For example, when analyzing the keyword weight of the target text data of a target communication account of an instant messaging software, the target data set may include not only the target text data of the target communication account, but also the text data associated with the target text data in other communication accounts. When judging whether the text data of other communication accounts is associated with the target text data, it can be judged through multiple dimensions such as group account dimension (including group name, region, population category, age group, hobbies, etc.) and case dimension (including whether the theme of the text content is related, etc.). In addition, the size of the target data set can be further controlled by the number of collections. For example, the target data set can include text data associated with the target text data in multiple other accounts.

[0149] As an implementation method of an embodiment of the present invention, target text data and associated text data associated with the target text data in other communication accounts are obtained; and the target text data and the associated text data are determined as a target data set, such as Figure 3 As shown, the following steps may be included:

[0150] S221, acquiring target text data, and analyzing a first content type of the target text data.

[0151] Specifically, after obtaining the target text, the target text data may be analyzed to determine the content type of the target text data. For the sake of clarity in describing the solution, this content type is referred to as a first content type.

[0152] S222, obtaining text data of other communication accounts, and analyzing the second content type of the text data of other communication accounts.

[0153] Specifically, the content type of the text data of many other communication accounts is analyzed. For the sake of clarity in describing the solution, this content type is referred to as the second content type.

[0154] S223: Calculate the similarity between the first content type and the second content type.

[0155] After analyzing the first content type and the second content type, the similarity between the two can be calculated, which can realize the above-mentioned embodiment of the text data classification process by case type. For example, if the first content type is type A and the second content type is also type A, then the similarity between the two can be 100%.

[0156] S224, when the similarity is greater than a preset similarity, determining the text data of the other communication accounts as associated text data of the target text data, and determining the target text data and the associated text data as a target data set.

[0157] If the similarity between the first content type and the second content type is high, it means that the similarity between the target text content and the text content of other communication accounts is also high. Therefore, the text content of other communication accounts can be determined as associated text data of the target text data, and the target text data and the associated text data can be determined as the target data set.

[0158] After the text data is grouped, since the keywords may be used repeatedly many times in the target dataset, but appear less frequently in the common dataset, the difference between the target dataset and the common dataset will be reflected, making the calculated keyword weights more accurate.

[0159] S230, segmenting the target data set and the common data set to obtain a first total number of the first keywords in the target data set and a first number of occurrences corresponding to the first keywords respectively; and a second total number of the second keywords in the common data set and a second number of occurrences corresponding to the second keywords respectively.

[0160] Specifically, the standard word segmenter of Hanlp, a powerful Chinese natural language processing library, is used to perform word segmentation on the target data set and the common data set, respectively, to obtain multiple first keywords of the target data set and multiple second keywords of the common data set. Through quantitative statistics, a first total number of the multiple first keywords and the first number of occurrences corresponding to the multiple first keywords can be obtained; and a second total number of the multiple second keywords and the second number of occurrences corresponding to the multiple second keywords can be obtained.

[0161] S240, for any target keyword among the multiple first keywords, calculate the first keyword frequency of the target keyword based on the first number of occurrences and the first total number corresponding to the target keyword; and calculate the second keyword frequency of the target keyword based on the second number of occurrences and the second total number corresponding to the target keyword in the common data set.

[0162] Assume that x is the number of times the target keyword appears in the target data set, that is, the first number of times it appears; y is the number of times the target keyword appears in the common data set, that is, the second number of times it appears; M t is the total number of keywords in the target data set, i.e., the first total number; M c It is the total number of keywords in the common data set, that is, the second total number.

[0163] Then, TF(x) is the first keyword frequency, and the calculation formula is:

[0164]

[0165] TF(y) is the second keyword frequency, and the calculation formula is:

[0166]

[0167] S250, determining the total number of the plurality of communication accounts in the general data set as the first number of accounts, and determining the number of the communication accounts whose text data includes the target keyword as the second number of accounts.

[0168] Specifically, A c is the total number of communication accounts corresponding to the common data set, that is, the first total number. That is, if the text data of N communication accounts are taken as the common data set, then the first total number is N. e is the number of communication accounts in which the target keyword appears in the general data set, that is, the second total number. That is to say, if the target keyword appears in the text data of n communication accounts among the text data of N communication accounts, then the second total number is n.

[0169] S260, calculating the inverse document frequency of the target keyword in the common data set according to the first account number and the second account number; and calculating the weight of the target keyword according to the first keyword frequency, the second keyword frequency and the inverse document frequency.

[0170] Specifically, IDF is the inverse document frequency of the target keyword in the common data set, and the calculation formula is:

[0171]

[0172] The weight calculation formula for the target keyword is:

[0173]

[0174] In the technical solution of the embodiment of the present application, the keyword weight of the text data of the communication account is asynchronously executed in a task-driven manner, which reduces the dependence on computing power performance compared with the technical solution of traditional word frequency analysis. In addition, a target data set including target text data and a common data set are determined. For any keyword in the target data set, the keyword frequency of the keyword in the target data set and the keyword frequency in the common data set are calculated; and the number of communication accounts is used as a weight calculation factor, that is, the inverse document frequency is calculated by determining the number of communication accounts in which the keyword appears in the common text set and the total number of communication accounts corresponding to the common data set, and finally the weight of the keyword is calculated by the keyword frequency of the keyword in the target data set, the keyword frequency of the keyword in the common data set, and the inverse document frequency. It can be seen that the present invention can significantly improve the discrimination of the weights of different keywords by redefining the weight calculation formula of TF-IDF, that is, taking the number of communication accounts as a weight calculation factor, thereby improving the accuracy of keyword weights.

[0175] On the basis of the above embodiment, in one implementation manner, a keyword weight analysis method for text data provided by an embodiment of the present invention may further include the following steps, namely, step a1 to step a4, after segmenting the target data set and the common data set:

[0176] Step a1, obtaining the first word segmentation result of the target data set and the second word segmentation result of the common data set.

[0177] Step a2: determine whether there are keywords satisfying the source text filtering rules in the first word segmentation result and the second word segmentation result respectively.

[0178] The source text filtering rule includes at least one of the following rules: filtering keywords whose source text length is greater than a preset length, filtering keywords whose source text format is a preset format, and filtering keywords whose source text is a link.

[0179] Step a3: if there are keywords satisfying the source text filtering rule in the first word segmentation result, the keywords satisfying the source text filtering rule are deleted from the first word segmentation result.

[0180] Step a4: if there are keywords satisfying the source text filtering rule in the second word segmentation result, the keywords satisfying the source text filtering rule are deleted from the second word segmentation result.

[0181] Specifically, after obtaining the first word segmentation result and the second word segmentation result, in order to prevent some useless keywords from interfering with the useful feature keywords and thus affecting the weight calculation result of the keywords, the useless keywords can be filtered by the source text filtering rule. In the above embodiment, the source text filtering rule has been described in detail and will not be repeated here.

[0182] On the basis of the above embodiment, in one implementation manner, a keyword weight analysis method for text data provided by an embodiment of the present invention may further include the following steps, namely, step b1 to step b4, after segmenting the target data set and the common data set:

[0183] Step b1, obtaining the first word segmentation result of the target data set and the second word segmentation result of the common data set.

[0184] Step b2, determining whether there are keywords satisfying keyword filtering rules in the first word segmentation result and the second word segmentation result respectively; the keyword filtering rules include at least one of the following rules: filtering keywords with consecutive identical characters, filtering keywords with preset parts of speech, and filtering keywords with preset content types;

[0185] Step b3, if there are keywords satisfying the keyword filtering rule in the first word segmentation result, delete the keywords satisfying the keyword filtering rule from the first word segmentation result;

[0186] Step b4: if there are keywords satisfying the keyword filtering rule in the second word segmentation result, delete the keywords satisfying the keyword filtering rule from the second word segmentation result.

[0187] Specifically, after obtaining the first word segmentation result and the second word segmentation result, in order to prevent some useless keywords from interfering with the useful feature keywords and thus affecting the weight calculation result of the keywords, useless keywords can be filtered through keyword filtering rules. In the above embodiment, the keyword filtering rules have been described in detail and will not be repeated here.

[0188] On the basis of the above embodiment, in one implementation manner, a keyword weight analysis method for text data provided by an embodiment of the present invention may further include the following steps, namely:

[0189] Step c1, after calculating the weight of each first keyword in the target data set, generate a weight result table.

[0190] The weight result table includes each first keyword and the weight of each first keyword.

[0191] Step c2, analyzing the weight result table to obtain summary keywords with weights greater than preset weights.

[0192] Step c3, generating an account summary of the target communication account through summary keywords.

[0193] Specifically, after analyzing and obtaining the weight of each keyword, the analysis results are persisted in the database in the form of a weighted result table, which improves the query speed and facilitates the display of various page views. In addition, by generating an account summary of the target communication account through the summary keyword, it can quickly obtain the summary of each communication account and provide a certain direction for the next step of analysis of other text content of the communication account.

[0194] The embodiment of the present invention also provides a keyword weight analysis device for text data, such as Figure 4 As shown, the device comprises:

[0195] The task acquisition module 410 is used to acquire a task to be processed; the task to be processed is used to indicate: analyzing the weights corresponding to the keywords of the target text data of the target communication account;

[0196] A data set determination module 420, configured to determine a target data set including the target text data, and determine a common data set based on text data of a plurality of communication accounts; the plurality of communication accounts and the target communication account belong to the same instant messaging software;

[0197] A data segmentation module 430 is used to segment the target data set and the common data set to obtain a first total number of first keywords in the target data set and a first number of occurrences corresponding to the first keywords; and a second total number of second keywords in the common data set and a second number of occurrences corresponding to the second keywords;

[0198] A keyword frequency calculation module 440 is used to calculate, for any target keyword among the plurality of first keywords, a first keyword frequency of the target keyword based on a first number of occurrences corresponding to the target keyword and the first total number; and calculate a second keyword frequency of the target keyword based on a second number of occurrences corresponding to the target keyword in the common data set and the second total number;

[0199] An account number determination module 450 is used to determine the total number of the multiple communication accounts in the general data set as the first account number, and the number of the communication accounts whose text data includes the target keyword as the second account number;

[0200] The weight calculation module 460 is used to calculate the inverse document frequency of the target keyword in the common data set according to the first number of accounts and the second number of accounts; and calculate the weight of the target keyword according to the first keyword frequency, the second keyword frequency and the inverse document frequency.

[0201] Optionally, the data set determination module is specifically used to:

[0202] Acquire the target text data, and determine the target text data as a target data set;

[0203] or,

[0204] The target text data and associated text data associated with the target text data in other communication accounts are obtained; and the target text data and the associated text data are determined as a target data set.

[0205] Optionally, the data set determination module is specifically used to:

[0206] Acquire the target text data, and analyze the first content type of the target text data;

[0207] Acquire text data of other communication accounts, and analyze the second content type of the text data of the other communication accounts;

[0208] Calculating the similarity between the first content type and the second content type;

[0209] When the similarity is greater than a preset similarity, the text data of the other communication account is determined as associated text data of the target text data, and the target text data and the associated text data are determined as a target data set.

[0210] Optionally, the device further includes a keyword filtering module, specifically configured to:

[0211] After segmenting the target data set and the common data set, obtaining a first segmentation result of the target data set and a second segmentation result of the common data set;

[0212] Determine whether there are keywords satisfying source text filtering rules in the first word segmentation result and the second word segmentation result respectively; the source text filtering rules include at least one of the following rules: filtering keywords whose source text length is greater than a preset length, filtering keywords whose source text format is a preset format, and filtering keywords whose source text is a link;

[0213] If there are keywords satisfying the source text filtering rule in the first word segmentation result, deleting the keywords satisfying the source text filtering rule from the first word segmentation result;

[0214] When the second word segmentation result contains keywords satisfying the source text filtering rule, the keywords satisfying the source text filtering rule are deleted from the second word segmentation result.

[0215] Optionally, the keyword filtering module is further specifically used for:

[0216] After segmenting the target data set and the common data set, obtaining a first segmentation result of the target data set and a second segmentation result of the common data set;

[0217] Determine whether there are keywords satisfying keyword filtering rules in the first word segmentation result and the second word segmentation result respectively; the keyword filtering rules include at least one of the following rules: filtering keywords with consecutive identical characters, filtering keywords with a preset part of speech, and filtering keywords with a preset type of content;

[0218] If there are keywords satisfying the keyword filtering rule in the first word segmentation result, deleting the keywords satisfying the keyword filtering rule from the first word segmentation result;

[0219] When there are keywords satisfying the keyword filtering rule in the second word segmentation result, the keywords satisfying the keyword filtering rule are deleted from the second word segmentation result.

[0220] Optionally, the device further includes an account summary generation module, which is specifically used to:

[0221] After calculating the weight of each first keyword in the target data set, generating a weight result table; the weight result table includes each first keyword and the weight of each first keyword;

[0222] Analyze the weight result table to obtain summary keywords with weights greater than preset weights;

[0223] An account summary of the target communication account is generated using the summary keywords.

[0224] The embodiment of the present invention further provides an electronic device, such as Figure 5As shown, including:

[0225] at least one processor 501;

[0226] a memory 502 for storing the at least one processor executable instruction;

[0227] The at least one processor is configured to execute the instructions to implement the keyword weight analysis method for text data according to an embodiment of the present invention.

[0228] In the technical solution of the embodiment of the present application, the keyword weight of the text data of the communication account is asynchronously executed in a task-driven manner, which reduces the dependence on computing power performance compared with the technical solution of traditional word frequency analysis. In addition, a target data set including target text data and a common data set are determined. For any keyword in the target data set, the keyword frequency of the keyword in the target data set and the keyword frequency in the common data set are calculated; and the number of communication accounts is used as a weight calculation factor, that is, the inverse document frequency is calculated by determining the number of communication accounts in which the keyword appears in the common text set and the total number of communication accounts corresponding to the common data set, and finally the weight of the keyword is calculated by the keyword frequency of the keyword in the target data set, the keyword frequency of the keyword in the common data set, and the inverse document frequency. It can be seen that the present invention can significantly improve the discrimination of the weights of different keywords by redefining the weight calculation formula of TF-IDF, that is, taking the number of communication accounts as a weight calculation factor, thereby improving the accuracy of keyword weights.

[0229] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the keyword weight analysis method of text data of an embodiment of the present invention.

[0230] In the technical solution of the embodiment of the present application, the keyword weight of the text data of the communication account is asynchronously executed in a task-driven manner, which reduces the dependence on computing power performance compared with the technical solution of traditional word frequency analysis. In addition, a target data set including target text data and a common data set are determined. For any keyword in the target data set, the keyword frequency of the keyword in the target data set and the keyword frequency in the common data set are calculated; and the number of communication accounts is used as a weight calculation factor, that is, the inverse document frequency is calculated by determining the number of communication accounts in which the keyword appears in the common text set and the total number of communication accounts corresponding to the common data set, and finally the weight of the keyword is calculated by the keyword frequency of the keyword in the target data set, the keyword frequency of the keyword in the common data set, and the inverse document frequency. It can be seen that the present invention can significantly improve the discrimination of the weights of different keywords by redefining the weight calculation formula of TF-IDF, that is, taking the number of communication accounts as a weight calculation factor, thereby improving the accuracy of keyword weights.

[0231] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the keyword weight analysis method for text data of an embodiment of the present invention.

[0232] In the technical solution of the embodiment of the present application, the keyword weight of the text data of the communication account is asynchronously executed in a task-driven manner, which reduces the dependence on computing power performance compared with the technical solution of traditional word frequency analysis. In addition, a target data set including target text data and a common data set are determined. For any keyword in the target data set, the keyword frequency of the keyword in the target data set and the keyword frequency in the common data set are calculated; and the number of communication accounts is used as a weight calculation factor, that is, the inverse document frequency is calculated by determining the number of communication accounts in which the keyword appears in the common text set and the total number of communication accounts corresponding to the common data set, and finally the weight of the keyword is calculated by the keyword frequency of the keyword in the target data set, the keyword frequency of the keyword in the common data set, and the inverse document frequency. It can be seen that the present invention can significantly improve the discrimination of the weights of different keywords by redefining the weight calculation formula of TF-IDF, that is, taking the number of communication accounts as a weight calculation factor, thereby improving the accuracy of keyword weights.

[0233] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and intent of the present invention.

Claims

1. A keyword weight analysis method for text data, characterized in that: The method comprises: Obtaining a task to be processed; the task to be processed is used to indicate: analyzing the weights corresponding to keywords of target text data of a target communication account; Determine a target data set including the target text data, and determine a common data set based on the text data of multiple communication accounts; the multiple communication accounts and the target communication account belong to the same instant messaging software; Segmenting the target data set and the common data set to obtain a first total number of a plurality of first keywords in the target data set and a first number of occurrences respectively corresponding to the plurality of first keywords; and a second total number of a plurality of second keywords in the common data set and a second number of occurrences respectively corresponding to the plurality of second keywords; For any target keyword among the plurality of first keywords, based on a first number of occurrences corresponding to the target keyword and the first total number, a first keyword frequency of the target keyword is calculated; and based on a second number of occurrences corresponding to the target keyword in the common data set and the second total number, a second keyword frequency of the target keyword is calculated; Determine the total number of the plurality of communication accounts in the general data set as the first number of accounts, and the number of the communication accounts whose text data includes the target keyword as the second number of accounts; The inverse document frequency of the target keyword in the common data set is calculated according to the first number of accounts and the second number of accounts; and the weight of the target keyword is calculated according to the first keyword frequency, the second keyword frequency and the inverse document frequency.

2. The method according to claim 1, characterized in that The determining of a target data set including the target text data comprises: Acquire the target text data, and determine the target text data as a target data set; or, The target text data and associated text data associated with the target text data in other communication accounts are obtained; and the target text data and the associated text data are determined as a target data set.

3. The method according to claim 2, characterized in that The step of acquiring the target text data and associated text data associated with the target text data in other communication accounts; and determining the target text data and the associated text data as a target data set includes: Acquire the target text data, and analyze the first content type of the target text data; Acquire text data of other communication accounts, and analyze the second content type of the text data of the other communication accounts; Calculating the similarity between the first content type and the second content type; When the similarity is greater than a preset similarity, the text data of the other communication account is determined as associated text data of the target text data, and the target text data and the associated text data are determined as a target data set.

4. The method according to any one of claims 1 to 3, characterized in that: After segmenting the target data set and the common data set, the method further includes: Obtaining a first word segmentation result of the target data set and a second word segmentation result of the common data set; Determine whether there are keywords satisfying source text filtering rules in the first word segmentation result and the second word segmentation result respectively; the source text filtering rules include at least one of the following rules: filtering keywords whose source text length is greater than a preset length, filtering keywords whose source text format is a preset format, and filtering keywords whose source text is a link; If there are keywords satisfying the source text filtering rule in the first word segmentation result, deleting the keywords satisfying the source text filtering rule from the first word segmentation result; When the second word segmentation result contains keywords satisfying the source text filtering rule, the keywords satisfying the source text filtering rule are deleted from the second word segmentation result.

5. The method according to any one of claims 1 to 3, characterized in that: After segmenting the target data set and the common data set, the method further includes: Obtaining a first word segmentation result of the target data set and a second word segmentation result of the common data set; Determine whether there are keywords satisfying keyword filtering rules in the first word segmentation result and the second word segmentation result respectively; the keyword filtering rules include at least one of the following rules: filtering keywords with consecutive identical characters, filtering keywords with a preset part of speech, and filtering keywords with a preset type of content; If there are keywords satisfying the keyword filtering rule in the first word segmentation result, deleting the keywords satisfying the keyword filtering rule from the first word segmentation result; When there are keywords satisfying the keyword filtering rule in the second word segmentation result, the keywords satisfying the keyword filtering rule are deleted from the second word segmentation result.

6. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: After calculating the weight of each first keyword in the target data set, generating a weight result table; the weight result table includes each first keyword and the weight of each first keyword; Analyze the weight result table to obtain summary keywords with weights greater than preset weights; An account summary of the target communication account is generated using the summary keywords.

7. A keyword weight analysis device for text data, characterized in that: The device comprises: A task acquisition module is used to acquire tasks to be processed; the tasks to be processed are used to indicate: analyzing the weights corresponding to the keywords of the target text data of the target communication account; A data set determination module, used to determine a target data set including the target text data, and determine a common data set based on text data of a plurality of communication accounts; the plurality of communication accounts and the target communication account belong to the same instant messaging software; a data segmentation module, configured to segment the target data set and the common data set to obtain a first total number of a plurality of first keywords in the target data set and a first number of occurrences corresponding to the plurality of first keywords respectively; and a second total number of a plurality of second keywords in the common data set and a second number of occurrences corresponding to the plurality of second keywords respectively; a keyword frequency calculation module, configured to calculate, for any target keyword among the plurality of first keywords, a first keyword frequency of the target keyword based on a first number of occurrences corresponding to the target keyword and the first total number; and to calculate a second keyword frequency of the target keyword based on a second number of occurrences corresponding to the target keyword in the common data set and the second total number; an account number determination module, configured to determine the total number of the plurality of communication accounts in the general data set as the first account number, and the number of the communication accounts whose text data includes the target keyword as the second account number; A weight calculation module is used to calculate the inverse document frequency of the target keyword in the common data set according to the first number of accounts and the second number of accounts; and to calculate the weight of the target keyword according to the first keyword frequency, the second keyword frequency and the inverse document frequency.

8. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images