Character string data grouping method for mining data management

Abstract

The invention relates to the technical field of text processing and data analysis, in particular to a character string data grouping method for mining data management, which comprises the following steps of: 1, traversing all character string data, and grouping the character string data into character string groups; step 2, performing grouping effect evaluation on each grouping scheme; and step 3, performing pairwise comparison on the plurality of grouping schemes, and performing grouping effect optimization iteration, specifically, randomly selecting two grouping schemes, exchanging character string groups in the two grouping schemes to form a new grouping scheme, and if the effect of the new grouping scheme is superior to that of the original grouping scheme, replacing the original scheme. According to the method, iterative optimization is carried out according to the grouping effect, the length and accuracy of the common substrings of the character strings in the group are comprehensively considered, the length of the common substrings in the group can be increased according to the output grouping result, and meanwhile the requirement for accurate classification is met.

Description

Technical Field

[0001] The present application relates to the technical field of text processing and data analysis, and in particular to a string data grouping method for mining data management. Background Art

[0002] In the data management of coal mining operations, there is a large amount of background data. In the process of informatization construction, some information that can be transmitted through sensors can be automatically uploaded to generate reports, which brings convenience to the unified management of the project.

[0003] The existing string data grouping technology mainly includes keyword grouping and text similarity grouping. The word segmentation method can divide the string into meaningful simple or short phrases, but it cannot form complex common points and has certain defects in the expression of meaning; text similarity calculation aims to calculate the similarity between two or more text fragments, but it is easy to group string data with high similarity but important differences into one group, which often requires further screening. In coal mining, some very similar data contents often appear, with only one or two words different, but the meanings they express are very different. For example, F11 return airway and F10 return airway return air, these two data mark two completely different parameters in two completely different geographical locations. The former records the structural parameters of F11 return airway, and the latter may record the air parameters in F10 return airway. Whether it is keyword grouping or text similarity grouping, these two words are very likely to be grouped together for screening, resulting in low data classification accuracy. Summary of the invention

[0004] The technical problem to be solved by the present invention is that the existing character string data grouping method has low data classification accuracy.

[0005] To this end, the present invention provides a character string data grouping method for mining data management.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] A method for grouping character string data for mining data management, comprising:

[0008] Step 1: traverse all string data and group the string data into string groups;

[0009] Step 2: Evaluate the grouping effect of each grouping scheme;

[0010] Step 3: Compare multiple grouping schemes in pairs and optimize the grouping effect iteratively, specifically:

[0011] Randomly select two grouping schemes and swap the string groups in the two grouping schemes to form a new grouping scheme. If the new grouping scheme is better than the original one, it will replace the original one.

[0012] Furthermore, in step 1, all string data are traversed, all possible substrings are calculated for each string, and the frequency of occurrence of each substring in all strings is calculated to divide the string groups.

[0013] Furthermore, in step one, after a string group is obtained based on a substring with the highest frequency of occurrence and the longest character length, all remaining string data are traversed, all possible substrings are calculated for each remaining string, and the frequency of occurrence of each substring in all remaining strings is calculated to divide the string groups, until all strings are grouped to form a string grouping scheme.

[0014] Furthermore, in step 2, the grouping effect evaluation is calculated according to the following formula: Among them, E is the evaluation score of the grouping result. The higher the score, the better the grouping effect. N is the total number of groups. L i is the length of the common substring in the i-th group; C i is the number of members in the i-th group.

[0015] Furthermore, in step three, all initial groupings of the string data are formed according to the string grouping scheme and the random grouping scheme obtained by the algorithm in step one, and the grouping effect optimization iteration is performed in all the initial groupings.

[0016] Furthermore, in step three, a certain number of grouping schemes are randomly selected, a character string is randomly selected, and the effect evaluation of the grouping schemes after classifying them into other groups is calculated in turn, and the best one is selected as a new grouping scheme to replace the original one.

[0017] Furthermore, each string grouping scheme includes all string data and has no repeated strings.

[0018] The beneficial effect of the present invention is that the present invention comprehensively considers the length and accuracy of the common substrings of the character strings in the group by iterative optimization according to the grouping effect, so that the output grouping result can increase the length of the common substrings in the group while meeting the requirements of accurate classification.

[0019] In order to ensure the accuracy of grouping, in this grouping method, only string data containing common substrings of the group can be added to the group. This is an important difference between this grouping method and the grouping method of text similarity calculation class.

[0020] In addition, grouping effect evaluation and heuristic algorithms are used to optimize the grouping effect, which can increase the length of the common substrings within the group in the output grouping results. The common substring length and accuracy of the strings within the group are comprehensively considered, which can increase the length of the common substrings within the group in the output grouping results and meet the requirements of accurate classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0022] Figure 1 It is a flow chart of the string grouping algorithm in the present invention.

[0023] Figure 2 It is a flow chart of the iterative optimization algorithm in the present invention. DETAILED DESCRIPTION

[0024] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0025] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0026] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0027] A method for grouping string data, comprising the following steps:

[0028] S1 traverses all string data and calculates all possible substrings for each string.

[0029] For example, there are 6 character strings to be grouped, namely: F21406 return air lane, F21406 return air lane return air, F21406 return air lane air inlet, second horizontal south wing belt lane, second horizontal south wing return air lane, second horizontal south wing track lane, and these character strings are named M1, M2, M3, M4, M5, and M6 respectively.

[0030] According to the length of the characters, the substrings are divided. Taking the M1 string F21406 return air lane as an example, the substrings with a length of 1 are: F, 2, 1, 4, 0, 6, return, air, lane; the substrings with a length of 2 are: F2, 21, 14, 40, 06, 6 return, return air, air lane; the substrings with a length of 3 are: F21, 214, 140, 406, 06 return, 6 return air, return air lane; the substrings with a length of 4 are: F214, 2140, 1406, 406 return, 06 return air , 6 return air lanes; substrings of length 5 are: F2140, 21406, 1406 return, 406 return air, 06 return air lane; substrings of length 6 are: F21406, 21406 return, 1406 return air, 406 return air lane; substrings of length 7 are: F21406 return, 21406 return air, 1406 return air lane; substrings of length 8 are: F21406 return air, 21406 return air lane; substrings of length 9 are: F21406 return air lane.

[0031] Take the M1 string as an example to divide the remaining 5 strings to be grouped.

[0032] S2 calculates the frequency of occurrence of each substring in all ungrouped character strings, and selects the longest substring among the substrings with the highest frequency as the basis for grouping.

[0033] For example, the substring: F21406 return air lane, the frequency is 3 (appearing in the string: F21406 return air lane, F21406 return air lane return air, F21406 return air lane inlet air outlet), and the length is 9;

[0034] Substring: F21406 return air, frequency is 3 (appears in the string: F21406 return air lane, F21406 return air lane return air, F21406 return air lane inlet air outlet), length is 8;

[0035] Substring: 21406 return air lane, frequency is 3 (appears in the string: F21406 return air lane, F21406 return air lane return air, F21406 return air lane inlet air outlet), length is 8;

[0036] Substring: F21406 return, frequency is 3 (appears in the string: F21406 return air lane, F21406 return air lane return air, F21406 return air lane inlet air outlet), length is 7;

[0037] Substring: 21406 return air, frequency is 3 (appears in the string: F21406 return air lane, F21406 return air lane return air, F21406 return air lane inlet air outlet), length is 7;

[0038] Substring: 1406 return air lane, frequency is 3 (appears in the string: F21406 return air lane, F21406 return air lane return air, F21406 return air lane inlet air outlet), length is 7;

[0039] Substring: 1406 return air, frequency is 3 (appears in the string: F21406 return air lane, F21406 return air lane return air, F21406 return air lane inlet air outlet), length is 6;

[0040] …

[0041] Substring: Second level south wing, frequency is 3 (appears in the string: Second level south wing belt lane, Second level south wing return air lane, Second level south wing track lane), length is 5

[0042] …

[0043] Finally, it is found that among the substrings with the highest frequency of occurrence, the substring with the longest length is: F21406 return air duct.

[0044] S3 classifies the character data containing the substring into the group. The substring F21406 appears in the return air channel as a character string of M1, M2, M3, and the first group L1 {M1, M2, M3} is obtained.

[0045] S4 counts all possible substrings in the remaining character strings M4, M5, and M6 by referring to the substring partitioning method of step S1.

[0046] S5 repeats steps S2 to S4 to obtain the final grouping result, forming grouping scheme A: group L1 {M1, M2, M3}, the common substring is "F21406 return air channel"; group L2 {M4, M5, M6}, the common substring is "second level south wing".

[0047] S6 evaluates the overall grouping results

[0048] There are two goals of grouping: first, the more concentrated the groups are, the better, that is, we expect to have more group members and fewer groups; second, the more common points within the groups, the better, that is, the longer the length of the common substring is, the better.

[0049] The effect of a single grouping is equal to the length of the common substring multiplied by the number of group members. The effect of the overall grouping scheme is equal to the sum of the effects of each grouping divided by the square of the number of groups:

[0050]

[0051] Among them, E is the evaluation score of the grouping result, the higher the score, the better the grouping effect; N is the total number of groups; Li is the length of the common substring in the i-th group; Ci is the number of members in the i-th group.

[0052] S7 grouping effect optimization iteration:

[0053] S7.1. Perform algorithmic grouping and random grouping with reference to S1-S7 at a certain scale (which can be adjusted according to the actual grouping effect) to complete the initial grouping of string data.

[0054] S7.2, randomly select two grouping schemes m and n (m and n are both constants), randomly select a certain proportion of character strings 1 to N, change the grouping of character strings 1 to N in the m scheme to their grouping in the n scheme, and change the grouping of character strings 1 to N in the n scheme to their grouping in the m scheme.

[0055] S7.3, randomly select a certain number of grouping schemes, randomly select a string, calculate the effect evaluation of the grouping schemes after classifying it into other groups, and select the best one as the new grouping scheme to replace the original one.

[0056] S7.4, calculate the effect evaluation of each grouping scheme (part 2), and find the one with the largest effect evaluation value as the optimal grouping scheme for this round of iteration.

[0057] S7.5, repeat steps S8.2 to S8.4 until the improvement is less than a certain amplitude or the upper limit of the number of iterations is reached.

[0058] It should be noted that there may be multiple groups of longest substrings among the most frequently occurring substrings. If the grouping effect is evaluated by exhaustively enumerating each option, the algorithm complexity will increase exponentially with the number of substrings involved in the grouping, and the grouping will consume too much time. If any one of them is selected as the basis for grouping, the grouping quality will be reduced. In order to resolve this contradiction, an optimization algorithm is used to extract the required substrings to improve the grouping effect and quality.

[0059] For example:

[0060] Data to be grouped: M1, ABC; M2, BCD; M3, CDE; M4, DEF; M5, EFG; M6, FGH; M7, GHI; M8, HIJK.

[0061] The most frequent substrings and their lengths:

[0062] Substring: BC, frequency: 2, length: 2, grouping L1{M1, M2}

[0063] Substring: CD, frequency: 2, length: 2, grouping L2{M2, M3}

[0064] Substring: DE, frequency: 2, length: 2, grouping L3{M3, M4}

[0065] Substring: EF, frequency: 2, length: 2, grouping L4{M4, M5}

[0066] Substring: FG, frequency: 2, length: 2, grouping L5{M5, M6}

[0067] Substring: GH, frequency: 2, length: 2, grouping L6{M6, M7}

[0068] Substring: HI, frequency: 2, length: 2, grouping L7{M7, M8}

[0069] There are 7 groups of substrings with the same length and the substrings with the highest frequency will have more substrings with the same length when the grouping is continued. If all the grouping schemes are exhausted, a total of more than 7×5×3=105 calculations are required.

[0070] Assume that the scale of this optimization algorithm is 6, that is, the method is used 6 times to group them one by one, and each time the most frequent substrings have the same length, a substring is randomly selected.

[0071] Through step S7.1, assuming that the final grouping results of the 6 grouping schemes are:

[0072] Grouping scheme 1: {L1, L3, L5, L7}, specifically, string group L1 {ABC, BCD} containing common substring BC; string group L3 {CDE, DEF} containing common substring DE; string group L5 {EFG, FGH} containing common substring FG; string group L7 {GHI, HIJK} containing common substring HI. The grouping effect E of this grouping scheme is 1.

[0073] Grouping scheme 2: {L2, L4, L6, M1, M8}, specifically, string group L2 {BCD, CDE} containing the common substring CD; string group L4 {(DEF, EFG} containing the common substring EF; string group L6 {FGH, GHI} containing the common substring GH; and two string groups consisting of a single string. The grouping effect E of this grouping scheme is 0.76.

[0074] Grouping scheme 3: {L3, L6, L1, M5, M8}. The grouping effect of this grouping scheme is E=0.76.

[0075] Grouping scheme 4: {L4, L1, L5, M3, M8}. The grouping effect of this grouping scheme is E=0.76.

[0076] Grouping scheme 5: {L5, L2, L7, M1, M4}. The grouping effect of this grouping scheme is E=0.72.

[0077] Grouping scheme 6: {L6, L1, L3, M5, M8}. The grouping effect of this grouping scheme is E=0.76.

[0078] Each of the above groups includes each string. Referring to step 7.2, taking two of the grouping schemes 5 and 6 as examples, the randomly selected strings are string ABC and string DEF. The groups ABC and DEF of the strings ABC and DEF in grouping scheme 5 are changed to their groups BC and DE in grouping scheme 6. BCD and CDE in the original group CD are therefore ungrouped. After the adjustment, the ungrouped strings are further grouped, ABC and BCD are divided into group BC, and CDE and DEF are divided into group DE; the groups of the strings ABC and DEF in grouping scheme 6 are changed to their groups ABC and DEF in grouping scheme 5. After the adjustment, the ungrouped strings are further grouped, and DEF and EFG are divided into group EF.

[0079] The grouping scheme 5 after step S7.2 is: {L1, L3, L5, L7}, specifically including, the string group L1 {ABC, BCD} containing the common substring BC; the string group L3 {CDE, DEF} containing the common substring DE; the string group L5 {EFG, FGH} containing the common substring FG; the string group L7 {GHI, HIJK} containing the common substring HI, and the new grouping effect E=1.

[0080] Grouping scheme 6 is: {L2, L4, L6, M1, M8}, specifically, string group L2 {BCD, CDE} containing the common substring CD; string group L4 {(DEF, EFG} containing the common substring EF; string group L6 {FGH, GHI} containing the common substring GH; and two string groups consisting of a single string. The grouping effect E of this grouping scheme is 0.76.

[0081] Referring to step S7.3, taking two of the grouping schemes 3 as an example, the randomly selected string is FGH, the group GH where FGH is located is split, and the other groups are retained, and the strings containing G and H are EFG, FGH, GHI, and HIJK. All possible grouping schemes and their grouping effects are calculated, and the optimal grouping schemes are FG (EFG, FGH), HI (GHI, HIJK). Adding the previously retained groups, we can get a new grouping scheme: {L1, L3, L5, L7}, and the new grouping effect E=1.

[0082] Referring to step S7.4, the optimal grouping schemes of this iteration are grouping schemes 1, 3, and 5, and the grouping results and grouping effects are the same. By continuing the iteration through step S7.5, the optimal grouping result is obtained: {L1, L3, L5, L7}, and the grouping effect E=1.

[0083] In summary, the present invention comprehensively considers the length and accuracy of the common substrings of the strings in the group by iteratively optimizing according to the grouping effect, so that the output grouping result can increase the length of the common substrings in the group while meeting the requirements of accurate classification.

[0084] In order to ensure the accuracy of grouping, in this grouping method, only string data containing common substrings of the group can be added to the group. This is an important difference between this grouping method and the grouping method of text similarity calculation class.

[0085] In addition, grouping effect evaluation and heuristic algorithms are used to optimize the grouping effect, which can increase the length of the common substrings within the group in the output grouping results. The common substring length and accuracy of the strings within the group are comprehensively considered, which can increase the length of the common substrings within the group in the output grouping results and meet the requirements of accurate classification.

[0086] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for grouping string data for mining data management, characterized in that: include, Step 1: traverse all string data and group the string data into string groups; Step 2: Evaluate the grouping effect of each grouping scheme; Step 3: Compare multiple grouping schemes in pairs and optimize the grouping effect iteratively, specifically: Randomly select two grouping schemes and swap the string groups in the two grouping schemes to form a new grouping scheme. If the new grouping scheme is better than the original one, it will replace the original one.

2. The method for grouping character string data for mining data management according to claim 1, characterized in that: In step 1, all string data are traversed, all possible substrings are calculated for each string, and the frequency of each substring appearing in all strings is calculated to divide the string groups.

3. The method for grouping character string data for mining data management according to claim 2, characterized in that: In step 1, after a string group is obtained based on a substring with the highest frequency of occurrence and the longest character length, all remaining string data are traversed, all possible substrings are calculated for each remaining string, and the frequency of occurrence of each substring in all remaining strings is calculated to divide the string groups until all strings are grouped to form a string grouping scheme.

4. The method for grouping character string data for mining data management according to claim 1, characterized in that: In step 2, the grouping effect evaluation is calculated according to the following formula: Among them, E is the evaluation score of the grouping result. The higher the score, the better the grouping effect. N is the total number of groups. L i is the length of the common substring in the i-th group; C i is the number of members in the ith group.

5. The method for grouping character string data for mining data management according to claim 1, characterized in that: In step three, all initial groupings of the string data are formed according to the string grouping scheme and the random grouping scheme obtained by the algorithm in step one, and the grouping effect optimization iteration is performed in all initial groupings.

6. The method for grouping character string data for mining data management according to claim 5, characterized in that: In step three, a certain number of grouping schemes are randomly selected, a character string is randomly selected, and the effectiveness evaluation of the grouping schemes after classifying them into other groups is calculated in turn, and the best one is selected as the new grouping scheme to replace the original one.

7. The method for grouping character string data for mining data management according to any one of claims 1 to 6, characterized in that: Each string grouping scheme contains all string data and no repeated strings.

Images