A method and system for determining target text

Abstract

The present application relates to the technical field of text processing, and in particular to a method and system for determining a target text, the method comprising the following steps: obtaining a number of initial text keywords and a number of processed initial text fragments from each received initial text, and inputting them together into a language processing model to output a key text, wherein the key text generated in this way is a standard text with logical coherence and high integrity, and the text weight of each initial text is calculated through parameter information corresponding to the key text and the initial text respectively, and when the text weight corresponding to the initial text is greater than a preset text weight threshold, the initial text is determined as the target text; it can be seen that the present invention makes full use of the structure and content information of the text to comprehensively evaluate the importance of the text, can quickly and effectively screen out the required target text, and is conducive to the processing and analysis of a large amount of text data.

Description

Technical Field

[0001] The present invention relates to the technical field of text processing, and in particular to a method and system for determining a target text. Background Art

[0002] With the rapid development of science and technology and the popularization of Internet technology, the number of text files is increasing day by day. When processing and analyzing large amounts of text data, it is particularly important to quickly and effectively evaluate the importance of the text or whether the text is a high-value text.

[0003] In the prior art, information retrieval and information classification techniques are usually used to screen text files. Taking the most common word frequency-reverse file frequency method as an example, by constructing keywords related to the screening conditions, calculating the frequency of occurrence of the keyword in each text, and the proportion of the number of texts containing the keyword in the entire text database, the importance of each text under this keyword is judged, and then a number of text files with high importance are screened out. This evaluation method can only be applied to texts with high integrity, and the evaluation method is relatively simple and has low accuracy. Summary of the invention

[0004] In view of the above technical problems, the present invention provides a method and system for determining a target text to achieve accurate and effective evaluation of the text.

[0005] According to a first aspect of the present invention, a method for determining a target text is provided, the method comprising the following steps:

[0006] S100: Acquire an initial text keyword set and an initial text segment set corresponding to each initial text from the received initial text list set.

[0007] S200: Inputting an initial text keyword set and an initial text segment set corresponding to each initial text into a preset language processing model to obtain a key text corresponding to each initial text.

[0008] S300 , obtaining a text weight corresponding to each initial text according to a plurality of initial texts, an initial text keyword set corresponding to each initial text, and a key text corresponding to each initial text.

[0009] S400: When there is an initial text whose corresponding text weight is greater than a preset text weight threshold, the initial text whose corresponding text weight is greater than the preset text weight threshold is determined as a target text.

[0010] According to a second aspect of the present invention, a system for determining a target text is provided, the system comprising:

[0011] The keyword and text segment acquisition module is used to acquire an initial text keyword set and an initial text segment set corresponding to each initial text from the received initial text list set.

[0012] The key text acquisition module is used to input the initial text keyword set and the initial text segment set corresponding to each initial text into a preset language processing model to acquire the key text corresponding to each initial text.

[0013] The text weight acquisition module is used to acquire the text weight corresponding to each initial text according to a number of initial texts, an initial text keyword set corresponding to each initial text and a key text corresponding to each initial text.

[0014] The target text determination module is used to determine the initial text whose corresponding text weight is greater than the preset text weight threshold as the target text when there is an initial text whose corresponding text weight is greater than the preset text weight threshold.

[0015] Compared with the prior art, the present invention has obvious beneficial effects. By means of the above technical solution, the target text determination method and system provided by the present invention can achieve considerable technical advancement and practicality, and has wide industrial utilization value, and has at least the following beneficial effects:

[0016] The present invention provides a method for determining a target text. First, a plurality of initial text keywords and a plurality of initial text fragments after standardization are obtained from each received initial text, and the keywords are input into a language processing model together to output a key text. The key text generated in this way is a standard text with logical coherence and high integrity. Parameter information corresponding to the key text and the initial text is obtained through natural language query and character recognition, and the text weight of each initial text is calculated. When the text weight corresponding to the initial text is greater than a preset text weight threshold, the initial text is determined as the target text. By combining the parameters of the initial text and the key text after standardization, the structure and content information of the text are fully utilized to comprehensively evaluate the importance of the text, so that the calculated text weight of the initial text is more accurate, and the required target text can be quickly and effectively screened out, which is conducive to the processing and analysis of a large amount of text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1A flowchart of a method for determining a target text provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of a target text determination system provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of a portion of the structure of a keyword and text fragment acquisition module provided in an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of another part of the structure of the keyword and text fragment acquisition module provided in an embodiment of the present invention;

[0022] Figure 5 A schematic diagram of the structure of a target text determination module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0025] This embodiment provides a method for determining a target text, the method comprising the following steps: Figure 1 As shown:

[0026] S100: Acquire an initial text keyword set and an initial text segment set corresponding to each initial text from the received initial text list set.

[0027] Specifically, the initial text list set includes a number of initial texts; the initial texts correspond to the initial text keyword set one by one; the initial text keyword set includes a number of initial text keywords; and the initial text segment set includes a number of initial text segments.

[0028] In a specific embodiment, step S100 includes the following steps:

[0029] S101, based on the keyword extraction model, extracting several initial text keywords corresponding to the initial text itself from each initial text. Those skilled in the art are aware of any keyword extraction model in the prior art that can extract keywords from text, which will not be described in detail here.

[0030] S102, performing deduplication processing on the several initial text keywords, and combining the remaining initial text keywords into an initial text keyword set.

[0031] In a specific embodiment, step S100 also includes the following steps:

[0032] S110, removing a plurality of initial text keywords corresponding to each initial text from the corresponding initial text, and obtaining a plurality of pre-processed text segments corresponding to each initial text.

[0033] S120, searching for a plurality of target redundant words from a plurality of preprocessed text segments corresponding to each initial text according to a preset redundant word list.

[0034] Specifically, the preset redundant word table refers to a data table recording a number of preset redundant words; for example, commonly used modal particles.

[0035] Specifically, the target redundant word refers to a word in the preprocessed text segment that is consistent with any preset redundant word.

[0036] S130, deleting the plurality of target redundant words from the plurality of preprocessed text segments, and obtaining an initial text segment corresponding to each preprocessed text segment.

[0037] As mentioned above, after obtaining the initial text keywords, the preprocessed text fragments in the initial text are also obtained and converted into standard initial text fragments. Through the combination of the initial text keywords and the initial text fragments, the generated key text is more accurate and more in line with the original meaning of the initial text, and the standardized text is conducive to processing and analysis.

[0038] S200: Inputting an initial text keyword set and an initial text segment set corresponding to each initial text into a preset language processing model to obtain a key text corresponding to each initial text.

[0039] In another specific embodiment, the method can also directly obtain key text through keywords, including the following steps:

[0040] E100, the keyword set of the text to be processed CE={CE 1 , C.E. 2 ,……,CE α ,……,CE β}CE α Add to the preset keyword sequence and add CE α Remove from CE and get a new CE, where CE α is the αth text keyword to be processed, α=1, 2,…, β, β is the number of text keywords to be processed.

[0041] Specifically, the initial state of the preset keyword sequence is empty.

[0042] Specifically, the to-be-processed text keyword refers to any keyword collected by the target user based on the target person's verbal statement text information.

[0043] Specifically, the preset keyword sequence refers to a sequence for sequentially storing keywords of the text to be processed that are removed from the CE.

[0044] E200, traverse the text keywords to be processed in the new CE, calculate the correlation between each text keyword to be processed in the new CE and the CE α The co-occurrence probability of CE; that is, by constructing a co-occurrence matrix, find the co-occurrence probability of CE from the new CE. α The vector distance of the text to be processed is the closest and most reasonable. Those skilled in the art are aware of the specific implementation method of calculating the co-occurrence probability in the prior art, which will not be repeated here.

[0045] E300, according to each to-be-processed text keyword in the new CE and CE α The co-occurrence probability determines the maximum co-occurrence probability, and CE α Update to the to-be-processed text keyword corresponding to the maximum co-occurrence probability, return to execute E100, until the new CE is an empty set, and obtain CE α The corresponding maximum co-occurrence probabilities.

[0046] As described above, since a maximum co-occurrence probability is obtained in each cycle, when the new CE is an empty set, it has cycled β-1 times and obtained β-1 maximum co-occurrence probabilities.

[0047] E400, according to the maximum co-occurrence probabilities corresponding to each keyword of the text to be processed, obtain the first maximum co-occurrence probability list set P corresponding to CE={P 1 , P2 , ..., P α , ..., P β}, where P α ={P α1 , P α2 , ..., P αe , ..., P α(β-1)}, P αe For CE α The e-th maximum co-occurrence probability corresponding to the first to-be-processed text keyword added to the preset keyword sequence, e=1, 2, ..., β-1.

[0048] As mentioned above, by inputting β keywords of the text to be processed as starting keywords into the preset keyword sequence in turn, β maximum co-occurrence probability lists can be obtained, which is conducive to calculating the priority of each maximum co-occurrence probability list separately, so as to find the most reasonable order of sorting several keywords of the text to be processed and realize the order correction of keywords.

[0049] E500: According to P, obtain a first target co-occurrence probability list from β first maximum co-occurrence probability lists.

[0050] Specifically, in step E500, the first target co-occurrence probability list is obtained through the following steps:

[0051] E501, obtain CE α The co-occurrence probability list priority CS of the corresponding maximum co-occurrence probability list α ; can be understood as: CE α The corresponding maximum co-occurrence probability list refers to CE α The maximum co-occurrence probability list corresponding to when it is added as the starting keyword to the preset keyword sequence.

[0052] Among them, CS α Meet the following conditions:

[0053] CS α =∑ β-1 e=1 P αe .

[0054] E502 , according to the co-occurrence probability list priorities corresponding to each to-be-processed text keyword, determine the maximum co-occurrence probability list corresponding to the maximum co-occurrence probability list priority as the first target co-occurrence probability list.

[0055] As mentioned above, by adding up several co-occurrence probabilities in each maximum co-occurrence probability list respectively, the calculation result with the largest value can be found and the result can be used as the priority of the corresponding co-occurrence probability list. The larger the overall co-occurrence probability, the closer the average distance of several preset keywords is, and the most reasonable keyword sorting order can be found.

[0056] In another specific embodiment, the method further comprises:

[0057] E01, if CE α When only one text keyword to be processed remains in the new CE after being removed from the CE, the remaining text keyword to be processed is added to the preset keyword sequence and the CE is obtained. α The corresponding β-2 maximum co-occurrence probabilities.

[0058] E02, according to the β-2 maximum co-occurrence probabilities corresponding to each keyword in the text to be processed, obtain the second maximum co-occurrence probability list set corresponding to CE; the second maximum co-occurrence probability list set includes β second maximum co-occurrence probability lists, and each second maximum co-occurrence probability list includes β-2 maximum co-occurrence probabilities.

[0059] E03: According to the second maximum co-occurrence probability list set, a second target co-occurrence probability list is acquired from the second maximum co-occurrence probability list set.

[0060] Specifically, the method for acquiring the second target co-occurrence probability list is consistent with the method for acquiring the first target co-occurrence probability list.

[0061] As mentioned above, when there is only one keyword left in the keyword set, there is no other keyword in the keyword set. Therefore, there is no need to calculate and compare to find the maximum co-occurrence probability. It can be directly added to the preset keyword sequence.

[0062] E600, input the keywords of the text to be processed in the preset keyword sequence corresponding to the first target co-occurrence probability list into the preset language processing model in the order of addition to the preset keyword sequence, so as to obtain the key text corresponding to CE; those skilled in the art are aware of any language processing model in the prior art that can output corresponding text according to several keywords, and will not be described in detail here.

[0063] In another specific embodiment, in step E600, the first target co-occurrence probability list is replaced with the second target co-occurrence probability list to obtain the key text corresponding to CE.

[0064] In the above, by obtaining a reasonable arrangement order of several keywords of the text to be processed, the order correction of several keywords of the text to be processed is realized, and several preset keywords are input into the preset language processing model in sequence according to the corrected order to generate standardized processed text to realize the management and complete storage of the text.

[0065] In another specific embodiment, the method comprises:

[0066] E001, determining the initial text keyword set as a new text keyword set to be processed.

[0067] E002, acquiring key texts corresponding to the initial text keyword set according to the new to-be-processed text keyword set.

[0068] The above provides another form of obtaining standardized text, that is, when the data information collected by the target user is in text form rather than one-sided keywords, it is possible to extract several initial text keywords from the text and generate corresponding standard key text based on the several initial text keywords.

[0069] S300 , obtaining a text weight corresponding to each initial text according to a plurality of initial texts, an initial text keyword set corresponding to each initial text, and a key text corresponding to each initial text.

[0070] Specifically, the text weight corresponding to the initial text meets the following conditions:

[0071] W 00 r =W 0 r / max(W 0 1 ,W 0 2 ,……,W 0 r ,……,W 0 g ), where W 00 r is the text weight corresponding to the rth initial text in the initial text list, W 0 r =δ r1 ×N r ×δ r2 ×(δ r1 / δ rg ), δ r1 is the number of characters in the rth initial text, N r is the number of initial text keywords corresponding to the rth initial text, δ r2is the number of characters of the key text corresponding to the rth initial text, the value range of r is 1 to g, and g is the number of initial texts in the initial text list set.

[0072] S400: When there is an initial text whose corresponding text weight is greater than a preset text weight threshold, the initial text whose corresponding text weight is greater than the preset text weight threshold is determined as a target text.

[0073] The above provides a technical solution for obtaining the corresponding key text based on the initial text keywords and the initial text fragments, and then calculating the text weight of the corresponding initial text based on the initial text and the key text. This calculation method makes full use of the structure and content information of the text to achieve a comprehensive evaluation of the text, and can quickly determine the effective and high-value target text, which is conducive to the processing and analysis of large amounts of text data.

[0074] In a specific embodiment, in step S400, the preset text weight threshold is obtained through the following steps:

[0075] S401, receiving a plurality of preprocessed texts and obtaining a text weight corresponding to each preprocessed text; the method for obtaining the text weight corresponding to the preprocessed text is consistent with the method for obtaining the text weight corresponding to the initial text.

[0076] S402, sorting the text weights corresponding to the plurality of preprocessed texts from small to large to obtain a weight sorting result.

[0077] S403: Determine the text weight at the target sorting position in the weight sorting result as a preset text weight threshold.

[0078] Specifically, the target ranking position refers to the position determined by technicians in this field in the weight ranking results according to actual needs; for example, it can be the middle position or the value at the 80% position.

[0079] In the above, by taking a number of preprocessed texts as sample sets and determining the text weight threshold according to the text weights of the preprocessed texts, the set text weight threshold is made more reasonable, which is conducive to selecting effective target texts.

[0080] When classifying and managing texts, the texts are usually tagged according to the existing tagging system and then classified. There are multiple service platforms for storing and tagging texts, and each service platform has its own corresponding tagging system, that is, there are several scattered and different tagging systems. Therefore, it is necessary to effectively integrate these tagging systems. The specific implementation methods are as follows:

[0081] In a specific embodiment, when a number of target texts are determined, the target texts are used as target text samples to update the original label system in the system. The specific implementation method is as follows:

[0082] The system further includes: a first service platform and m second service platforms, wherein the first service platform is in communication connection with each of the second service platforms, and the first service platform stores a first text tag set B={B 1 , B 2 , ..., B j , ..., B n}, each of the second service platforms stores a second text tag set and a target text sample set corresponding to the second service platform itself; it can be understood that: the initial text list set is a collection of initial texts obtained from any second service platform, and each second service platform can filter out the corresponding target text sample set.

[0083] The second text label set includes a plurality of second text labels, and the target text sample set includes a plurality of target text samples, wherein B j refers to the jth first text label, j=1, 2, ..., n, n is the number of first text labels, and when the computer program is executed by a processor, the following steps are implemented:

[0084] M100: input each target text sample in the obtained m target text sample sets into the first service platform for labeling, and obtain the first label usage frequency list set F corresponding to the m target text sample sets. 1 ={F 1 1 , F 1 2 , ..., F 1 i , ..., F 1 m}, F 1 i ={F 1 i1 , F 1 i2 , ..., F 1 ij , ..., F 1 in}, F 1 ij For B j The frequency of label usage in the i-th target text sample set, i=1, 2, ..., m.

[0085] Specifically, the first service platform refers to the parent platform of the second service platform; it can be understood that: the label system in the second service platform is constructed with reference to the first service platform, and is expanded with other text labels according to its own region or service characteristics.

[0086] Specifically, F 1 ij Meet the following conditions:

[0087] F 1 ij =L 1 ij / L 1 i , where L 1 ij For B j The number of uses in the i-th target text sample set, L 1 i is the number of target text samples in the i-th target text sample set.

[0088] In another specific embodiment, the target text sample set can also be obtained through the following steps:

[0089] M001, obtaining an initial text list pre-stored in each second service platform; the initial text list includes a plurality of initial texts.

[0090] Specifically, the initial text refers to any text pre-stored in any second service platform.

[0091] M002, according to the preset screening conditions, a number of initial texts are screened out from any initial text list and combined into a target text sample set; it can be understood as: according to any initial text list stored in the second service platform, the corresponding target text sample set is obtained after screening.

[0092] In a specific implementation, in step M002, the target text sample can also be determined by the following steps:

[0093] M0021, obtaining the text priority of each initial text in each initial text list according to the content data information of each initial text in the m initial text lists.

[0094] Specifically, the text priority of the initial text meets the following conditions:

[0095] W = ε 1 ×(K 1 / K 2 )+ε 2 ×(K 2 / K 3 )+ε 3×(1 / K 4 ), where W refers to the text priority of the corresponding initial text, K 1 is the total number of characters of several keywords in the corresponding initial text, K 2 is the total number of characters of the corresponding initial text, K 3 is the average number of characters of several initial texts in the m initial text list, K 4 is the difference between the current time and the creation time of the corresponding initial text, ε 1 is the preset first weight coefficient, ε 2 is the preset second weight coefficient, ε 3 is the preset third weight coefficient.

[0096] Specifically, the several keywords in the initial text refer to several keywords obtained from the initial text through a keyword extraction model.

[0097] M0023, sort the text priorities corresponding to the initial texts from high to low, and determine the initial texts corresponding to the first η text priorities as target text samples, where η is a preset text screening quantity threshold.

[0098] As described above, by calculating the text priorities of all initial texts in all initial text lists, a text screening quantity threshold number of initial texts can be screened out as target text samples according to the text priorities, so that the screened target text samples are at the forefront of priority, which is conducive to the analysis of text data.

[0099] In another specific implementation, in step M002, the target text sample is determined by the following steps:

[0100] M0022, obtaining the text priority of each initial text in the m initial text lists according to the content data information of each initial text in each initial text list; wherein, the method for obtaining the text priority of the initial text is consistent with the method for obtaining in step M0021, which will not be repeated here.

[0101] M0024, according to a preset text priority threshold, select a number of intermediate texts corresponding to the initial text list from each initial text list; the intermediate text refers to any initial text whose corresponding text priority is greater than the preset text priority threshold.

[0102] M0026, determining the number of target texts according to the number of intermediate texts corresponding to each initial text list.

[0103] Specifically, the target text quantity refers to a quantity that is not greater than the minimum intermediate text quantity, wherein the minimum intermediate text quantity refers to the minimum value of the m intermediate text quantities screened out from the m initial text lists; it can be understood that: each initial text list corresponds to an intermediate text quantity.

[0104] M0028, sorting the text priorities of several initial texts corresponding to any initial text list, and determining the intermediate texts corresponding to the text priorities of the first target text as the target text samples.

[0105] As described above, by determining the number of target texts and taking the number of target texts as the standard, the same number of initial texts can be screened out from each initial text list as target text samples, so that the same number of samples can be selected according to the differences in texts between different second service platforms to ensure the comprehensiveness of text analysis.

[0106] M200, for F 1 i The n first tag usage frequencies are sorted, and the target tag usage frequency corresponding to the i-th second service platform is determined according to a preset frequency determination condition.

[0107] Specifically, the preset frequency determination condition refers to: determining the value at the target position in the sorting result as the target tag usage frequency.

[0108] Specifically, the target position refers to a position determined by those skilled in the art in the art in the sorting result of the first tag usage frequency according to actual needs; for example, the target position may be the lower quartile in the sorting result of the first tag usage frequency.

[0109] M300: Input the obtained m target text sample sets into m second service platforms for labeling, and obtain the second label usage frequency list set F corresponding to the m target text sample sets. 2 ={F 2 1 , F 2 2 , ..., F 2 i , ..., F 2 m}, F 2 i ={F 2 i1 , F 2 i2 , ..., F 2 iu , ..., F 2 iq(i)}, F 2 iu={F 2 iu1 , F 2 iu2 , ..., F 2 iux , ..., F 2 ium}, where F 2 iux It refers to the label usage frequency of the u-th second text label in the i-th second text label set in the x-th target text sample set, u=1, 2, ..., q(i), q(i) is the number of second text labels in the i-th second text label set, x=1, 2, ..., m.

[0110] Specifically, F 2 iux Meet the following conditions:

[0111] F 2 iux =L 2 iux / L 2 x , where L 2 iux is the number of uses of the u-th second text label in the i-th second text label set in the x-th target text sample set, L 2 x is the number of target text samples in the xth target text sample set.

[0112] M400, according to F 2 iu Obtain a third tag usage frequency list; the third tag usage frequency list includes y third tag usage frequencies, the third tag usage frequency refers to F 2 iu Any second tag usage frequency in which is greater than the target tag usage frequency.

[0113] M500, when y / m≥θ, F 2 iu The corresponding second text label is determined as a key text label, so that the determined key text labels are added to B and a new label system is constructed, wherein θ is a preset ratio threshold; for example, 80%.

[0114] As mentioned above, when the proportion of the number of second tag usage frequencies that are greater than the target tag usage frequency among the several second tag usage frequencies corresponding to any second text tag is not less than a preset ratio threshold, this second text tag is considered to be an important tag, and then this second text tag is added to the first text tag set to construct a more comprehensive and effective new tag system.

[0115] In a specific embodiment, step M500 further includes the following steps:

[0116] M501, traverse B and several key text tags, when there is a key text tag that is the same as any first text tag in B, delete the key text tag that is the same as any first text tag in B.

[0117] M502, combines the remaining key text labels with B to construct a new label system.

[0118] In the above, by filtering out the key text tags that are the same as any first text tag from the filtered key text tags, there are no duplicate tags in the constructed new tag system, thus avoiding duplicate storage of tags.

[0119] In another specific embodiment, when constructing the new tag system, in order to further improve the effectiveness of the filtered key text tags, the key text tags are further filtered. The specific implementation method is as follows:

[0120] L100, input each initial text in the initial text list set into the first service platform for labeling, and obtain the first text list set B 1 ={B 1 1 , B 1 2 , ..., B 1 j , ..., B 1 n}, B 1 j ={B 1 j1 , B 1 j2 , ..., B 1 jd , ..., B 1 jh(j)}, where B 1 jd It refers to the dth initial text corresponding to the jth first text label, d=1, 2, ..., h(j), h(j) is the number of initial texts corresponding to the jth first text label; it can be understood that: the first text refers to any initial text marked with any first text label.

[0121] Specifically, the initial text list set is TE={TE 1 ,TE 2 ,……,TE r ,……,TE g}, where TEr It refers to the rth initial text, r=1, 2,…, g, where g is the number of initial texts.

[0122] As mentioned above, by inputting each initial text into the first service platform for labeling, several first text labels corresponding to each initial text can be obtained, and therefore several initial texts corresponding to each first text label can also be obtained. By obtaining the number of initial texts corresponding to each first text label, it is helpful to measure the importance of each first text label.

[0123] L200, based on the text priority of each initial text and B 1 , get the first text label score set B corresponding to B 0 ={B 0 1 , B 0 2 , ..., B 0 j , ..., B 0 n}, where B 0 j For B j The corresponding first text label score.

[0124] Among them, B 0 j Meet the following conditions:

[0125] B 0 j =(h(j) / g)×log(g / ∑ g r=1 (λ jr / W r )), where W r refers to the text priority of the rth initial text, λ jr The value of is 0 or 1. When the rth initial text tag has the jth first text tag, λ jr is 1, otherwise, λ jr is 0.

[0126] As described above, by obtaining the number of initial texts corresponding to each first text label and the priority of each initial text, the score of each first text label can be calculated. The score can reflect the importance of the first text label, which is conducive to achieving a comprehensive evaluation of the importance of the label.

[0127] L300, for B 0 The n first text label scores in the set are sorted, and the target label score is determined according to a preset score determination condition.

[0128] Specifically, the preset score determination condition refers to: determining the value at the target position in the sorting result as the target tag score.

[0129] L400, input each initial text in TE to m second service platforms for labeling, and obtain the second text list set B 2 ={B 2 1 , B 2 2 , ..., B 2 a , ..., B 2 c}, where B 2 a It refers to the ath second text list, and the ath second text list refers to a list composed of several initial texts corresponding to the ath key text tag; it can be understood that: the m second service platforms all receive g initial texts in the initial text list set.

[0130] Specifically, the system stores a key text tag set TA={TA 1 , T.A. 2 ,……,TA a ,……,TA c}, the ath key text label is TA a , a=1, 2, ..., c, c is the number of key text tags; it can be understood that: when the initial text is input to the second service platform for tagging, a second text tag consistent with the ath key text tag is obtained from the second service platform, and a number of initial texts corresponding to the second text tag are obtained as B 2 a .

[0131] L500, based on the text priority of each initial text and B 2 , get the second text label score set TA corresponding to TA 0 ={TA 0 1 ,TA 0 2 ,……,TA 0 a ,……,TA 0 c}, where TA 0 a For TA a The corresponding second text label score.

[0132] Specifically, the method for obtaining the second text label score is the same as the method for obtaining the first text label score, which will not be described in detail here.

[0133] L600, when TA 0 a When the score of the target tag is not less than the target tag score, TA a Determine the target text label.

[0134] As described above, by inputting the initial text into the first service platform for labeling, and calculating several first text label scores according to the labeling result and the text priority of the initial text, the set target label score is made more reasonable, which is conducive to screening out more valuable target text labels.

[0135] In a specific embodiment, after step L600, the following steps are also included:

[0136] L700, traverse B and several target text labels determined from TA, and when there is a target text label that is the same as any first text label in B, delete the target text label that is the same as any first text label in B.

[0137] L800, combines the remaining target text labels with B to construct a new label system.

[0138] As described above, by filtering out the target text tags that are the same as any first text tag from the filtered target text tags, there are no duplicate tags in the constructed new tag system, thus avoiding duplicate storage of tags.

[0139] As a parallel embodiment, the target text label can also be obtained by the following implementation method, including the following steps:

[0140] P100, from TE r Get TE r The corresponding keyword intermediate information set; the TE r The corresponding keyword intermediate information set includes TE r The number of initial text keywords in z r TE r The number of initial text keywords repeated in r 、z r The average number of characters of initial text keywords z 0 r and the number of characters in the intermediate text segment between each two adjacent initial text keywords.

[0141] For a better understanding, the number of repeated initial text keywords is given as an example: rWhen there are six initial text keywords in total, they are: initial text keyword 1, initial text keyword 1, initial text keyword 1, initial text keyword 2, initial text keyword 2, initial text keyword 3, among which initial text keyword 1 is repeated twice and initial text keyword 2 is repeated once, so the total number of repetitions is 2+1=3; it can also be understood as: for TE r The number of initial text keywords that need to be deleted when deduplicating the initial text keywords in the .

[0142] In a specific embodiment, in step P100, TE is obtained by the following steps: r The number of characters in the intermediate text segment between each two adjacent initial text keywords:

[0143] P101, Get TE r The z initial text keywords in TE r The location information in.

[0144] P102, according to TE r The location information of each initial text keyword in TE r The intermediate text segment between each two adjacent initial text keywords in ; the intermediate text segment is also the preprocessed text segment obtained above.

[0145] P103, according to TE r The intermediate text segment between each two adjacent initial text keywords in the TE r The number of characters in the intermediate text fragment between each two adjacent initial text keywords.

[0146] P200, according to TE r and TE r Corresponding keyword intermediate information set, obtain TE r The corresponding initial text score ψ r .

[0147] Among them, ψ r Meet the following conditions:

[0148] ψ r =(z r ×ρ r × 0 r ) / (z 00 r ×(ζ r +1)), where ρ r For TE r The corresponding target number of characters, z 00 r For TE rThe average number of characters in the corresponding intermediate text segments.

[0149] In this embodiment, TE r The corresponding target number of characters refers to TE r The number of characters.

[0150] In another specific embodiment, ρ can also be determined by the following steps: r :

[0151] P201, obtain the number of characters of each initial text in TE.

[0152] P202, according to the number of characters of some initial text, TE r The number of characters is normalized to between 0 and λ, and the normalized result is determined as ρ r , where λ is the preset normalization threshold.

[0153] As mentioned above, when calculating the initial text score, taking into account the situation that other parameters are small and relatively close, while the number of characters in the initial text is large, the method of normalizing the number of characters in the initial text is adopted. This can effectively reduce the influence of the number of characters in the initial text on the calculated initial text score and improve the rationality of calculating the initial text score.

[0154] P300, based on TA, labels each initial text in TE and obtains the second text list set B 2 ={B 2 1 , B 2 2 , ..., B 2 a , ..., B 2 c}, where B 2 a It refers to the ath second text list, and the ath second text list refers to a list composed of several initial texts corresponding to the ath key text tag.

[0155] P400, based on the initial text score and B corresponding to each initial text 2 , get the second text label score set ST corresponding to TA = {ST 1 ,ST 2 , ..., ST a , ..., ST c}, where ST a For TA a The corresponding second text label score.

[0156] Specifically, in step P400, ST a Meet the following conditions:

[0157] ST a =(ξ(a) / g)×log(g / ∑ g r=1 (λ ar / ψ r ), where λ ar The value of is 0 or 1. When the rth initial text is marked with the ath key text label, λ ar is 1, otherwise, λ ar is 0, ξ(a) is B 2 a The number of initial texts in .

[0158] P500, when ST a When the score is not less than the preset tag score, the TA is obtained. a The target text label.

[0159] Specifically, in one embodiment, the preset tag score is a score set by a person skilled in the art according to actual needs or experience. In a parallel embodiment, the text priority in step E200 is replaced with the initial text score, that is, W r Replace with ψ r The target label score obtained after calculating the first text label score is used as the preset target label score. The calculation process is the same and will not be repeated here.

[0160] In a specific embodiment, each of the second service platforms is correspondingly provided with a preset label model for labeling text, and when the computer program is executed by the processor, the following steps are also implemented:

[0161] P001, receiving a plurality of first predicted text labels marked by a target user for any preset text in a preset text set.

[0162] Specifically, the preset text refers to any initial text in the initial text list set or any text among several initial texts randomly selected from the initial text list set.

[0163] P002, input any of the preset texts into several preset label models, and obtain several second predicted text labels marked by each preset label model for the any of the preset texts; it can be understood that: the any of the preset texts here refers to the preset text marked with several first preset text labels in step P001.

[0164] P003, according to the plurality of first predicted text labels and the plurality of second predicted text labels marked by each preset label model for any preset text, obtain the label overlap rate corresponding to each preset label model.

[0165] In a specific embodiment, in step P003, the label overlap rate corresponding to any preset label model is obtained through the following steps:

[0166] P0031, traverse a number of first predicted text labels, and obtain a third predicted text label from a number of second predicted text labels corresponding to any of the preset label models; the third predicted text label refers to a second predicted text label that is the same as any first predicted text label among the number of second predicted text labels corresponding to any of the preset label models.

[0167] Specifically, the plurality of second predicted text labels corresponding to the preset label model refer to the plurality of second predicted text labels marked by the preset label model for any preset text.

[0168] P0032, according to the number of the third predicted text labels and the number of the first predicted text labels, obtain the label overlap rate corresponding to any one of the preset label models.

[0169] Specifically, the label overlap rate corresponding to any of the preset label models meets the following conditions:

[0170] RE v =φ v / φ, where RE v It refers to the label overlap rate corresponding to the vth preset label model, φ v refers to the same number of the second predicted text labels as the first predicted text labels, and φ refers to the number of the first predicted text labels.

[0171] As mentioned above, for a preset text, each preset label model can obtain a corresponding label overlap rate. The higher the label overlap rate, the higher the label accuracy of the preset label model, which is conducive to obtaining the priority of several preset label models according to the preset text set.

[0172] In a preferred embodiment, the key text tag set TA can also be obtained through the following steps:

[0173] P1, according to the number of preset texts in the preset text set, obtain several label overlap rates corresponding to any preset label model; it can be understood that: any preset label model labels each preset text separately, and several label overlap rates corresponding to the preset label model will be obtained.

[0174] P2, normalize the label overlap rates corresponding to each preset label model to obtain the label model priority corresponding to each preset label model; a person skilled in the art is aware of any implementation method of normalization in the prior art, for example, a method of averaging the label overlap rates, which will not be described here.

[0175] P3, when g≥γ, the preset label model with the highest corresponding label model priority is determined as the target label model, and several text labels in the second service platform corresponding to the target label model are used as TA, where γ is a preset quantity threshold.

[0176] P4, when g<γ, a preset number of preset label models are screened out according to the label model priority, and text labels in a plurality of second service platforms corresponding to the preset number of preset label models are merged as TA.

[0177] As mentioned above, taking into account the amount of initial text, when the amount of initial text is small, the text tags in several second service platforms corresponding to the multiple preset tag models screened out are used as key text tags to increase the number of key text tags, making the tags for the initial text more comprehensive. When the amount of initial text is large, using a preset tag model can reduce the complexity of operations and improve tagging efficiency.

[0178] In another specific embodiment, the rationality of the acquired second predicted text label can also be verified, that is, after step P003, the following steps are also included:

[0179] P004, based on a number of second predicted text labels of each preset label model for any preset text tag, the label overlap rate corresponding to each preset label model and the preset model credibility of each preset label model, obtain the credibility priority of each second predicted text label; it can be understood that: the preset label model is an open source label model carrying credibility information, and the model credibility can also be tested through samples.

[0180] In a specific embodiment, in step P004, the credibility priority of any second predicted text label is obtained through the following steps:

[0181] P0041, according to the preset model credibility of each preset label model and a number of second predicted text labels corresponding to each preset label model, obtain the target model credibility of each preset label model.

[0182] Among them, the target model credibility of any preset label model meets the following conditions:

[0183] CR 0 v =CR v ×(QU 1 / QU), where CR 0 v is the target model credibility of the vth preset label model, CR vis the preset model credibility of the vth preset label model, QU 1 To mark the number of preset label models of any second predicted text label, QU is the number of preset label models.

[0184] P0042, obtaining the credibility priority of any second predicted text label according to the label overlap rate corresponding to each preset label model and the target model credibility of each preset label model.

[0185] Specifically, the credibility priority of any second predicted text label meets the following conditions:

[0186] ω=∑ QU v=1 (RE v ×CR 0 v ).

[0187] As described above, the preset model credibility of each preset label model is adjusted by marking the number of preset label models of any second predicted text label and the total number of preset label models. That is, the fewer the number of preset label models that type out a second predicted text label, the less credible the typed second predicted text label is, which will reduce the credibility of the preset label model that types out the second predicted text label. Therefore, the credibility of the preset label model needs to be adjusted to make the calculated credibility priority more accurate.

[0188] P005, when there is a second predicted text label whose credibility priority is greater than a preset credibility priority threshold, the result of verifying that the second predicted text label whose corresponding credibility priority is greater than the preset credibility priority threshold meets the preset requirements; it can be understood that: verifying that the result of the second predicted text label meets the preset requirements means that the second predicted text label is a reasonable label.

[0189] In a specific embodiment, the credibility priority threshold is obtained through the following steps:

[0190] P0051, receiving a number of target processing texts and obtaining the credibility priority of each fourth predicted text label among a number of fourth predicted text labels corresponding to each target processing text; the method for obtaining the credibility priority of the fourth predicted text label is consistent with the method for obtaining the credibility priority of the second predicted text label.

[0191] Specifically, the target processing text refers to any initial text randomly selected from the initial text list set.

[0192] P0052, sorting the credibility priorities of a plurality of fourth predicted text labels from small to large to obtain a priority sorting result.

[0193] P0053, determining the credibility priority at the target ranking position in the priority ranking result as the credibility priority threshold.

[0194] As mentioned above, by taking several target processed texts as sample sets, several corresponding fourth predicted text labels and the credibility priority of each fourth predicted text label are obtained, and a credibility priority threshold is determined according to several credibility priorities, so that the set credibility priority threshold is more reasonable, which can improve the accuracy and rationality of label verification.

[0195] The embodiment of the present invention also provides a system for determining a target text, such as Figure 2 As shown, the system comprises:

[0196] The keyword and text segment acquisition module 100 is used to acquire an initial text keyword set and an initial text segment set corresponding to each initial text from the received initial text list set.

[0197] Specifically, the initial text list set includes a number of initial texts; the initial texts correspond to the initial text keyword set one by one; the initial text keyword set includes a number of initial text keywords; and the initial text segment set includes a number of initial text segments.

[0198] In a specific embodiment, Figure 3 As shown, the keyword and text segment acquisition module 100 includes:

[0199] The keyword extraction module 101 is used to extract a number of initial text keywords corresponding to the initial text itself from each initial text based on the keyword extraction model. Those skilled in the art are aware of any keyword extraction model in the prior art that can extract keywords from text, which will not be described in detail here.

[0200] The keyword deduplication module 102 is used to perform deduplication processing on the several initial text keywords and combine the remaining initial text keywords into an initial text keyword set.

[0201] In a specific embodiment, Figure 4 As shown, the keyword and text segment acquisition module 100 also includes:

[0202] The first acquisition module 110 is used to remove a plurality of initial text keywords corresponding to each initial text from the corresponding initial text, and acquire a plurality of pre-processed text segments corresponding to each initial text.

[0203] The search module 120 is used to search for a plurality of target redundant words from a plurality of pre-processed text segments corresponding to each initial text according to a preset redundant word list.

[0204] The second acquisition module 130 is used to delete the plurality of target redundant words from the plurality of pre-processed text segments, and acquire an initial text segment corresponding to each pre-processed text segment.

[0205] As mentioned above, after obtaining the initial text keywords, the preprocessed text fragments in the initial text are also obtained and converted into standard initial text fragments. Through the combination of the initial text keywords and the initial text fragments, the generated key text is more accurate and more in line with the original meaning of the initial text, and the standardized text is conducive to processing and analysis.

[0206] The key text acquisition module 200 is used to input the initial text keyword set and the initial text segment set corresponding to each initial text into a preset language processing model to acquire the key text corresponding to each initial text.

[0207] The text weight acquisition module 300 is used to acquire the text weight corresponding to each initial text according to a number of initial texts, an initial text keyword set corresponding to each initial text, and a key text corresponding to each initial text.

[0208] Specifically, the text weight corresponding to the initial text meets the following conditions:

[0209] W 00 r =W 0 r / max(W 0 1 ,W 0 2 ,……,W 0 r ,……,W 0 g ), where W 00 r is the text weight corresponding to the rth initial text in the initial text list, W 0 r =δ r1 ×N r ×δ r2 ×(δ r1 / δ rg ), δ r1 is the number of characters in the rth initial text, N r is the number of initial text keywords corresponding to the rth initial text, δ r2is the number of characters of the key text corresponding to the rth initial text, the value range of r is 1 to g, and g is the number of initial texts in the initial text list set.

[0210] The target text determination module 400 is used to determine the initial text whose corresponding text weight is greater than the preset text weight threshold as the target text when there is an initial text whose corresponding text weight is greater than the preset text weight threshold.

[0211] The above provides a technical solution for obtaining the corresponding key text based on the initial text keywords and the initial text fragments, and then calculating the text weight of the corresponding initial text based on the initial text and the key text. This calculation method makes full use of the structure and content information of the text to achieve a comprehensive evaluation of the text, and can quickly determine the effective and high-value target text, which is conducive to the processing and analysis of large amounts of text data.

[0212] In a specific embodiment, Figure 5 As shown, the target text determination module 400 also includes:

[0213] The third acquisition module 401 is used to receive a number of preprocessed texts and obtain the text weight corresponding to each preprocessed text; the method for obtaining the text weight corresponding to the preprocessed text is consistent with the method for obtaining the text weight corresponding to the initial text.

[0214] The sorting module 402 is used to sort the text weights corresponding to the plurality of pre-processed texts from small to large to obtain a weight sorting result.

[0215] The weight threshold determination module 403 is used to determine the weight of the text at the target sorting position in the weight sorting result as a preset text weight threshold.

[0216] Specifically, the target ranking position refers to the position determined by technicians in this field in the weight ranking results according to actual needs; for example, it can be the middle position or the value at the 80% position.

[0217] In the above, by taking a number of preprocessed texts as sample sets and determining the text weight threshold according to the text weights of the preprocessed texts, the set text weight threshold is made more reasonable, which is conducive to selecting effective target texts.

[0218] Although some specific embodiments of the present invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will also be appreciated by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for determining a target text, characterized in that: The method comprises the following steps: S100, acquiring an initial text keyword set and an initial text segment set corresponding to each initial text from the received initial text list set; S200, inputting an initial text keyword set and an initial text segment set corresponding to each initial text into a preset language processing model to obtain a key text corresponding to each initial text; S300, obtaining a text weight corresponding to each initial text according to a plurality of initial texts, an initial text keyword set corresponding to each initial text, and a key text corresponding to each initial text; S400: When there is an initial text whose corresponding text weight is greater than a preset text weight threshold, the initial text whose corresponding text weight is greater than the preset text weight threshold is determined as a target text.

2. The method for determining a target text according to claim 1, characterized in that: Step S100 includes the following steps: S101, extracting a number of initial text keywords corresponding to the initial text itself from each initial text based on a keyword extraction model; S102, performing deduplication processing on the several initial text keywords, and combining the remaining initial text keywords into an initial text keyword set.

3. The method for determining the target text according to claim 2, characterized in that: Step S100 also includes the following steps: S110, removing a plurality of initial text keywords corresponding to each initial text from the corresponding initial text, and obtaining a plurality of pre-processed text segments corresponding to each initial text; S120, searching for a plurality of target redundant words from a plurality of preprocessed text segments corresponding to each initial text according to a preset redundant word list; S130, deleting the plurality of target redundant words from the plurality of preprocessed text segments, and obtaining an initial text segment corresponding to each preprocessed text segment.

4. The method for determining a target text according to claim 1, characterized in that: In step S400, the preset text weight threshold is obtained through the following steps: S401, receiving a plurality of preprocessed texts and obtaining a text weight corresponding to each preprocessed text; the method for obtaining the text weight corresponding to the preprocessed text is consistent with the method for obtaining the text weight corresponding to the initial text; S402, sorting the text weights corresponding to the plurality of preprocessed texts from small to large to obtain a weight sorting result; S403: Determine the text weight at the target sorting position in the weight sorting result as a preset text weight threshold.

5. A target text determination system, characterized in that: The system comprises: A keyword and text segment acquisition module, used to acquire an initial text keyword set and an initial text segment set corresponding to each initial text from the received initial text list set; A key text acquisition module is used to input the initial text keyword set and the initial text segment set corresponding to each initial text into a preset language processing model to acquire the key text corresponding to each initial text; A text weight acquisition module, used to acquire a text weight corresponding to each initial text according to a number of initial texts, an initial text keyword set corresponding to each initial text, and a key text corresponding to each initial text; The target text determination module is used to determine the initial text whose corresponding text weight is greater than the preset text weight threshold as the target text when there is an initial text whose corresponding text weight is greater than the preset text weight threshold.

6. The target text determination system according to claim 5, characterized in that: The keyword and text fragment acquisition module includes: A keyword extraction module, used for extracting a number of initial text keywords corresponding to the initial text itself from each initial text based on the keyword extraction model; The keyword deduplication module is used to perform deduplication processing on the several initial text keywords and combine the remaining initial text keywords into an initial text keyword set.

7. The target text determination system according to claim 6, characterized in that: The keyword and text fragment acquisition module also includes: A first acquisition module is used to remove a number of initial text keywords corresponding to each initial text from the corresponding initial text, and acquire a number of pre-processed text segments corresponding to each initial text; A search module, used to search for a plurality of target redundant words from a plurality of pre-processed text segments corresponding to each initial text according to a preset redundant word list; The second acquisition module is used to delete the plurality of target redundant words from the plurality of preprocessed text segments, and acquire an initial text segment corresponding to each preprocessed text segment.

8. The target text determination system according to claim 5, characterized in that: The target text determination module comprises: A third acquisition module is used to receive a plurality of preprocessed texts and acquire a text weight corresponding to each preprocessed text; the method for acquiring the text weight corresponding to the preprocessed text is consistent with the method for acquiring the text weight corresponding to the initial text; A sorting module is used to sort the text weights corresponding to a number of preprocessed texts from small to large to obtain a weight sorting result; The weight threshold determination module is used to determine the weight of the text at the target sorting position in the weight sorting result as a preset text weight threshold.

Images