Text processing method and device, equipment and storage medium

Abstract

The present disclosure provides a text processing method and device, equipment and storage medium, relates to the field of data processing, in particular to the field of artificial intelligence and big data processing. The specific implementation scheme is: selecting an initial to-be-processed sentence containing a target keyword from a plurality of initial to-be-processed sentences as a candidate to-be-processed sentence, obtaining a candidate set containing at least one candidate to-be-processed sentence; wherein the target keyword is selected from a plurality of initial keywords in a keyword set; the initial keyword is a keyword belonging to a target field; comparing the candidate to-be-processed sentence with a target key sentence containing the target keyword to obtain first similarity information; and taking the candidate to-be-processed sentence satisfying a first similarity rule as a target to-be-processed sentence belonging to the target field. In this way, the recognition efficiency and accuracy are improved, and technical support is provided for purifying the network environment.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to the fields of artificial intelligence and big data processing. BACKGROUND

[0002] The problem of network prohibited illegal text circulating in the Internet information center affects the social harmony and stability. Therefore, how to effectively identify prohibited illegal text has become an indispensable topic for purifying the network environment. SUMMARY

[0003] The present disclosure provides a text processing method, device, equipment and storage medium.

[0004] According to an aspect of the present disclosure, a text processing method is provided, comprising:

[0005] Filtering an initial to-be-processed sentence containing a target keyword from a plurality of initial to-be-processed sentences as a candidate to-be-processed sentence, to obtain a candidate set containing at least one candidate to-be-processed sentence; wherein the target keyword is selected from a plurality of initial keywords in a keyword set; and the initial keyword is a keyword belonging to a target domain;

[0006] Comparing the candidate to-be-processed sentence with a target key sentence containing the target keyword to obtain first similarity information;

[0007] Taking the candidate to-be-processed sentence satisfying a first similarity rule of the first similarity information as a target to-be-processed sentence belonging to the target domain.

[0008] According to another aspect of the present disclosure, a text processing device is provided, comprising:

[0009] A filtering unit is configured to filter an initial to-be-processed sentence containing a target keyword from a plurality of initial to-be-processed sentences as a candidate to-be-processed sentence, to obtain a candidate set containing at least one candidate to-be-processed sentence; wherein the target keyword is selected from a plurality of initial keywords in a keyword set; and the initial keyword is a keyword belonging to a target domain;

[0010] A comparison unit is configured to compare the candidate to-be-processed sentence with a target key sentence containing the target keyword to obtain first similarity information;

[0011] A target determination unit is configured to take the candidate to-be-processed sentence satisfying a first similarity rule of the first similarity information as a target to-be-processed sentence belonging to the target domain.

[0012] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein

[0015] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0016] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method described above.

[0017] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method described above.

[0018] In this way, the present solution can effectively identify target processing sentences belonging to a target field, improve the identification efficiency and accuracy, and provide technical support for purifying the network environment.

[0019] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not limit the present disclosure. Among them:

[0021] Figure 1 is an implementation flowchart of a text processing method according to an embodiment of the present disclosure;

[0022] Figures 2(a) to 2(d) is an implementation flowchart of constructing a field sentence library and a field word library according to a text processing method in a specific example according to an embodiment of the present disclosure;

[0023] Figures 3(a) and 3(b) are flowcharts of identifying a text to be processed according to a text processing method according to an embodiment of the present disclosure;

[0024] Figure 4 is a structural schematic diagram of a text processing device according to an embodiment of the present disclosure

[0025] Figure 5 is a block diagram of an electronic device for implementing a text processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are cited as illustrative examples. Various details of the embodiments of the present disclosure are described herein in order to provide a thorough understanding thereof. It will be understood by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in the following description, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

[0027] The present disclosure provides a text processing method, specifically, as shown in the following steps: Figure 1

[0028] Step S101: Selecting an initial to-be-processed sentence containing a target keyword from a plurality of initial to-be-processed sentences as a candidate to-be-processed sentence, and obtaining a candidate set containing at least one candidate to-be-processed sentence.

[0029] Here, the target keyword is selected from a plurality of initial keywords in a keyword set; the initial keyword is a keyword belonging to a target domain; based on this, the target keyword is also a keyword belonging to the target domain, which lays a foundation for subsequent efficient screening.

[0030] Step S102: Comparing the candidate to-be-processed sentence with a target keyword sentence containing the target keyword to obtain first similarity information.

[0031] Step S103: Taking the candidate to-be-processed sentence whose first similarity information satisfies a first similarity rule as a target to-be-processed sentence belonging to the target domain.

[0032] In a specific example, the candidate to-be-processed sentence and the target keyword sentence can be subjected to word feature vectorization processing, and then the feature vector of the candidate to-be-processed sentence and the feature vector of the target keyword sentence are obtained. The feature vector of the target keyword sentence is compared with the feature vector of the candidate to-be-processed sentence in terms of similarity, such as measuring the similarity between sentences based on cosine similarity, to obtain the first similarity information. In this way, it is determined whether the candidate to-be-processed sentence and the target keyword sentence compared are similar sentences, and the comparison result is highly interpretable.

[0033] It should be noted that in the process of word feature vectorization, the weight of the target domain word in the target keyword sentence can be increased, so as to determine the target to-be-processed sentence belonging to the target domain from the plurality of initial to-be-processed sentences.

[0034] ​In actual application, the first similarity information corresponding to the target processing sentence, the feature vector of the target processing sentence, and the feature vector of the target key sentence in a similar sentence relationship with the target processing sentence can also be output, so as to further improve the interpretability of the comparison result (i.e., the first similarity information).

[0035] It can be understood that the first similarity rule in the present disclosure can be based on the actual needs of the actual scene, and the present disclosure does not limit this. For example, the first similarity rule can be a threshold, such as a first threshold. At this time, the two sentences corresponding to the first similarity information greater than the first threshold can be regarded as similar sentences, and the candidate to-be-processed sentence corresponding to the similar sentence is the target processing sentence.

[0036] In this way, the present disclosure can effectively identify the initial to-be-processed sentence, and the identification process can anchor the target field, so as to achieve the purpose of effectively identifying the target processing sentence belonging to the target field, and provide technical support for purifying the network environment.

[0037] Moreover, since the present disclosure can identify based on the sentence dimension and obtain the target processing sentence, it has strong identification ability for sentences with large variants, improves the overall identification efficiency and identification accuracy, and the result has strong interpretability.

[0038] In a specific example of the present disclosure, the initial to-be-processed sentence can be obtained in the following manner. Specifically, the to-be-processed text is preprocessed to obtain the plurality of initial to-be-processed sentences. For example, the stop words, punctuation marks, special symbols, etc. in the to-be-processed text are replaced with spaces, and then the to-be-processed text is processed based on the obtained spaces to obtain a plurality of initial to-be-processed sentences. In actual application, there can be other sentence processing schemes, and the present disclosure does not limit this.

[0039] In this way, the identification and screening of sentences related to the target field in the to-be-processed text, such as the target processing sentence belonging to the target field, are realized, which provides technical support for purifying the network environment. At the same time, the present disclosure is applied to the text field, i.e., the application scenario is expanded, which lays a foundation for engineering application and popularization.

[0040] In a specific example of the present disclosure, in the case of obtaining a target processing sentence belonging to the target field, the to-be-processed text can be marked in at least one of the following manners, specifically including: marking the target processing sentence; marking the paragraph where the target processing sentence is located. That is, after obtaining the target processing sentence, the target processing sentence in the to-be-processed text can be marked, or the paragraph where the target processing sentence is located can be marked, or both the target processing sentence in the to-be-processed text and the paragraph where the target processing sentence is located can be marked. Here, the marking can be in the form of font enlargement, highlighting, underlining, or a marked box, and the present disclosure does not make a specific limitation on the marking manner. In this way, technical support is provided for effectively intercepting prohibited and illegal texts, and efficiency is improved.

[0041] In a specific example of the present disclosure, the above-mentioned filtering of the initial to-be-processed sentence containing the target keyword from the plurality of initial to-be-processed sentences as a candidate to-be-processed sentence, and obtaining a candidate set containing at least one of the candidate to-be-processed sentences can specifically include:

[0042] The initial to-be-processed sentence containing the target keyword is filtered from the plurality of initial to-be-processed sentences as a candidate to-be-processed sentence; the initial to-be-processed sentence containing the preset keyword is filtered from the plurality of initial to-be-processed sentences as a non-candidate processing sentence; here, the preset keyword is one of the preset keywords in the preset keyword set; the preset keyword is a keyword not belonging to the target field; further, based on the candidate to-be-processed sentence and other initial to-be-processed sentences in the plurality of initial to-be-processed sentences except for the non-candidate processing sentence, a candidate set is obtained. That is, the candidate to-be-processed sentence and other initial to-be-processed sentences in the plurality of initial to-be-processed sentences except for the non-candidate processing sentence are all added to the candidate set for further screening.

[0043] That is, based on the target keyword and the preset keyword, the plurality of initial to-be-processed sentences are classified. Here, it needs to be clear that the target keyword is a keyword belonging to the target domain, and the preset keyword is a specified keyword not belonging to the target domain (the specified keyword can be a word having a large gap with the target domain, or can be a word related to the target domain but excluded from the target domain after manual screening); in this way, candidate to-be-processed sentences and non-candidate to-be-processed sentences are obtained. In actual application, there can be a case that an initial to-be-processed sentence is neither identified as a candidate to-be-processed sentence nor as a non-candidate to-be-processed sentence. At this time, in order to further expand the screening range and avoid missing, the candidate to-be-processed sentences and other initial to-be-processed sentences except the non-candidate to-be-processed sentences in the plurality of initial to-be-processed sentences can be added to the candidate set, so as to expand the screening and identification range and lay a foundation for effectively avoiding missing.

[0044] In another specific example, the initial to-be-processed sentence that is neither identified as a candidate to-be-processed sentence nor as a non-candidate to-be-processed sentence can also be directly excluded, that is, identified as a non-candidate to-be-processed sentence; in other words, only the candidate to-be-processed sentence obtained by screening the target keyword is added to the candidate set for next step screening, so as to improve the screening efficiency.

[0045] In a specific example of the present disclosure scheme, the target keyword can also be obtained in the following manner; specifically, based on the historical text of the target domain, a plurality of initial keywords belonging to the target domain are determined, and the keyword set is obtained; the initial keywords in the keyword set are sorted, and at least one target keyword is selected from the plurality of initial keywords based on the sorting result.

[0046] For example, the initial keywords in the keyword set are sorted, such as sorting the initial keywords in the keyword set from high to low based on the importance or priority, selecting the initial domain words located in the front of the preset number as the target domain words based on the sorting result, and screening the initial to-be-processed sentences based on the target domain words. In this way, the screening efficiency can be effectively improved on the basis of ensuring the identification accuracy.

[0047] In a specific example of the present disclosure, the initial keywords can be obtained in the following manner. Specifically, the initial keywords belonging to the target domain are determined based on the historical text of the target domain, including: obtaining a plurality of feature words, i.e., a plurality of domain feature words, belonging to the target domain based on the historical text of the target domain; and then directly taking the feature words as the initial keywords, or taking the feature words and similar words of the feature words as the initial keywords. Here, the historical text is the text identified as belonging to the target domain. In this way, the initial keywords are determined from the historical text, which can effectively improve the recognition efficiency. Moreover, when the similar words of the feature words are also taken as the initial keywords, the range of the initial keywords can be effectively expanded, laying a foundation for subsequent improvement of the recognition accuracy.

[0048] In a specific example of the present disclosure, the similar words can be obtained in the following manner, specifically including: performing vector processing on the historical text to obtain a feature vector of the feature word; and selecting a word from a segmentation dictionary established based on the historical text, which satisfies a preset rule in similarity with the feature vector of the feature word, as the similar word of the feature word. In this way, the range of the initial keywords can be effectively expanded, so that the present disclosure has strong generalization ability, laying a foundation for subsequent improvement of the recognition accuracy. Moreover, the expansion manner has strong interpretability.

[0049] It should be noted that the present disclosure does not limit the manner of vector processing, as long as the feature vector of the feature word can be obtained, and the similar words can be obtained based on the vector similarity matching of the feature vector.

[0050] In a specific example of the present disclosure, the segmentation dictionary can be constructed in the following manner, specifically including: obtaining a plurality of initial words based on the historical text of the target domain, and obtaining at least one feature word belonging to the target domain; and constructing a segmentation dictionary containing the plurality of initial words and the at least one feature word.

[0051] For example, a self-defined dictionary can be constructed first. For example, the key words (including key phrases) are mined from the historical text of the target domain by using cohesion degree, mutual information, etc., and added to the self-defined dictionary. The historical text of the target domain can also be segmented by using a hidden Markov model (HMM), and a plurality of segmentation results (also referred to as key words) obtained after the segmentation are added to the self-defined dictionary. In actual applications, other manners can also be used to mine the key words from the historical text of the target domain, and the present disclosure does not limit this.

[0052] Further, it needs to be noted that the keywords contained in the custom dictionary can be keywords of the target domain or can not be keywords of the target domain; here, the purpose of constructing the custom dictionary is to find words (including phrases, word groups, etc.) so as to subsequently find initial keywords belonging to the target domain in the dimension of words. Here, the keywords in the custom dictionary are initial words.

[0053] Further, the seed words (also including seed phrases, i.e., feature words) of the target domain are further obtained by small-batch screening from the historical text of the target domain.

[0054] A segmentation dictionary is constructed, which contains seed words and keywords (i.e., initial words) in the custom dictionary.

[0055] It needs to be noted that the keywords contained in the segmentation dictionary can be keywords of the target domain or can not be keywords of the target domain; here, the purpose of constructing the segmentation dictionary is to find all the words in the historical text of the target domain so as to determine the keywords belonging to the target domain from them (i.e., in the segmentation dictionary).

[0056] In this way, data foundation is laid for subsequent accurate determination of initial keywords in the dimension of words, and processing efficiency is also improved.

[0057] In a specific example of the present disclosure, the key sentence set can be constructed in the following manner, thereby providing data support for screening from the dimension of sentences, and also providing data support for improving the overall recognition ability of the present disclosure. Specifically, the historical text of the target domain is preprocessed to obtain a plurality of initial sentences; the initial sentences containing the initial keywords are screened from the plurality of initial sentences as initial key sentences to obtain a key sentence set.

[0058] For example, continuous numbers, stop words, punctuation marks, special symbols, etc. in the historical text of the target domain are replaced with spaces. The historical text is processed by sentences according to the preprocessed spaces to obtain a plurality of sentences (i.e., a plurality of initial sentences). The initial sentences obtained after the sentence processing are preliminarily screened by using the initial keywords in the keyword set, and the initial sentences containing the initial keywords screened are used as initial key sentences to obtain a key sentence set.

[0059] The target key sentence described above is one of the initial key sentences contained in the key sentence set. That is, in the case where the target keyword is obtained, the target key sentence containing the target keyword is determined from the key sentence set.

[0060] The initial keyword described in the example can be an initial keyword in the keyword set obtained based on the above manner.

[0061] In a specific example of the present disclosure, the obtained initial key sentences in the key sentence set can be further screened in the following manner to avoid selecting key sentences not belonging to the target domain as initial key sentences, thereby laying a foundation for subsequent improvement of recognition accuracy and laying a foundation for improvement of recognition efficiency. Specifically, it includes:

[0062] The initial key sentences in the key sentence set are compared with preset key sentences to obtain second similarity information; wherein the preset key sentences are key sentences containing key words not belonging to the target domain; the preset key words are one of the preset key words in the preset word set described above; in other words, the preset word set is set in advance, and the preset key words in the preset word set are all specified key words not belonging to the target domain. Further, the preset key sentences are one of the preset key sentences in the preset sentence set; similarly, the preset sentence set is set in advance, and the preset key sentences in the preset sentence set are all key sentences including specified key words not belonging to the target domain.

[0063] Further, the initial key sentences whose second similarity information satisfies the second similarity rule are deleted from the key sentence set. For example, the initial key sentences whose second similarity information is greater than the second threshold value are deleted from the key sentence set, that is, the initial key sentences whose second similarity information is greater than the second threshold value are considered as non-domain sentences.

[0064] For example, the initial domain sentences (i.e. initial key sentences) in the domain sentence library (i.e. key sentence set) are compared with the non-domain sentences (i.e. preset key sentences) in the non-domain sentence library (which can also be called a preset sentence set) to obtain similarity information (i.e. second similarity information), if the obtained similarity information (i.e. second similarity information) satisfies the second similarity rule, the initial domain sentences whose similarity information (i.e. second similarity information) satisfies the second similarity rule (such as similarity exceeding the second threshold value) are deleted from the domain sentence library, and the domain sentence library is updated.

[0065] It should be noted that if the similarity between a specific initial domain sentence and a specific non-domain sentence exceeds the second threshold value, it means that the specific initial domain sentence is not a domain sentence belonging to the target domain, and at this time, the specific initial domain sentence can be deleted from the domain sentence library.

[0066] According to the above manner, each initial domain sentence in the domain sentence library is compared with a non-domain sentence to update the domain sentence library, thereby effectively avoiding the appearance of non-target domain domain sentences in the domain sentence library.

[0067] In addition, it should be noted that in actual application, the initial domain sentences in the domain sentence library obtained based on the above manner can be clustered to reduce the number of initial domain sentences in the domain sentence library, so as to facilitate reducing the retrieval amount in the later stage.

[0068] It should be noted that the historical text in the present disclosure refers to one or more texts, and the number and length of the historical text are not limited by the present disclosure, as long as the text is identified as belonging to the target domain.

[0069] Meanwhile, it should also be noted that the words in the present disclosure include but are not limited to keywords, initial words, feature words, etc., which can include phrases, word groups, etc.

[0070] In this way, the present disclosure can effectively identify the initial processing sentence, and the identification process can anchor the target domain, so as to achieve the purpose of effectively identifying the target processing sentence belonging to the target domain, and provide technical support for purifying the network environment.

[0071] Moreover, since the present disclosure can identify based on the sentence dimension and obtain the target processing sentence, it has strong identification ability for sentences with large variations, and improves the overall identification efficiency and accuracy, and the result is highly interpretable.

[0072] Meanwhile, the present disclosure can also continuously maintain the data sample, i.e., the historical text, so as to further improve the control ability and further improve the identification accuracy.

[0073] The present disclosure will be further described in detail below in combination with specific examples; the detailed steps include:

[0074] First part: constructing a domain sentence library (i.e., a key sentence set) and a domain word library (i.e., a key word set).

[0075] It can be understood that the domain word can be understood as a keyword belonging to a specific domain, such as a pornographic domain, which can be collectively referred to as a domain word of the specific domain; similarly, the domain sentence can be understood as a sentence containing the domain word of the specific domain, which can be referred to as the domain sentence of the specific domain. Here, the specific domain is not limited by the present disclosure, and accordingly, the domain word or domain sentence belonging to the specific domain is also not limited by the present disclosure.

[0076] This example takes the pornographic domain in the content control scene as an example, and specifically constructs the domain word library and the domain sentence library related to the target domain, such as the pornographic domain.

[0077] Here, it can be understood that the domain words contained in the domain word library can be specific words or phrases, and the form of the specific domain words is not limited by the present disclosure. Any word, phrase or short phrase belonging to a specific domain can be used as the domain word described in the present example.

[0078] As shown in FIGS. 2(a) and 2(b), the specific steps include:

[0079] Step 201: Mining initial domain words (i.e., initial keywords) from historical texts belonging to the target domain, and constructing a domain word library based on the mined initial domain words. For example, as shown in FIG. 2(c), the initial domain words can be obtained in the following manner:

[0080] Step 2011: Constructing a custom dictionary. For example, keywords (including key phrases) can be mined from historical texts belonging to the target domain by using cohesion, mutual information, maximum entropy, speech models, etc. (i.e., word mining shown in FIG. 2(b)), and added to the custom dictionary. In addition, the historical texts belonging to the target domain can be processed by using a hidden Markov model (HMM) to perform word segmentation, and a plurality of word segmentation results (which can also be referred to as keywords) obtained after the word segmentation are added to the custom dictionary.

[0081] In actual applications, other ways can also be used to mine keywords from historical texts belonging to the target domain, and the present disclosure does not limit this.

[0082] Further, it should be noted that the keywords contained in the custom dictionary can be keywords of the target domain or can not be keywords of the target domain. Here, the purpose of constructing the custom dictionary is to find words (including phrases, word groups, etc.) so as to subsequently find initial keywords belonging to the target domain in the dimension of words.

[0083] Step 2012: Using a manual screening method to screen a small amount of seed words (including seed phrases, i.e., feature words) belonging to the target domain from historical texts belonging to the target domain.

[0084] Step 2013: Constructing a word segmentation dictionary containing seed words and keywords in the custom dictionary. In other words, the word segmentation dictionary is constructed based on the keywords in the custom dictionary obtained in step 2011 and the seed words obtained in step 2012.

[0085] It should be noted that the keywords contained in the word segmentation dictionary can be keywords of the target domain or can not be keywords of the target domain. Here, the purpose of constructing the word segmentation dictionary is to find all words in the historical texts belonging to the target domain so as to determine keywords belonging to the target domain from among them (i.e., in the word segmentation dictionary).

[0086] In practical applications, the words in the constructed word segmentation dictionary can also be preprocessed, such as removing stop words, converting traditional Chinese characters to simplified Chinese characters, converting case, symbol processing, etc.

[0087] Step 2014: Vector processing is performed on the historical text of the target field to obtain the feature vector of the seed word (i.e., the feature vector), and the words matched with the feature vector of the seed word are determined from the word segmentation dictionary as the similar words (including similar phrases) of the seed word. For example, the words with a similarity to the feature vector of the seed word satisfying a preset rule are selected from the word segmentation dictionary as the similar words of the seed word.

[0088] It can be understood that, in practical applications, the words in the preprocessed word segmentation dictionary are subjected to vector processing, such as obtaining the feature vectors of the words in the word segmentation dictionary based on an incremental training manner, using a management tool (such as manually controlling the words, etc.), or a word2vec (word to vector), GloVe model, shallow network (FastText) training word vector manner.

[0089] Step 2015: The seed word and the similar words of the seed word are taken as the initial field words (i.e., the initial key words) of the target field to obtain a field word library (i.e., a key word set).

[0090] It should be noted that, in practical applications, the seed word can also be added manually to obtain the similar words of the added seed word based on the feature vector of the seed word, so as to expand the key word set.

[0091] Step 202: The initial field sentences (i.e., the initial key sentences) belonging to the target field are mined from the historical text belonging to the target field to obtain a field sentence library.

[0092] As shown in FIG. 2(d), the specific steps are as follows:

[0093] Step 2021: The historical text of the target field is preprocessed, such as replacing continuous numbers, stop words, punctuation marks, special symbols, etc. with spaces.

[0094] Step 2022: The historical text is segmented according to the preprocessed spaces to obtain multiple sentences (i.e., multiple initial sentences).

[0095] Step 2023: The initial sentences obtained after segmentation are preliminarily screened using the initial field words in the field word library, and the screened initial sentences containing the initial field words are taken as the initial field sentences (i.e., the initial key sentences) to obtain a field sentence library (i.e., a key sentence set).

[0096] It can be understood that, in order to further improve the number and accuracy of the obtained initial domain sentences, the initial domain words can also be combined to obtain combined domain words, or manual intervention (such as manually adding domain words) and the like are used to mine the initial domain words to obtain the domain sentence library.

[0097] Step 2024: Comparing the initial domain sentences in the domain sentence library with the non-domain sentences (i.e., preset key sentences) in the non-domain sentence library (also referred to as a preset sentence set) to obtain similarity information (i.e., the second similarity information described above), and if the similarity information (i.e., the second similarity information described above) satisfies the second similarity rule, deleting the initial domain sentences whose similarity information (i.e., the second similarity information described above) satisfies the second similarity rule (such as exceeding the second threshold) from the domain sentence library and updating the domain sentence library.

[0098] It should be noted that if the similarity between a specific initial domain sentence and a specific non-domain sentence exceeds the second threshold, it means that the specific initial domain sentence is not a domain sentence belonging to the target domain, and at this time, the specific initial domain sentence can be deleted from the domain sentence library.

[0099] According to the above manner, each initial domain sentence in the domain sentence library is compared with the non-domain sentence one by one to update the domain sentence library, so as to avoid the appearance of non-target domain sentences in the domain sentence library.

[0100] In addition, it should be noted that in actual applications, the initial domain sentences in the domain sentence library obtained based on the above manner can also be subjected to clustering processing, such as clustering processing of the domain sentence library based on feature extraction, short text clustering algorithm, etc., so as to reduce the number of initial domain sentences in the domain sentence library, and thus facilitate the reduction of the later retrieval amount.

[0101] Second part: based on the constructed domain word library and domain sentence library, the target scene, such as the text (i.e., the to-be-processed text) of the obscene scene under the content wind control scene, is screened and identified, as shown in FIGS. 3(a) and 3(b), and the specific steps include:

[0102] Step 301: inputting the to-be-processed text, and dividing the to-be-processed text into a positive sample sentence library and a negative sample sentence library according to the target domain word (i.e., the target key word).

[0103] Here, the sentences contained in the positive sample sentence library (i.e., the candidate set) are sentences containing the target keyword (i.e., the above-mentioned candidate to-be-processed sentence); the sentences contained in the negative sample sentence library are sentences not including the target keyword of the target field (i.e., the above-mentioned non-candidate to-be-processed sentence); further, in order to facilitate screening, a preset word set can be pre-set, the preset word set including a plurality of preset keywords, the preset keywords being specified keywords not belonging to the target field, i.e., specified non-field words; based on this, the sentences contained in the negative sample sentence library are sentences containing the preset keywords (i.e., the non-candidate to-be-processed sentence).

[0104] The specific steps include:

[0105] Step 3011: Preprocessing the to-be-processed text, such as replacing consecutive numbers with <num>, non-Chinese English numbers and very used punctuation marks are replaced with spaces. According to the pre-processed space, the to-be-processed text is processed to obtain a plurality of initial to-be-processed sentences.

[0106] Step 3012: From the plurality of initial to-be-processed sentences, the initial to-be-processed sentences including the target domain word (i.e. the target keyword) are screened out as candidate to-be-processed sentences, and a positive sample sentence library is obtained; From the plurality of initial to-be-processed sentences, the initial to-be-processed sentences including the domain whitelist (generally manually preset) (the domain whitelist includes the specified non-domain word, i.e. the preset keyword described above) are screened out to obtain a domain sentence white library (also called a negative sample library), The sentences in the domain sentence white library are all non-domain sentences, i.e. non-candidate processing sentences.

[0107] It should be noted that not all non-domain words can be used as the domain whitelist. In actual application, the specified non-domain words can be used as the non-domain words of the domain whitelist based on manual screening.

[0108] It should be noted that some initial to-be-processed sentences may not fall into the positive sample sentence library and also not fall into the negative sample sentence library. At this time, the initial to-be-processed sentences that do not fall into the positive sample can be used as non-candidate to-be-processed sentences and fall into the negative sample sentence library.

[0109] In actual application, the target domain word can be obtained in the following manner: The initial domain words in the domain word library are sorted, such as sorting the initial domain words in the domain word library from high to low based on importance or priority (here, in actual application, in order to improve the accuracy of matching, the sorting can also be based on words), and selecting the initial domain words located in the front of the preset number as the target domain words based on the sorting result. Then, the initial to-be-processed sentences are screened based on the target domain words.

[0110] It can be understood that English / numbers and the like can be specially processed during the sorting process.

[0111] Step 302: The candidate to-be-processed sentences in the obtained positive sample sentence library are further screened; specifically, after the target domain word is determined, the initial domain sentence corresponding to the target domain word is selected from the domain sentence library as the target domain sentence (i.e. the target key sentence); the feature vectors of each target domain sentence and the feature vectors of the candidate to-be-processed sentences are determined, such as performing feature vectorization processing (i.e. sentence dimension Embedding) on the target domain sentence and the candidate to-be-processed sentence, and then comparing the feature vectors of the target domain sentence and the feature vectors of the candidate to-be-processed sentence, such as measuring the similarity between sentences based on cosine similarity.

[0112] It should be noted that in the process of feature vectorization, that is, in the process of constructing the sentence vector, the weight of the target field word in the target field sentence can be increased, for example, the weight of the target field word in the target field sentence can be adjusted using TF-IDF (term frequency-inverted document frequency); or the keyword extraction method, word vector splicing method, etc. can also be used to construct the sentence vector, etc. In this way, the sentences belonging to the target field can be determined from the text to be processed.

[0113] Further, when performing similarity comparison, comparison can be performed based on edit distance, sentence vector cosine similarity, field weight, etc. In order to further improve the accuracy of the result, the similarity matching result can also be subjected to deep text matching after the similarity matching is completed, such as shallow conversion (Transformer) model, or text classification model (TextCNN) / shallow network (FastText) for matching, so as to obtain the matching result.

[0114] In actual application, in the process of similarity matching based on sentence dimension, different lengths of text or sentences can also be treated differently, so as to improve the matching accuracy; or GPU resources can be used for rapid processing, such as rapid exclusion of complex models or distillation of complex models.

[0115] Step 303: obtaining a similar sentence based on the comparison result, that is, determining that the similarity between the feature vector of the target field sentence and the feature vector of the candidate sentence to be processed is greater than the first threshold value based on the comparison result, at this time, it can be considered that the target field sentence and the candidate sentence to be processed are similar sentences, and then the candidate sentence to be processed corresponding to the similar sentence is regarded as a target processing sentence, so as to determine all target processing sentences in the candidate set, that is, the positive sample sentence library, and mark each target processing sentence or the paragraph where each target processing sentence is located in the text to be processed, completing the processing process.

[0116] In actual application, after obtaining the similar sentence, the similar sentence can also be input into the text matching model for verification, so as to further improve the matching accuracy.

[0117] It should be noted that the above-mentioned preset sentence set (including the preset key sentence, that is, the non-field sentence) and the preset word set (including the preset keyword, that is, the specified non-field word) can be obtained through the historical text of the target field, for example, it can be obtained based on the obtained field word library and field sentence library, and the present disclosure does not limit this.

[0118] It can be understood that in actual application, the field word library, the field sentence library, and the non-field word library and the non-field sentence library can be updated through continuous iteration of historical files, so as to improve the accuracy of field sample control.

[0119] In this way, the initial processing sentence can be effectively identified by the method, and the identification process can anchor the target field, so that the purpose of effectively identifying the target processing sentence belonging to the target field is achieved, and the recognition ability of the variant word (i.e., the related word described above) is also strong, which improves the overall recognition efficiency and recognition accuracy, and provides technical support for purifying the network environment.

[0120] In addition, since the method can identify based on the sentence dimension and obtain the target processing sentence, it also has strong recognition ability for sentences with large variants, and further improves the overall recognition efficiency and recognition accuracy, and the result is highly interpretable.

[0121] At the same time, the method can also continuously maintain the data sample, i.e., the historical text, so as to further improve the control ability and further improve the recognition accuracy.

[0122] The method also provides a text processing device, as shown in Figure 4 The device includes:

[0123] The screening unit 401 is configured to screen an initial processing sentence containing a target keyword from a plurality of initial processing sentences as a candidate processing sentence, and obtain a candidate set containing at least one candidate processing sentence; wherein the target keyword is selected from a plurality of initial keywords in a keyword set; and the initial keyword is a keyword belonging to a target field.

[0124] The comparison unit 402 is configured to compare the candidate processing sentence with a target key sentence containing the target keyword, and obtain first similarity information.

[0125] The target determination unit 403 is configured to determine the candidate processing sentence satisfying a first similarity rule as a target processing sentence belonging to the target field.

[0126] In a specific example of the method, the method further includes:

[0127] The first preprocessing unit is configured to preprocess the processing text to obtain the plurality of initial processing sentences.

[0128] In a specific example of the method, the method further includes:

[0129] The text marking unit is configured to, in a case where a target processing sentence belonging to the target field is obtained, mark the text to be processed based on at least one of the following manners:

[0130] marking the target processing sentence;

[0131] marking a paragraph in which the target processing sentence is located.

[0132] In a specific example of the present disclosure, the screening unit specifically includes:

[0133] screening, from the plurality of initial processing sentences, an initial processing sentence containing a target keyword as a candidate processing sentence;

[0134] screening, from the plurality of initial processing sentences, an initial processing sentence containing a preset keyword as a non-candidate processing sentence; wherein the preset keyword is a keyword not belonging to the target field;

[0135] obtaining a candidate set based on the candidate processing sentence and other initial processing sentences in the plurality of initial processing sentences except the non-candidate processing sentence.

[0136] In a specific example of the present disclosure, it further includes:

[0137] A keyword processing unit is configured to determine a plurality of initial keywords belonging to the target field based on historical text of the target field, to obtain the keyword set; and perform sorting processing on the initial keywords in the keyword set, and select at least one target keyword from the plurality of initial keywords based on a sorting result.

[0138] In a specific example of the present disclosure, the keyword processing unit specifically includes:

[0139] obtaining a plurality of feature words belonging to the target field based on historical text of the target field;

[0140] taking the feature words as the initial keywords, or taking the feature words and similar words of the feature words as the initial keywords.

[0141] In a specific example of the present disclosure, the keyword processing unit is further configured to:

[0142] performing vector processing on the historical text to obtain a feature vector of the feature words;

[0143] selecting, from a segmentation dictionary established based on the historical text, a word having a similarity satisfying a preset rule with the feature vector of the feature words as a similar word of the feature words.

[0144] In a specific example of the technical scheme of the present disclosure, the keyword processing unit is further configured to:

[0145] obtain a plurality of initial words based on the historical text of the target domain, and obtain at least one feature word belonging to the target domain;

[0146] construct a word segmentation dictionary containing the plurality of initial words and the at least one feature word.

[0147] In a specific example of the technical scheme of the present disclosure, the technical scheme further comprises:

[0148] The second preprocessing unit is configured to preprocess the historical text of the target domain to obtain a plurality of initial sentences.

[0149] The key sentence processing unit is configured to filter an initial sentence containing the initial keyword from the plurality of initial sentences as an initial key sentence to obtain a key sentence set, wherein the target key sentence is one of the initial key sentences contained in the key sentence set.

[0150] In a specific example of the technical scheme of the present disclosure, the key sentence processing unit is further configured to:

[0151] compare the initial key sentence in the key sentence set with a preset key sentence to obtain second similarity information, wherein the preset key sentence is a key sentence containing a keyword not belonging to the target domain;

[0152] delete the initial key sentence from the key sentence set, which satisfies a second similarity rule.

[0153] The specific functions of each unit in the above device can refer to the description of the above method, which will not be repeated here.

[0154] In the technical scheme of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0155] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0156] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0157] As shown in Figure 5 The device 500 includes a computing unit 501 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 502 or a computer program loaded into a random access memory (RAM) 503 from a storage unit 508. Various programs and data required for the operation of the device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0158] Various components in the device 500 are connected to the I / O interface 505, including an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; the storage unit 508, such as a magnetic disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0159] The computing unit 501 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs various methods and processes described above, such as the text processing method. For example, in some embodiments, the text processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded onto the RAM 503 and executed by the computing unit 501, one or more steps of the text processing method described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the text processing method by any other appropriate means, such as by means of firmware.

[0160] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0161] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine or server, or entirely on a remote machine or server.

[0162] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage medium can include, but are not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0163] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0164] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0165] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0166] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps removed, for example. The steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, and the present disclosure is not limited herein.

[0167] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.< / num>

Claims

1. A text processing method, comprising: selecting, from a plurality of initial sentences to be processed, an initial sentence to be processed containing a target keyword as a candidate sentence to be processed, to obtain a candidate set containing at least one candidate sentence to be processed, comprising: selecting, from a plurality of initial sentences to be processed, an initial sentence to be processed containing a target keyword as a candidate sentence to be processed; wherein the target keyword is selected from a plurality of initial keywords in a keyword set; and the initial keyword is a keyword belonging to a target domain; selecting, from a plurality of initial sentences to be processed, an initial sentence to be processed containing a preset keyword as a non-candidate sentence to be processed; wherein the preset keyword is a keyword not belonging to the target domain; obtaining the candidate set based on the candidate sentence to be processed and other initial sentences to be processed except the non-candidate sentence to be processed; comparing the candidate sentence to be processed with a target key sentence containing the target keyword to obtain first similarity information; taking the candidate sentence to be processed satisfying a first similarity rule as a target sentence to be processed belonging to the target domain; wherein the method further comprises: preprocessing historical text of the target domain to obtain a plurality of initial sentences; selecting, from the plurality of initial sentences, an initial sentence containing an initial keyword as an initial key sentence to obtain a key sentence set; wherein the target key sentence is one of the initial key sentences contained in the key sentence set; comparing the initial key sentence in the key sentence set with a preset key sentence to obtain second similarity information; wherein the preset key sentence is a key sentence containing a keyword not belonging to the target domain; deleting the initial key sentence in the key sentence set satisfying a second similarity rule from the key sentence set to update the key sentence set. 2.The method of claim 1, further comprising: preprocessing a text to be processed to obtain the plurality of initial sentences to be processed. 3.The method of claim 2, further comprising: in a case where the target sentence to be processed belonging to the target domain is obtained, marking the text to be processed based on at least one of the following ways: marking the target sentence to be processed; marking a paragraph in which the target sentence to be processed is located. 4.The method of claim 1, further comprising: determining a plurality of initial keywords belonging to the target domain based on historical text of the target domain to obtain the keyword set; sorting the initial keywords in the keyword set, and selecting at least one target keyword from the plurality of initial keywords based on a sorting result.

5. The method of claim 4, wherein, The method of determining a plurality of initial keywords belonging to the target domain based on historical text of the target domain, comprising: obtaining a plurality of feature keywords belonging to the target domain based on the historical text of the target domain; taking the feature keywords as the initial keywords, or taking the feature keywords and similar keywords of the feature keywords as the initial keywords. 6.The method of claim 5, further comprising: performing vector processing on the historical text to obtain a feature vector of the feature keywords. Select a word that meets a preset rule in similarity with the feature vector of the feature word from a word segmentation dictionary established based on the historical text, as a similar word of the feature word.

7. The method of claim 6, further comprising: obtaining a plurality of initial words based on historical text of the target domain, and obtaining at least one feature word belonging to the target domain; constructing a word segmentation dictionary containing the plurality of initial words and the at least one feature word.

8. A text processing apparatus, comprising: a screening unit configured to screen an initial sentence containing a target keyword from a plurality of initial sentences to be processed as a candidate sentence to be processed, and obtain a candidate set containing at least one candidate sentence to be processed; wherein the target keyword is selected from a plurality of initial keywords in a keyword set; and the initial keyword is a keyword belonging to a target domain; a comparison unit configured to compare the candidate sentence to be processed with a target keyword sentence containing the target keyword, and obtain first similarity information; a target determination unit configured to determine the candidate sentence to be processed that meets a first similarity rule in the first similarity information as a target sentence to be processed belonging to the target domain; wherein the screening unit specifically comprises: screening an initial sentence containing a target keyword from a plurality of initial sentences to be processed as a candidate sentence to be processed; screening an initial sentence containing a preset keyword from the plurality of initial sentences to be processed as a non-candidate sentence to be processed; wherein the preset keyword is a keyword not belonging to the target domain; obtaining the candidate set based on the candidate sentence to be processed and other initial sentences to be processed except the non-candidate sentence to be processed; wherein the apparatus further comprises: a second preprocessing unit configured to preprocess historical text of the target domain to obtain a plurality of initial sentences; a key sentence processing unit configured to screen an initial sentence containing an initial keyword from the plurality of initial sentences as an initial key sentence to obtain a key sentence set; wherein the target keyword sentence is one of the initial key sentences contained in the key sentence set; compare the initial key sentence in the key sentence set with a preset keyword to obtain second similarity information; wherein the preset keyword is a keyword not belonging to the target domain; and delete the initial key sentence meeting a second similarity rule in the second similarity information from the key sentence set to update the key sentence set.

9. The apparatus of claim 8, further comprising: a first preprocessing unit configured to preprocess the text to be processed to obtain the plurality of initial sentences to be processed.

10. The apparatus of claim 9, further comprising: a text marking unit configured to mark the text to be processed in the case that the target sentence to be processed belongs to the target domain based on at least one of the following ways: marking the target sentence to be processed; marking a paragraph where the target sentence to be processed is located.

11. The apparatus of claim 8, further comprising: a keyword processing unit configured to determine a plurality of initial keywords belonging to the target domain based on historical text of the target domain to obtain the keyword set. The initial keywords in the keyword set are sorted, and at least one target keyword is selected from the plurality of initial keywords based on a sorting result.

12. The apparatus of claim 11, wherein, The keyword processing unit specifically comprises: Based on the historical text of the target field, a plurality of feature words belonging to the target field are obtained. The feature words are used as the initial keywords, or the feature words and similar words of the feature words are used as the initial keywords.

13. The apparatus of claim 12, wherein, The keyword processing unit is further configured to: The historical text is vector processed to obtain a feature vector of the feature word. Words with a similarity to the feature vector of the feature word satisfying a preset rule are selected from a segmentation dictionary established based on the historical text as similar words of the feature word.

14. The apparatus of claim 13, wherein, The keyword processing unit is further configured to: Based on the historical text of the target field, a plurality of initial words and at least one feature word belonging to the target field are obtained. A segmentation dictionary containing the plurality of initial words and the at least one feature word is constructed. 15.An electronic device comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-7. 17.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-7.

Images